Skip to content

Commit

Permalink
refactor build plan pipeline
Browse files Browse the repository at this point in the history
Previously the BuildPlan pipeline didn't execute generators and
transformers concurrently.  All steps were sequentially executed.  Holos
was primarily concurrent by executing multiple BuildPlans at once.

This patch changes the Build implementation for each BuildPlan to
execute a GoRoutine pipeline.  One producer fans out to a group of
routines each executing the pipeline for one artifact in the build plan.
The pipeline has 3 stages:

1: Fan-out to build each Generator concurrently.
2: Fan-in to build each Transformer sequentially.
3: Fan-out again to run each validator concurrently.

When the artifact pipelines return, the producer closes the tasks
channel causing the worker tasks to return.

Note the overall runtime for 8 BuildPlans is roughly equivalent to
previously at 160ms with --concurrency=8 on my M3 Max.  I expect this to
perform better than previously when multiple artifacts are rendered for
each BuildPlan.
  • Loading branch information
jeffmccune committed Nov 29, 2024
1 parent 7530345 commit ee16f14
Show file tree
Hide file tree
Showing 4 changed files with 376 additions and 293 deletions.
8 changes: 4 additions & 4 deletions cmd/holos/tests/v1alpha5/guides/helm.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@ cd $WORK
exec holos generate platform v1alpha5 --force

# Platforms are empty by default.
exec holos render platform ./platform
exec holos render platform
stderr -count=1 '^rendered platform'

# Holos uses CUE to build a platform specification.
exec cue export --expression holos --out=yaml ./platform
exec holos show platform
cmp stdout want/1.platform_spec.yaml

# Define the host and port in projects/blackbox.schema.cue
Expand All @@ -22,7 +22,7 @@ mv projects/platform/components/prometheus/prometheus.cue.disabled projects/plat
mv platform/prometheus.cue.disabled platform/prometheus.cue

# Render the platform to render the prometheus chart.
exec holos render platform ./platform
exec holos render platform
stderr -count=1 '^rendered prometheus'
stderr -count=1 '^rendered platform'
cmp deploy/components/prometheus/prometheus.gen.yaml want/1.prometheus.gen.yaml
Expand Down Expand Up @@ -73,8 +73,8 @@ core.#BuildPlan & {
metadata: name: _Tags.component.name
}
-- want/1.platform_spec.yaml --
kind: Platform
apiVersion: v1alpha5
kind: Platform
metadata:
name: default
spec:
Expand Down
1 change: 1 addition & 0 deletions internal/artifact/artifact.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ var _ Store = NewStore()
type Store interface {
Get(path string) (data []byte, ok bool)
Set(path string, data []byte) error
// Save previously set path to dir preserving directories.
Save(dir, path string) error
}

Expand Down
Loading

0 comments on commit ee16f14

Please sign in to comment.