Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Look for recipes in ./recipes/, build files in ./files/, and Containerfiles in ./containerfiles/ #157

Merged

Conversation

gmpinder
Copy link
Member

@gmpinder gmpinder commented Apr 11, 2024

Recipe files can now be put into their own directory ./recipes/. This directory is NEVER copied into the build so changes to a recipe will no longer cause cache misses for builds. Here is an example of my build changing the second to last module and only requiring the last 2 RUN layers to be run again.

 => CACHED [stage-config 1/1] COPY ./config /config                                                                                                       0.0s
 => CACHED [stage-modules 1/2] COPY --from=ghcr.io/blue-build/modules:latest /modules /modules                                                            0.0s
 => CACHED [stage-modules 2/2] COPY ./modules /modules                                                                                                    0.0s
 => CACHED [stage-keys 1/1] COPY cosign.pub /keys/jp-desktop-gaming.pub                                                                                   0.0s
 => CACHED [stage-4  2/16] RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys   mkdir -p /usr/etc/pki/containers/   && cp /tmp/keys/* /usr/et  0.0s
 => CACHED [stage-bins 1/3] COPY --from=gcr.io/projectsigstore/cosign /ko-app/cosign /bins/cosign                                                         0.0s
 => CACHED [stage-bins 2/3] COPY --from=docker.io/mikefarah/yq /usr/bin/yq /bins/yq                                                                       0.0s
 => CACHED [stage-bins 3/3] COPY --from=ghcr.io/blue-build/cli:main-installer /out/bluebuild /bins/bluebuild                                              0.0s
 => CACHED [stage-4  3/16] RUN --mount=type=bind,from=stage-bins,src=/bins,dst=/tmp/bins   mkdir -p /usr/bin/   && cp /tmp/bins/* /usr/bin/   && ostree   0.0s
 => CACHED [stage-4  4/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  5/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  6/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  7/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  8/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  9/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 10/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 11/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 12/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 13/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 14/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => [stage-4 15/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind,from=  33.4s
 => [stage-4 16/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind,from=s  0.7s

Making it so that an upgrade only required pulling those last 2 layers:

$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop (digest: sha256:01073d98adf4041f3b91840af1135b15aee97db90de6bc3a3846d67c07345e6a)
ostree chunk layers already present: 65
custom layers already present: 16
custom layers needed: 2 (255.0 MB)

Support was also added to put all build files into ./files/ instead of ./config/. This is an all or nothing operation, meaning if there exists a directory of files then the config directory will be completely ignored. Work will have to be done in https://github.com/blue-build/modules to allow users to put their files directly in ./files/ and not ./files/files for the files module or ./files/scripts for the scripts module.

Support was also added to move the ./config/containerfiles/ directory to the root of the project. Now the directories you can find in the root of projects are:

files/
containerfiles/
recipes/

@gmpinder gmpinder added this to the v0.8.4 milestone Apr 11, 2024
@gmpinder gmpinder self-assigned this Apr 11, 2024
@gmpinder gmpinder linked an issue Apr 11, 2024 that may be closed by this pull request
@gmpinder
Copy link
Member Author

Just realized I didn't do the same for the containerfile module. Let me fix that

@gmpinder gmpinder changed the title feat: Look for recipes in ./recipes/ and build files in ./files/ feat: Look for recipes in ./recipes/, build files in ./files/, and Containerfiles in ./containerfiles/ Apr 11, 2024
@gmpinder gmpinder enabled auto-merge (squash) April 11, 2024 20:45
@fiftydinar
Copy link
Contributor

fiftydinar commented Apr 12, 2024

I agree with this one. It improves the builds as you mentioned & directory layout looks cleaner to manage.

The only thing is that putting usr for example directly in files folder can get lost if user has a lot of folder inclusions from other modules, like systemd, gschema-overrides & similar.
This is very apparent in cases when there are multiple custom image variants.

Maybe files/os_files/ is better solution here?

@xynydev
Copy link
Member

xynydev commented Apr 12, 2024

@fiftydinar In my view, after modules are refactored.

The files/ directory could look like this:

  • usr/
  • gschema-overrides/
  • systemd/

Or it could look like this:

  • system_files/
    • usr/
    • etc/
  • modules/
    • systemd/
    • gschema/

etc. etc.

And so, it would be up to the maintainer to decide how to arrange their repository.

@fiftydinar
Copy link
Contributor

fiftydinar commented Apr 12, 2024

The files/ directory could look like this:

...

Or it could look like this:

- `system_files/`
  
  - `usr/`
  - `etc/`

- `modules/`
  
  - `systemd/`
  - `gschema/`

I like this one

@gmpinder
Copy link
Member Author

Yeah the general idea still is to keep all the files in one directory still so the CLI doesn't have to keep track of logic that determines where to pull files from

@gmpinder gmpinder added the type: feature Brand new functionality, features, pages, workflows, endpoints, etc. label Apr 12, 2024
@xynydev
Copy link
Member

xynydev commented Apr 13, 2024

Cool, yeah! I think I've come around on the ./recipes/ folder naming, and it's my understanding that this PR doesn't have any breaking changes, so I'll approve. We should discuss the details of this transition elsewhere.

@gmpinder gmpinder merged commit f8b7334 into main Apr 13, 2024
3 checks passed
@gmpinder gmpinder deleted the 138-proposal-support-having-recipe-files-stored-in-recipes branch April 13, 2024 19:08
@gmpinder gmpinder mentioned this pull request Apr 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature Brand new functionality, features, pages, workflows, endpoints, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: Support having recipe files stored in ./recipes
3 participants