fedora-kinoite:rawhide doesn't have a /usr/etc directory (it is created during rpm-ostree/bootc/ostree-rs-ext deploy)

[prydom]

After #132 was merged, recipes based on quay.io/fedora/fedora-kinoite/fedora-kinoite:rawhide no longer build because /usr/etc/ doesn't exist in that image.

You can see at https://github.com/ostreedev/ostree-rs-ext/tree/main?tab=readme-ov-file#filesystem-layout, https://github.com/ostreedev/ostree-rs-ext/blob/8d972c19b0a8410b5a74de6375a8b157eecc48bd/lib/src/tar/export.rs#L49 and https://github.com/ostreedev/ostree-rs-ext/blob/8d972c19b0a8410b5a74de6375a8b157eecc48bd/lib/src/tar/write.rs#L186 that /etc content is expected to be found at /etc/ in the image and once deployed into ostree it is moved to /usr/etc.

This is the error I currently get when building. I am building this recipe https://github.com/prydom/my-ostree-build/blob/main/config/fedora-kinoite-laptop.yml.

 > [stage-5  2/18] RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys   cp /tmp/keys/* /usr/etc/pki/containers/   && ostree container commit:
0.059 cp: cannot create regular file '/usr/etc/pki/containers/': No such file or directory
------
Containerfile:86
--------------------
  85 |     # Key RUN
  86 | >>> RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys \
  87 | >>>   cp /tmp/keys/* /usr/etc/pki/containers/ \
  88 | >>>   && ostree container commit
  89 |     
--------------------

https://github.com/prydom/my-ostree-build/actions/runs/8411510667/job/23031186736

[prydom]

The code which actually copies the /etc in the oci image to /usr/etc/ in ostree is here: https://github.com/ostreedev/ostree-rs-ext/blame/8d972c19b0a8410b5a74de6375a8b157eecc48bd/lib/src/tar/write.rs#L155

[gmpinder]

Ok so we'll have to create those directories then. Easy fix.

[prydom]

2c98a7a worked great!
Rebooted into the image and double checked that the sigstore keys were in the expected place. Thanks so much for the quick fix.

I am wondering if having that layer so early in the file will cause issues with layer caching since the public key file has a modify date equal to the time of the build. I'll trigger the workflow again and let you know if it is able to reuse that layer and those below.

/etc/pki/containers
❯ l
total 4.0K
-rw-r--r--. 1 root root 178 Mar 24 14:51 fedora-kinoite-laptop.pub

/etc/pki/containers
❯ stat fedora-kinoite-laptop.pub 
  File: fedora-kinoite-laptop.pub
  Size: 178             Blocks: 8          IO Block: 4096   regular file
Device: 0,34    Inode: 6485948     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:cert_t:s0
Access: 2024-03-24 14:51:23.015459704 -0700
Modify: 2024-03-24 14:51:23.015459704 -0700
Change: 2024-03-24 14:51:23.015459704 -0700
 Birth: 2024-03-24 14:51:23.015459704 -0700

❯ rpm-ostree status -v          
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree-image-signed:docker://ghcr.io/prydom/fedora-kinoite-laptop:latest (index: 0)
                   Digest: sha256:bb4d06e2fc35252ec712e7d046190c68b4ee6f5a3396df9205a06387a861db37
                  Version: Rawhide.20240324.n.0 (2024-03-24T21:46:41Z)
                   Commit: 45dd9b2b7d81d6558e4dcf423c4e641e83aa881788b0bbb24917cf8a3fb1c8c3
                   Staged: no
                StateRoot: fedora-kinoite

  ostree-image-signed:docker://ghcr.io/prydom/fedora-kinoite-laptop:latest (index: 1)
                   Digest: sha256:e8213bbe131f010947bed23c3d521a63b191107f4821298cde897c0a0dae314b
                  Version: Rawhide.20240323.n.1 (2024-03-24T05:32:45Z)
                   Commit: bcd36c95a480048e44482d5229453751b5178cf634b05ffdc45a69e250f1e6fd
                StateRoot: fedora-kinoite

❯ sudo ostree admin config-diff | grep pki

[prydom]

I am wondering if having that layer so early in the file will cause issues with layer caching since the public key file has a modify date equal to the time of the build. I'll trigger the workflow again and let you know if it is able to reuse that layer and those below.

Layer caching (enabled in my blue-build/github-actions fork: here and here) is working great. https://github.com/prydom/my-ostree-build/actions/runs/8412412632/job/23033237457 took only one minute to build and bootc didn't see any added layers.

So it fortunately seems my concern was unfounded.

❯ sudo bootc upgrade

No changes in ostree-image-signed:docker://ghcr.io/prydom/fedora-kinoite-laptop:latest => sha256:5d3f475f497ddef6e05c76ef9a2391bad72a12785a2c3224c8f960311af7002a
Pruned images: 0 (layers: 0, objsize: 863.6 MB)
Queued for next boot: ostree-image-signed:docker://ghcr.io/prydom/fedora-kinoite-laptop:latest
  Version: Rawhide.20240324.n.0
  Digest: sha256:5d3f475f497ddef6e05c76ef9a2391bad72a12785a2c3224c8f960311af7002a
Total new layers: 82    Size: 4.0 GB
Removed layers:   0     Size: 0 bytes
Added layers:     0     Size: 0 bytes

❯ rpm-ostree status -v
State: idle
AutomaticUpdates: disabled
Deployments:
  ostree-image-signed:docker://ghcr.io/prydom/fedora-kinoite-laptop:latest (index: 0)
                   Digest: sha256:5d3f475f497ddef6e05c76ef9a2391bad72a12785a2c3224c8f960311af7002a
                  Version: Rawhide.20240324.n.0 (2024-03-24T21:46:41Z)
                   Commit: 7bc442e4051a6ba0c7985e03fea8ab7f1a76ebaa96fa60649310d1f84b6cb7ab
                   Staged: yes
                StateRoot: fedora-kinoite

● ostree-image-signed:docker://ghcr.io/prydom/fedora-kinoite-laptop:latest (index: 1)
                   Digest: sha256:bb4d06e2fc35252ec712e7d046190c68b4ee6f5a3396df9205a06387a861db37
                  Version: Rawhide.20240324.n.0 (2024-03-24T21:46:41Z)
                   Commit: 45dd9b2b7d81d6558e4dcf423c4e641e83aa881788b0bbb24917cf8a3fb1c8c3
                StateRoot: fedora-kinoite
[...]

[gmpinder]

I didn't think about it that way. I could preserve the file's original timestamp for that purpose, though you'll most likely run into cache misses whenever you make a change in your recipe. Due to the legacy way of keeping all your scripts/files/recipes in ./config, the entire build would end up having to be re-run due to the stage-configs having new data. I have an inkling of an idea to attempt to support legacy and a new format where recipes are stored in ./recipes allowing changes to your recipe to not affect the state of your other files.

[gmpinder]

I've created a proposal for folks to look at here regarding the above idea. I'll be closing your issue for the time being.

[prydom]

@gmpinder, just a quick note. In the current rawhide builds - I'm not sure about Fedora 39's or uBlue's behavior - if there any files in /etc in the container they override those put in /usr/etc.

For container signing I actually need to add a step at the end of my build to copy /usr/etc/containers/policy.json to /etc/containers/policy.json for upgrades to work properly:
https://github.com/prydom/my-ostree-build/blob/cc03e7ff10cea5fa778666bca1eff76022a779db/config/fedora-kinoite-laptop.yml#L168

(Note: the || true is just there in case the upstream module changes its behavior to put the policy.json in /etc instead of /usr/etc since I don't yet pin the modules.)

[prydom]

To clarify, it seems that ostree-rs-ext will merge /etc in the container into /usr/etc in the decapsulated ostree commit so new files in /usr/etc stay but any conflicts will be resolved using the files in /etc.

[prydom]

Actually, looking at https://github.com/ostreedev/ostree-rs-ext/blob/3d686ab6bf6990d6bd07fead09805df19c22112b/lib/src/tar/write.rs#L284 I don't think the behavior is deterministic.

I think whichever file is processed last in that loop is what ends up in the final ostree commit and it's probable the entry list isn't stable (i.e. depends on your container layers). Which may explain why I didn't start experiencing this behavior until today.