Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support OSTree archive as tarball #715

Open
agners opened this issue Apr 28, 2020 · 4 comments
Open

Support OSTree archive as tarball #715

agners opened this issue Apr 28, 2020 · 4 comments

Comments

@agners
Copy link
Contributor

agners commented Apr 28, 2020

To decouple the build system from OSTree repository handling a tarball containing the OSTree repository in archive mode with just the single commit would be useful. This tarball can then be copied/moved/uploaded to a server where it then gets moved into a (potentially public) repository (e.g. using pull-local).

This is how the Fedora CoreOS builder currently works, where one of the build output artifacts is a tarball with the OSTree with just the single commit of this particular build.

@pattivacek
Copy link
Collaborator

To decouple the build system from OSTree repository handling a tarball containing the OSTree repository in archive mode with just the single commit would be useful.

If the goal is just decoupling, don't we achieve that with garage-push as well as with the tarballs that are already generated? Is the difference in your goal that you want a tarball with the repo/filesystem as OSTree sees it? That sounds reasonable to me and also shouldn't be particularly hard to add as an output or entry in IMAGE_FSTYPES.

@agners
Copy link
Contributor Author

agners commented Apr 28, 2020

The tarballs we generate are either pure rootfs or images with OSTree in bare mode. What I am looking for is a tarball which just contains an OSTree in archive mode. Basically whatever ${OSTREE_REPO} tar'ed. This then allows to do the garage-push later in a separate step (e.g. after testing/promoting the release...).

Currently there is no IMAGE_FSTYPES for that, but it is actually fairly easy to do by just adding the following to classes/image_types_ostree.bbclass:

TAR_IMAGE_ROOTFS_task-image-ostreecommit = "${OSTREE_REPO}"

Then adding

IMAGE_FSTYPES += "ostreecommit.tar.xz"

Creates a tarball of that OSTree in archive mode. However, if that OSTREE_REPO is shared, then this of course contains more than the commit of that particular build. Also OSTREE_REPO is not cleaned after a build.

So I see two ways here:

  • Add a variable e.g. OSTREE_REPO_VOLATILE, which if set just deletes the OSTREE_REPO after the build. One has to make sure that ${OSTREE_REPO} points to a location which is not shared with other builds. With this approach the user can have either a shared ${OSTREE_REPO} or the tarballs.
  • Do something more involved, and build a OSTree archive somewhere in build tmp and then ostree pull-local that commit to ${OSTREE_REPO}. This way the user can have a tarball as well as a repo with all builds...

@eu-siemann
Copy link
Contributor

I think it would be nice to have this. For our general use-case, garage-push is still better as it uploads only the missing objects. But it might become useful for the potential "offline USB-update" use-case. Having a tarball of the repo as a build artifact is definitely more convenient than doing pull-local on the build machine to the USB drive.

Also OSTREE_REPO is not cleaned after a build.

BTW, I'm not sure if that's intentional. AFAIK older commits are not used anywhere. Normally, ${IMGDEPLOYDIR} will contain only the latest image files, so this behavior may be surprising for users, as after a while it can take a significant amount of disk space.

Also, is there a reason to have ${OSTREE_REPO} in ${IMGDEPLOYDIR} and not in ${WORKDIR}?

@pattivacek
Copy link
Collaborator

pattivacek commented May 5, 2020

Also OSTREE_REPO is not cleaned after a build.

BTW, I'm not sure if that's intentional. AFAIK older commits are not used anywhere. Normally, ${IMGDEPLOYDIR} will contain only the latest image files, so this behavior may be surprising for users, as after a while it can take a significant amount of disk space.

Agreed, I don't think we rely on this functionality.

Also, is there a reason to have ${OSTREE_REPO} in ${IMGDEPLOYDIR} and not in ${WORKDIR}?

Good question, but this might be philosophical. It is arguably deployed, and it's definitely an image, so I don't think it's unreasonable to be in $IMGDEPLOYDIR, but $WORKDIR seems fine, too.

Currently there is no IMAGE_FSTYPES for that, but it is actually fairly easy to do by just adding the following to classes/image_types_ostree.bbclass:

TAR_IMAGE_ROOTFS_task-image-ostreecommit = "${OSTREE_REPO}"

Then adding

IMAGE_FSTYPES += "ostreecommit.tar.xz"

Creates a tarball of that OSTree in archive mode.

Yep, makes sense and sounds reasonable to me. I'm open to this route.

So I see two ways here:

* Add a variable e.g. `OSTREE_REPO_VOLATILE`, which if set just deletes the `OSTREE_REPO` after the build. One has to make sure that ${OSTREE_REPO} points to a location which is not shared with other builds. With this approach the user can have either a shared `${OSTREE_REPO}` or the tarballs.

* Do something more involved, and build a OSTree archive somewhere in build tmp and then ostree pull-local that commit to `${OSTREE_REPO}`. This way the user can have a tarball as well as a repo with all builds...

How about this: by default we keep the repo volatile and clean it up preferably before reusing it. (It is sometimes useful to keep it after bitbaking to examine it.) Then we have a separate option that enables persisting a separate repo somewhere that won't get deleted as often (not sure what would be appropriate). Then we'd just pull from the temporary repo into the persistent one if the option is enabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants