Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compose: Add package-extensions treefile key #2055

Closed
cgwalters opened this issue Apr 13, 2020 · 7 comments · Fixed by #2439
Closed

compose: Add package-extensions treefile key #2055

cgwalters opened this issue Apr 13, 2020 · 7 comments · Fixed by #2439

Comments

@cgwalters
Copy link
Member

As part of implementing coreos/fedora-coreos-tracker#401 I propose adding an extensions key to the manifest. It'd look like:

package-extensions:
  - usbguard
  - firewalld

Same semantics as the packages key.

Then one would rpm-ostree compose tree ... --output-package-extensions /path/to/dir and we'd depsolve and download those packages plus the dependencies that aren't in the base tree into the output directory.

It'd be up to higher level tooling (like coreos-assembler) to run createrepo_c etc. and manage them.

The reason to add this to rpm-ostree is to handle the "depsolving from base tree" problem.
@cgwalters cgwalters mentioned this issue Apr 13, 2020
Closed
@cgwalters
Copy link
Member Author

Perhaps we want a separate rpm-ostree compose extensions command, since we need to be able to handle the situation where e.g. the base tree changes but the extensions do (or vice versa), so we can't compress things easily into a single --touch-if-changed. Separating it would also make it easier to handle the case where one wants to add a new extension to an existing build.

@ashcrow
Copy link
Member

ashcrow commented Apr 21, 2020

In either case we'd have the package-extensions section, correct? If so, I like the idea of having rpm-ostree compose extensions command. Having the command be explicit could also be helpful for when one wants to just pull down extensions and deps for verification, etc..

👍 to adding package-extensions to the manifest file.

@Prashanth684
Copy link

adding glusterfs-fuse to the list of extensions.

@jlebon
Copy link
Member

jlebon commented Apr 28, 2020

Not against teaching this to rpm-ostree directly, though worth mentioning another approach which doesn't require changing rpm-ostree at all is to just have an extensions.yaml type file in the src config and a cosa buildextend-extensions which would just add the packages to the manifest and run something like rpm-ostree compose tree --download-only-rpms --ex-lockfile=manifest-lock.generated.$basearch.json and collect the RPMs.

@cgwalters
Copy link
Member Author

cgwalters commented Apr 28, 2020 •
edited

just have an extensions.yaml type file in the src config and a cosa buildextend-extensions which would just add the packages to the manifest and run something like rpm-ostree compose tree --download-only-rpms --ex-lockfile=manifest-lock.generated.$basearch.json and collect the RPMs.

But don't we want to filter out packages which are in the base tree? That could still be done outside of rpm-ostree of course but...

@jlebon
Copy link
Member

jlebon commented Apr 28, 2020

But don't we want to filter out packages which are in the base tree?

It would only download the new packages since the base packages are already cached. But yeah, it's not very clean. (E.g. in the developer case, we can't assume that the cache wasn't just nuked entirely).

I think this is somewhat related to #442 too where we'll want to solve the exact same problem depsolve-wise (adding packages to a base).

@cgwalters
Copy link
Member Author

It would only download the new packages since the base packages are already cached. But yeah, it's not very clean. (E.g. in the developer case, we can't assume that the cache wasn't just nuked entirely).

Ah wow, yes...not sure I'd want to rely on that.

I think this is somewhat related to #442 too where we'll want to solve the exact same problem depsolve-wise (adding packages to a base).

Yeah.

jlebon added a commit to jlebon/rpm-ostree that referenced this issue Jan 11, 2021
@jlebon
app: Add rpm-ostree compose extensions
bf28aae 
This adds support for a new `package-extensions` key in the treefile as
well as a new `rpm-ostree compose extensions` command which takes a
treefile and a base OSTree rev, performs a depsolve, and downloads the
extensions to a provided output directory.

This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions

The major advantage here is that we have a guaranteed depsolve match and
thus can avoid silly issues we've hit in RHCOS (like downloading the
wrong `libprotobuf` for `usbguard` -- rhbz#1889694).

Closes: coreos#2055
@jlebon jlebon mentioned this issue Jan 11, 2021
Merged
jlebon added a commit to jlebon/rpm-ostree that referenced this issue Jan 21, 2021
@jlebon
app: Add rpm-ostree compose extensions
f46fb76 
This adds support for a new `rpm-ostree compose extensions` command`
which takes a treefile, a new extensions YAML file, and an OSTree repo
and ref. It performs a depsolve and downloads the extensions to a
provided output directory.

This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions

The input YAML schema matches the one accepted by that script.

Some differences from the script:
- We have a guaranteed depsolve match and thus can avoid silly issues
  we've hit in RHCOS (like downloading the wrong `libprotobuf` for
  `usbguard` -- rhbz#1889694).
- We seamlessly re-use the same repos defined in the treefile, whereas
  the cosa script uses `reposdir=$dir` which doesn't have the same
  semantics (repo enablement is in that case purely based on the
  `enabled` flag in those repos, which may be different than what the
  rpm-ostree compose ran with).
- We perform more sanity-checks against the requested extensions, such
  as whether the extension is already in the base.
- We support no-change detection via a state SHA512 file for better
  integration in cosa and pipelines.
- We support a `match-base-evr` key, which forces the extension to have
  the same EVR as the one from a base package: this is helpful in the
  case of extensions which complement a base package, esp. those which
  may not have strong enough reldeps to enforce matching EVRs by
  depsolve alone (`kernel-headers` is an example of this).
- We don't try to organize the RPMs into separate directories by
  extension because IMO it's not at the right level. Instead, we should
  work towards higher-level metadata to represent extensions (see
  openshift/os#409 which is related to this).

Closes: coreos#2055
jlebon added a commit to jlebon/rpm-ostree that referenced this issue Jan 22, 2021
@jlebon
app: Add rpm-ostree compose extensions
84b5e00 
This adds support for a new `rpm-ostree compose extensions` command`
which takes a treefile, a new extensions YAML file, and an OSTree repo
and ref. It performs a depsolve and downloads the extensions to a
provided output directory.

This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions

The input YAML schema matches the one accepted by that script.

Some differences from the script:
- We have a guaranteed depsolve match and thus can avoid silly issues
  we've hit in RHCOS (like downloading the wrong `libprotobuf` for
  `usbguard` -- rhbz#1889694).
- We seamlessly re-use the same repos defined in the treefile, whereas
  the cosa script uses `reposdir=$dir` which doesn't have the same
  semantics (repo enablement is in that case purely based on the
  `enabled` flag in those repos, which may be different than what the
  rpm-ostree compose ran with).
- We perform more sanity-checks against the requested extensions, such
  as whether the extension is already in the base.
- We support no-change detection via a state SHA512 file for better
  integration in cosa and pipelines.
- We support a `match-base-evr` key, which forces the extension to have
  the same EVR as the one from a base package: this is helpful in the
  case of extensions which complement a base package, esp. those which
  may not have strong enough reldeps to enforce matching EVRs by
  depsolve alone (`kernel-headers` is an example of this).
- We don't try to organize the RPMs into separate directories by
  extension because IMO it's not at the right level. Instead, we should
  work towards higher-level metadata to represent extensions (see
  openshift/os#409 which is related to this).

Closes: coreos#2055
jlebon added a commit to jlebon/rpm-ostree that referenced this issue Jan 22, 2021
@jlebon
app: Add rpm-ostree compose extensions
7fa87a0 
This adds support for a new `rpm-ostree compose extensions` command`
which takes a treefile, a new extensions YAML file, and an OSTree repo
and ref. It performs a depsolve and downloads the extensions to a
provided output directory.

This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions

The input YAML schema matches the one accepted by that script.

Some differences from the script:
- We have a guaranteed depsolve match and thus can avoid silly issues
  we've hit in RHCOS (like downloading the wrong `libprotobuf` for
  `usbguard` -- rhbz#1889694).
- We seamlessly re-use the same repos defined in the treefile, whereas
  the cosa script uses `reposdir=$dir` which doesn't have the same
  semantics (repo enablement is in that case purely based on the
  `enabled` flag in those repos, which may be different than what the
  rpm-ostree compose ran with).
- We perform more sanity-checks against the requested extensions, such
  as whether the extension is already in the base.
- We support no-change detection via a state SHA512 file for better
  integration in cosa and pipelines.
- We support a `match-base-evr` key, which forces the extension to have
  the same EVR as the one from a base package: this is helpful in the
  case of extensions which complement a base package, esp. those which
  may not have strong enough reldeps to enforce matching EVRs by
  depsolve alone (`kernel-headers` is an example of this).
- We don't try to organize the RPMs into separate directories by
  extension because IMO it's not at the right level. Instead, we should
  work towards higher-level metadata to represent extensions (see
  openshift/os#409 which is related to this).

Closes: coreos#2055
jlebon added a commit to jlebon/rpm-ostree that referenced this issue Jan 22, 2021
@jlebon
app: Add rpm-ostree compose extensions
d8098d6 
This adds support for a new `rpm-ostree compose extensions` command`
which takes a treefile, a new extensions YAML file, and an OSTree repo
and ref. It performs a depsolve and downloads the extensions to a
provided output directory.

This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions

The input YAML schema matches the one accepted by that script.

Some differences from the script:
- We have a guaranteed depsolve match and thus can avoid silly issues
  we've hit in RHCOS (like downloading the wrong `libprotobuf` for
  `usbguard` -- rhbz#1889694).
- We seamlessly re-use the same repos defined in the treefile, whereas
  the cosa script uses `reposdir=$dir` which doesn't have the same
  semantics (repo enablement is in that case purely based on the
  `enabled` flag in those repos, which may be different than what the
  rpm-ostree compose ran with).
- We perform more sanity-checks against the requested extensions, such
  as whether the extension is already in the base.
- We support no-change detection via a state SHA512 file for better
  integration in cosa and pipelines.
- We support a `match-base-evr` key, which forces the extension to have
  the same EVR as the one from a base package: this is helpful in the
  case of extensions which complement a base package, esp. those which
  may not have strong enough reldeps to enforce matching EVRs by
  depsolve alone (`kernel-headers` is an example of this).
- We don't try to organize the RPMs into separate directories by
  extension because IMO it's not at the right level. Instead, we should
  work towards higher-level metadata to represent extensions (see
  openshift/os#409 which is related to this).

Closes: coreos#2055
openshift-merge-robot pushed a commit that referenced this issue Jan 23, 2021
@jlebon @openshift-merge-robot
app: Add rpm-ostree compose extensions
271954a 
This adds support for a new `rpm-ostree compose extensions` command`
which takes a treefile, a new extensions YAML file, and an OSTree repo
and ref. It performs a depsolve and downloads the extensions to a
provided output directory.

This is intended to replace cosa's `download-extensions`:
https://github.com/coreos/coreos-assembler/blob/master/src/download-extensions

The input YAML schema matches the one accepted by that script.

Some differences from the script:
- We have a guaranteed depsolve match and thus can avoid silly issues
  we've hit in RHCOS (like downloading the wrong `libprotobuf` for
  `usbguard` -- rhbz#1889694).
- We seamlessly re-use the same repos defined in the treefile, whereas
  the cosa script uses `reposdir=$dir` which doesn't have the same
  semantics (repo enablement is in that case purely based on the
  `enabled` flag in those repos, which may be different than what the
  rpm-ostree compose ran with).
- We perform more sanity-checks against the requested extensions, such
  as whether the extension is already in the base.
- We support no-change detection via a state SHA512 file for better
  integration in cosa and pipelines.
- We support a `match-base-evr` key, which forces the extension to have
  the same EVR as the one from a base package: this is helpful in the
  case of extensions which complement a base package, esp. those which
  may not have strong enough reldeps to enforce matching EVRs by
  depsolve alone (`kernel-headers` is an example of this).
- We don't try to organize the RPMs into separate directories by
  extension because IMO it's not at the right level. Instead, we should
  work towards higher-level metadata to represent extensions (see
  openshift/os#409 which is related to this).

Closes: #2055
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

app: Add rpm-ostree compose extensions jlebon/rpm-ostree
4 participants
@ashcrow @cgwalters @Prashanth684 @jlebon