Add first batch of alma8 CDTs #70

h-vetinari · 2024-09-17T19:37:12Z

This tackles the first batch of CDTs from #66, more specifically those currently marked ✅ here. Additionally, I've added:

The -devel packages from the dependencies we've added for pam; IMO it makes sense to always have libfoo and libfoo-devel for a given CDT, otherwise libfoo is going to be non-functional (unless as a dependency for another CDT)
mesa-libgbm because it was the only mesa-* package not to receive an ❌; I had asked if that was intentional, Isuru had answered in the affirmative (but only in a core call, not as a comment in the issue)
kmod-devel & mesa-libgbm-devel for the same reason; since we alread have kmod & mesa-libgbm

Doing this turned out to be a good exercise in the sense that the alma8 builds are separated across various subfolders that may depend on each other, which needed some updates to the logic.

Furthermore, I've switched away from skipped_cdts in favour of an allowlist-approach, as the impression I got from #66 was that people want to keep the list absolutely minimal, and it's easier to keep track of that using a positive list, rather than ensuring that we captured all exclusions correctly everywhere.

Finally, I repeatedly got permission errors (HTTP 403) when trying to urlopen https://vault.almalinux.org/8.9/BaseOS/{base_architecture}/os/repodata/repomd.xml, so I ended up switching to requests which doesn't have a problem there.

Given the large mechanical changes, it's probably easiest to review per commit.

Please see the repo readme for directions on how to make PRs on this repo.

Checklist:

if you have added a CDT, it appears in the cdt_slugs.yaml file and
you have rerun the script python gen_cdt_recipes.py.
if you have changed the CDT generator code (rpm.py), you have bumped
the build number in conda_build_config.yaml and have remade all of the
recipes via running python gen_cdt_recipes.py --force
if you have added a custom CDT recipe, you have added the name of the CDT
with custom: true in the cdt_slugs.yaml file.
all CDT recipes have build number set by {{ cdt_build_number }} for
old-style/legacy CDTs or {{ cdt_build_number|int + 1000 }} for new-style CDTs
if you see a warning about a CDT not having a license, you have added the
license_file key in the cdt_slugs.yaml file with the path to the appropriate
license in licenses/

NOTE: If you make any changes to cd_slugs.yaml, you need to reun the generator code
via python gen_cdt_recipes.py.

packages from different subfolders can depend on each other

cdts/pam-devel-alma8-x86_64/meta.yaml

beckermr

Let's put the commits of the new CDT recipes into another PR. I cannot review the new logic commit by commit.

h-vetinari · 2024-09-17T20:50:11Z

Let's put the commits of the new CDT recipes into another PR.

It's hard to validate the logic without actually rendering & building things... But if you prefer, I'll revert this at the end once it's green (or rebase it out) for ease of review.

I cannot review the new logic commit by commit.

Aside from the one rerender commit, the diffs are tiny, and GH makes it easy to click through (prev/next)...?

beckermr · 2024-09-17T20:55:19Z

I'm not currently figuring out how to do it for the commits but ok. I'll do my best.

beckermr

The yaml file had support for other things that had been important:

excluding certain deps (e.g., kernel-headers)
putting in custom logic in the build scripts for certain CDTs

When you added alma8 with the allow list, you didn't implement those features for alma8. How do you know we won't need them? Why not stick to the yaml so we can keep them?

beckermr · 2024-09-17T21:03:22Z

One concrete way to do this would be to have a separate cdt slugs file for alma8 and cos7. Then each one is an allow list on its own and we can keep the extra hooks if we need them.

h-vetinari · 2024-09-17T21:03:28Z

The yaml file had support for other things that had been important:

None of the logic except skipped_cdts is changed. Removing/replacing deps & custom logic is still possible

Why not stick to the yaml so we can keep them?

I haven't changed away from the yaml.

beckermr · 2024-09-17T21:05:01Z

So I have to list an alma8 CDT in both the yaml AND the allow list in the python file?

beckermr · 2024-09-17T21:05:45Z

Maybe the better question is, can you show me how to add a skipped dependency for an alma8 CDT?

h-vetinari · 2024-09-17T21:06:49Z

One concrete way to do this would be to have a separate cdt slugs file for alma8 and cos7.

So, with the caveat that all of this is still a draft, I'd prefer not to split the cdt slugs, because for all the ones we're interested (so far), the config is exactly the same as for cos7, so duplicating them would be kinda gratuitous.

FWIW, I had started with applying the skipped_cdts approach, and it just ends up with

  skipped_cdts:
    - alma8-x86_64
    - alma8-aarch64
    - alma8-ppc64le

for almost all slugs.

h-vetinari · 2024-09-17T21:08:15Z

So I have to list an alma8 CDT in both the yaml AND the allow list in the python file?

If it's an entirely new CDT then yes. If it's a CDT that's already on cos7, then only the allowlist. Again, please consider this a suggestion (it seemed more natural given how much people want to constrain CDT creation), not as something set in stone.

cdt_slugs.yaml

beckermr · 2024-09-17T21:32:37Z

OK thanks for answering my questions @h-vetinari!

Maybe we can refactor the cdt_slugs.yaml file to look like this:

allowed_cdts:
  alma8:
    - foo
    - bar
    - blah
  cos7:
    - foo
    - bar
    - blah
    - glarg
cdt_build_defs:
  # any CDT not listed here is equivalent to the default
  #
  #   <cdt name>:
  #     custom: false
  #
  foo:
    license_file: licenses/alsa-lib-license
  bar:
    dep_remove:
      - foobarg

This way all of the configuration is still in one spot and we only have to list things that are special/different.

h-vetinari · 2024-09-17T21:41:46Z

Maybe we can refactor the cdt_slugs.yaml file to look like this:

We can. Though it'll bust the indentation of everything in that file to introduce a new level (cdt_build_defs). 🤷

beckermr · 2024-09-17T21:45:40Z

Yeah I am ok with the indentation change.

h-vetinari · 2024-09-17T21:49:31Z

I'm not currently figuring out how to do it for the commits but ok. I'll do my best.

FWIW, there's a commit tab on top (between "Conversation" and "Checks"); if you click on the first commit therein, you'll get a prev/next navigation menu in the upper RH corner. Aside from the rerender commits (where you can just click next without review), the others are very small and should be easy to digest.

Yeah I am ok with the indentation change.

I might not finish this today, but it shouldn't be hard conceptually; for now I made the change that Isuru asked for, and it's getting late here.

beckermr · 2024-09-17T21:50:56Z

No rush on this and thanks for the dev work @h-vetinari!

h-vetinari

Another infrastructure question

cdt_slugs.yaml

h-vetinari · 2024-09-18T10:17:43Z

This way all of the configuration is still in one spot and we only have to list things that are special/different.

From a quick check, only 30 CDTs out of 180 would actually fall under

  # any CDT not listed here is equivalent to the default
  #
  #   <cdt name>:
  #     custom: false

It's a bit hard to argue that 83% are the special case IMO 😅

The vast majority of those only specifies the license_file, but unless we have a way to revamp the license handling too, I'm not sure this would be an improvement. It'd just be confusing look for a non-existent build definition of a CDT that you're interested in (e.g. often libfoo-devel needs a license override, but libfoo doesn't), only to realise you've fallen into the fallback case.

beckermr · 2024-09-18T11:05:08Z

We can list them all then, that's fine by me. What doesn't work is having to add a CDT name in two places, once in a yaml file and once in the python code, in order to add a new CDT. That is confusing for sure.

this also clashes with the dirname-function imported from `os.path`

ignore whitespace in git to see that this changes nothing, e.g. `git show -b <hash_of_this_commit>`, or add `?w=1` to the GH url

h-vetinari · 2024-09-18T16:56:17Z

@beckermr, I implemented your idea (minus the default fallback, as discussed), and a run of python gen_cdt_recipes.py --force yields no further changes (as desired). I added some basic validation (tested to work locally) to fail early in case of missing build_defs, or warn in case of superfluous ones.

Feel free to play with this branch according to your usual workflows, happy to fix. :)

beckermr

LGTM! Thank you!

isuruf · 2024-09-18T17:43:53Z

Why is there a java CDT? Is it a transitive dependency?

h-vetinari · 2024-09-18T18:10:07Z

Why is there a java CDT? Is it a transitive dependency?

I didn't touch those, they're pre-existing (also present in the list in #66 since last Nov.). I looked for current users and even checked now if they were used in any of the non-java packages we dropped in 779da0b, but can't find any users in the CDTs themselves.

Looking around conda-forge, it's used, but only in r-base. Once you remove that, we can consider dropping them here.

isuruf · 2024-09-18T18:13:56Z

Ah, it's just a whitespace change.

jakirkham

Thanks all for getting this going! 🙏

Had a question below about naming/versioning

jakirkham · 2024-10-15T18:40:58Z

build_cdt_recipes.py

+    Build all CDT recipes for a given DIST_ARCH_SLUG (e.g. conda-x86_64
+    for post-CentOS CDTs, or cos6-x86_64, cos7-aarch64, etc. historically)


Noticing that the new slugs don't have a version. Did we want to bake one in like we did with cos6 & cos7? For example conda-2_28-<arch>

Yes, that's intended. See your comment on https://github.com/conda-forge/conda-forge.github.io/blob/5e57e30a11e91dc127aab1543b2cffe8ed8b3290/community/minutes/2024-05-15.md?plain=1#L74-L77

h-vetinari added 14 commits September 17, 2024 21:36

switch order of (dist,arch) tuples, define only once

3ab6e95

add alma8

d8be351

add devel-packages for dependencies of pam

8c5dfba

add {kmod,mesa-libgbm}-devel companion packages

eecefb9

replace skipped_cdts mechanism with allowlist for newer distros

58dc551

switch to reading repomd with requests

a59e5a6

teach infra to handle separate distro channels

1e552c5

unify package information across subfolders

51256bb

packages from different subfolders can depend on each other

rerender cdts for alma8

cbd418f

add CI jobs for alma8 builds

2808d69

add kmod-libs; dependency of kmod-devel

74e8659

don't rely on kernel-headers for audit-libs-devel

f5c3581

fix dep_remove logic for alma

7550d6f

rerender cdts

f14a61b

isuruf reviewed Sep 17, 2024

View reviewed changes

cdts/pam-devel-alma8-x86_64/meta.yaml Outdated Show resolved Hide resolved

beckermr requested changes Sep 17, 2024

View reviewed changes

h-vetinari commented Sep 17, 2024

View reviewed changes

cdt_slugs.yaml Outdated Show resolved Hide resolved

h-vetinari force-pushed the alma branch 4 times, most recently from 7d23d2c to 5014799 Compare September 17, 2024 23:27

h-vetinari commented Sep 18, 2024

View reviewed changes

cdt_slugs.yaml Outdated Show resolved Hide resolved

h-vetinari added 3 commits September 18, 2024 18:06

switch cdt names to use conda instead of dist-name

49ed06e

remove unused dirname in distro-config

8db75a9

this also clashes with the dirname-function imported from `os.path`

rerender cdts

e0e9985

h-vetinari force-pushed the alma branch from 5014799 to e0e9985 Compare September 18, 2024 16:06

h-vetinari added 2 commits September 18, 2024 18:46

increase indentation in cdt_slugs.yaml

e27d70f

ignore whitespace in git to see that this changes nothing, e.g. `git show -b <hash_of_this_commit>`, or add `?w=1` to the GH url

move CDT allowlist to cdt_slugs.yaml

4982cf9

beckermr approved these changes Sep 18, 2024

View reviewed changes

h-vetinari merged commit d020455 into conda-forge:main Sep 18, 2024
7 checks passed

h-vetinari deleted the alma branch September 18, 2024 18:18

h-vetinari mentioned this pull request Sep 18, 2024

figure out which CDTs we can drop for cos8, maybe drop them all? #66

Closed

jakirkham reviewed Oct 15, 2024

View reviewed changes

h-vetinari mentioned this pull request Oct 16, 2024

[RFC]: Add os_version option to use alma8 conda-forge/conda-forge-pinning-feedstock#6548

Closed

5 tasks

h-vetinari mentioned this pull request Oct 25, 2024

roll-out of alma8-based infrastructure (aka centos 8) conda-forge/conda-forge.github.io#1941

Closed

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add first batch of alma8 CDTs #70

Add first batch of alma8 CDTs #70

h-vetinari commented Sep 17, 2024

beckermr left a comment

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

beckermr left a comment

beckermr commented Sep 17, 2024

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

beckermr commented Sep 17, 2024

h-vetinari commented Sep 17, 2024 •

edited

Loading

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

h-vetinari left a comment

h-vetinari commented Sep 18, 2024

beckermr commented Sep 18, 2024

h-vetinari commented Sep 18, 2024

beckermr left a comment

isuruf commented Sep 18, 2024

h-vetinari commented Sep 18, 2024

isuruf commented Sep 18, 2024

jakirkham left a comment

jakirkham Oct 15, 2024

isuruf Oct 15, 2024 •

edited

Loading

		Build all CDT recipes for a given DIST_ARCH_SLUG (e.g. conda-x86_64
		for post-CentOS CDTs, or cos6-x86_64, cos7-aarch64, etc. historically)

Add first batch of alma8 CDTs #70

Add first batch of alma8 CDTs #70

Conversation

h-vetinari commented Sep 17, 2024

beckermr left a comment

Choose a reason for hiding this comment

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

beckermr left a comment

Choose a reason for hiding this comment

beckermr commented Sep 17, 2024

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

beckermr commented Sep 17, 2024

h-vetinari commented Sep 17, 2024 • edited Loading

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

h-vetinari commented Sep 17, 2024

beckermr commented Sep 17, 2024

h-vetinari left a comment

Choose a reason for hiding this comment

h-vetinari commented Sep 18, 2024

beckermr commented Sep 18, 2024

h-vetinari commented Sep 18, 2024

beckermr left a comment

Choose a reason for hiding this comment

isuruf commented Sep 18, 2024

h-vetinari commented Sep 18, 2024

isuruf commented Sep 18, 2024

jakirkham left a comment

Choose a reason for hiding this comment

jakirkham Oct 15, 2024

Choose a reason for hiding this comment

isuruf Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

h-vetinari commented Sep 17, 2024 •

edited

Loading

isuruf Oct 15, 2024 •

edited

Loading