-
Notifications
You must be signed in to change notification settings - Fork 0
Sync meeting on EESSI compat layer (2023 03 07)
Kenneth Hoste edited this page Apr 6, 2023
·
1 revision
- Alexander, Bart, Kenneth, Thomas, Killian, Jörg, Alex, Guilherme, Bob
- Approach at The Alliance
- mostly done by Bart, with some helps with colleagues
- their compat layer is a superset of what we include in EESSI compat layer
- some changes are made for next compat layer based on lessons learned
- Canadian compat layer is "frozen in time" (currently March 2020)
- using custom profile (incl. in overlay)
- Collaborating
- Have some more people involved in the setup and maintenance of the compatibility layers
- Collaborate with the Alliance and Gentoo
- Collaborate on maintaining ebuilds?
- ebuilds used by The Alliance sometimes need to backported to older EAPI (copied from Gentoo upstream)
- often as simple as just changing EAPI=8 to EAPI=7
- another way could be to only update Portage without doing a full update
- maybe better to push our ebuilds back upstream to Gentoo
- especially for stuff that's not patched or in any way specific to EESSI
- can be done by pull requests to Gentoo mirror in GitHub
- may need to ping correct person if there's no response
- for Prefix-related stuff, can tag Guilherme or Sam
- can consider going through process to become a Gentoo developer
- requires reading Gentoo dev manual + filling out a detailed quiz
- see https://www.gentoo.org/get-involved/become-developer
- ebuilds used by The Alliance sometimes need to backported to older EAPI (copied from Gentoo upstream)
- Regular call (e.g. monthly?) to discuss open issues?
- 1st Tuesday of the month, 13:00 CE(S)T
- Try to keep overview of contributions/bugs/PRs related to EESSI + The Alliance?
- maybe through an EESSI label in Gentoo Bugzilla => Guilherme?
- PRs in GitHub can be linked to Gentoo bugs, see https://wiki.gentoo.org/wiki/Gentoo_git_workflow
- example https://bugs.gentoo.org/895240
- Status update on new EESSI compat layer version (2023.03)
- x86_64: works (thanks to Bart's PR https://github.com/gentoo/prefix/pull/22)
- aarch64: blocked by missing profile (cfr. https://bugs.gentoo.org/892876)
- Bob has made some good progress on this, mostly working now
- was not a problem when bootstrapping Prefix for EESSI pilot 2021.12
- seems to have been caused by recent changes related to cross-compiling
- ppc64le: will try to build, include if it works, skip for EESSI pilot 2023.03 if it doesn't work
- riscv64: Prefix bootstrap doesn't work (anymore)
- cfr. https://bugs.gentoo.org/890363 (easy workaround) + https://bugs.gentoo.org/890636
- Bart: open problem that is silently being ignored (will open Gentoo bug for this),
one more issue to fix in the bootstrap/glibc:
/home/oldeman/eessi3/usr/sbin/locale-gen: line 168: /home/oldeman/eessi3usr/bin/localedef: No such file or directory
- CI for / better testing of Prefix
- Also see: https://wiki.gentoo.org/wiki/Google_Summer_of_Code/2023/Ideas#Better_testing_infrastructure_for_Gentoo_Prefix
- Use the EESSI build bot to do daily/weekly tests?
- On which OSs + CPU archs?
- x86_64 / aarch64
- risc64 would also be interesting
- OS
- old/new Debian/RHEL based
- something new: latest Fedora + latest Ubuntu
- something old: CentOS 7 + oldest LTS Debian
- also in Gentoo?
- what about macOS/Windows?
- for Windows, WSL is probably best bet
- for macOS, Fabian Groffen is still maintaining it (also Benda helps)
- x86_64 / aarch64
- On which OSs + CPU archs?
- Alexander Neuwirth / APN-Pucky (who's on the EESSI slack) also was/is working some kind of CI? See: https://github.com/APN-Pucky/gentoo-prefix-tester
- we could leverage the sponsored AWS/Azure credits that are available for EESSI, via custom runners
- What should go into the compatibility layer, and what should not?
- Editors, debuggers, tools like htop, or even desktop environments, etc?
- Balance between thin compat layer (makes security updates less painful) and fat compat layer (batteries included)
- Should we remove build dependencies/tools after the bootstrap to prevent issues/clashes with the same tools from the software layer and to make it more clear for users? CMake, Ninja, Meson (https://github.com/EESSI/compatibility-layer/issues/139), maybe even pip?
- Already doing this for Go (build dependency of
direnv
) - Also removed ReFrame from the compat layer for similar reasons (see https://github.com/EESSI/software-layer/issues/127)
- removing build deps can be done using "
emerge -c
"
- Already doing this for Go (build dependency of
- Alex: maybe there should be a separate compat layer for building (fatter) vs user-facing (stripped down)
- could even remove Python from compat layer used to support software layer
- but EasyBuild (incl. in software layer) requires Python...
- The Alliance relies on compat layer to provide a bunch of things that may not be provided by the OS (especially in diskless systems with very limited OS)
- incl. MATE desktop environment, VNC, etc.
- EESSI should write out clear policy on what is incluced in EESSI
- also clearly state what the goal is of our compat layer
- currently, mainly there to support whatever is included in software layer
- can be revisited later, for example to also have a desktop environment in there like The Alliance does
- Security updates
- How to apply them? Could we automate this a bit more?
- Current approach:
- check out recent commit from gentoo Git repo
- install specific versions of packages for which there are security advisories
- see for example https://github.com/EESSI/compatibility-layer/blob/main/scripts/update-pkgs-EESSI-2021.12_2023-01-31.sh
- Best way to deal with host dependencies
- Bart has been working on this, see EESSI Slack and https://github.com/EESSI/compatibility-layer/issues/158
- motivated by use of SSSD in Canadian clusters, requires NSS
- see https://wiki.gentoo.org/wiki/Project:Prefix/FAQ#Username_becomes_invalid_inside_Prefix
- building libnss_ldap.so.2 from source to make sure it's compatible with compat layer, see https://github.com/EESSI/compatibility-layer/issues/158
- mainly needed to avoid trouble with libnss_ldap.so.2 from host that is built on top of a glibc that is newer than what we include in compat layer
- could avoid that glibc included in compat layer to pick up on host /etc/passwd and /etc/group
- Ideas for EESSI
- also start using our own profile, like The Alliance does
- Notes
- Bart is working on a hack related to /var (?)
- may be difficult to push back upstream
- Bart is working on a hack related to /var (?)