Skip to content

dev.eessi.io discussion (2024 05 24)

Jump to bottom
Kenneth Hoste edited this page May 24, 2024 · 2 revisions

dev.eessi.io discussion (20240524)

Attendees

  • Xin An (SURF)
  • Kenneth Hoste (HPC-UGent)
  • Lara Peeters (HPC-UGent)
  • Pedro Santos Neves (RUG)
  • Alan O'Cais (CECAM, U Barcelona)
  • Bob Dröge (RUG)
  • Davide Grassano (CECAM)
  • Jean-Noël Grad (U Stuttgart)
  • Matteo Zanfrognini (Leonardo)
  • Petra Papež (NIC)
  • Richard Topouchian (Univ. of Bergen)
  • Thomas Röblitz (Univ. of Bergen) :-)
  • Tilen Potisk (NIC)

Slides

Meeting notes

  • scripts used by the bot should be under control of us (who manage/run the bot)
    • scripts used by bot could live in dev.eessi.io GitHub repo
    • builds could be triggered by repo managed by developers
      • by "registering" a commit somewhere, or providing an easyconfig file
  • input from developers
    • Xin (for Hassan)
      • build & test against different architectures
      • running many jobs, would like to see this automated
      • run with different memory configurations
      • last two goes beyond deployment of builds into dev.eessi.io...
      • what would the input look like here?
    • Jean-Noël
      • currently doing builds through Fedora through Koji (aarch64, x86_64, power)
        • used for every release
        • also when doing major changes in source code, in Docker container using Fedora
          • using Kerberos ticket which gives access to Koji build cluster for 24h
        • would be helpful to give projects that rely on Espresso access to pre-release builds after major changes were made
        • ideally available for 6 months
        • to help people run Espresso from CI
      • scheduled builds at regular intervals (1/2 months)
      • using pre-release builds in PRs through GitHub Actions
      • input from developers
        • commit ID
        • list of CPU targets to build for
        • deploy: yes or no
      • current approach to testing
        • use some dependencies through modules (compiler, MPI, Python)
        • self-build some dependencies (Boost, FFTW)
        • manually build waLBerla + Espresso on top
      • trying to get shared directory to provide shared builds of Espresso
        • at various systems at Stuttgart, Heidelberg, etc.
        • EESSI is not available system-wide yet
        • containers via Singularity is an option, but troublesome with MPI runs
        • maybe cvmfsexec can be an a solution?
  • Thomas: what are our goals?
    • make life of developers easier (for this we should collect the pain points of developers)
      • getting rid of dependency hell, or easily allow playing with different (pre-release) versions of dependencies, compilers, ...
      • providing access to pre-release builds
    • building for specific CPU microarchitectures (maybe better: building ON specific CPU micro...)
    • making pre-release builds available through dev.eessi.io repo
      • so people can access them easily in GitHub Actions, on EuroHPC systems, etc.
      • allow people to do scaling tests
    • save compute credits on EuroHPC systems for testing, not building the software
  • Lara: use case for Tilen w/ LAMMPS is a bit different
    • input there is basically a patch file for LAMMPS
    • sensible approach here could be to provide easyconfig file as input
    • Tilen's code is in private Git repository
      • consists of code that should be copied into LAMMPS + changes to CMake file
      • bot would need to be able to clone/check out specific commits, so SSH pubkey of bot would need to added to private repo
  • how to control who has permission to trigger builds/deploys
    • private repository that holds bot configuration
    • developers can sent PRs to make changes
  • should there be a separate CernVM-FS setup to host dev.eessi.io
    • new Stratum-0, etc. (implies new pubkeys)
    • seems like overkill + a lot of extra work
    • would make adoption of dev.eessi.io more difficult
  • next steps
    • figure out who will work on this
    • come with up with a step-wise plan
      • set up dedicated Slurm cluster for this using Magic Castle [Kenneth/Alan]
      • set up dev.eessi.io GitHub repo (private for now?) [???]
      • dedicated S3 bucket for ingestion into dev.eessi.io [Bob?]
      • implement necessary scripts [Pedro?]
        • generate easyconfig file based on specific commit
        • bot/build.sh, etc. scripts
    • initial use case: build for specific commit of Espresso (through dev.eessi.io branch in Espresso GitHub repo)
Clone this wiki locally