Skip to content

meeting June 4th 2020

Jump to bottom
Kenneth Hoste edited this page Jul 2, 2020 · 5 revisions
  • date & time: June 4th 2020 - 3pm CEST
  • venue: (online)
  • agenda:
    1. Short introduction by new partners:
      • Alan O’Cais (JSC)
      • Thomas Röblitz (UiO)
      • Oriol Mula (HPCNow!)
    2. CVMFS update by RUG & SURFsara [Bob, Henk-Jan, Caspar, Marco]
    3. Steps to move forward with EESSI [Kenneth]
    4. Talk at HPC Knowledge Event (HPCKP) [Kenneth]
    5. Update on role Dell HPC team [Jaco]
    6. Need for introduction on Git & GitHub? [Kenneth]
    7. Who does what the coming weeks?

Slides

Progress in last month

1. New partners

  • Alan (JSC), Thomas (UiO) & Bob (RUG) briefly introduced themselves
  • Thomas has some connections w.r.t. European projects (EOSC) & PRACE

2. Update on CVMFS layer

  • Bob (UG) has managed to automate the process to set up a CVMFS Stratum-0/Stratum-1/proxy with Ansible
    • see pull request at https://github.com/EESSI/cvmfs-layer/pull/1
    • no firewall setup so Stratum-0 can be accessed directly, which is not how it should be used
      • intention is to access via local squid proxy -> Stratum-1 -> Stratum-0
    • TODO:
      • Ansible playbook for configuration files
      • document procedure so others can try setting up Stratum-1/proxy
    • Caspar was asking on how to set this up securely to keep control on what is added to central Stratum-0 repo
      • we should check on how Compute Canada organizes this (involves rsync from build system to Stratum-0?)
  • Caspar (SurfSARA) is working through internal politics to try and get sysadmin help for CVMFS layer
    • management showed interest in the project, so hopeful

3. Steps to move forward

  • Kenneth outlined high-level "roadmap"
    • complete first version of documentation: scope, layers, ...
    • set up Gentoo Prefix in test CVMFS repository
    • start figuring out EasyBuild layer
      • EasyBuild configuration (installation prefix, RPATH, module naming scheme...)
      • figure out workflow
        • how to make proposals for adding software vs central EasyBuild easyconfigs repository?
        • process for installing software (build system -> CVMFS Stratum 0), via rsync?
      • set goals for pilot setup: OpenFOAM, TensorFlow + bioinformatics pipeline?
      • automation & testing (using ReFrame?)

4. HPCKP presentation

  • see https://hpckp.org
  • Thu June 18th 2020 – 2.30pm CEST
  • talk proposal accepted (45min incl. Q&A)
  • title: European Environment for Scientific Software Installations (EESSI)
  • speaker: Kenneth Hoste (HPC-UGent)
  • presentation needs to be pre-recorded!
  • abstract:
What if there was a way to avoid having to install a broad range of scientific software from scratch
on every HPC cluster or cloud instance you use or maintain, without compromising on performance?

The European Environment for Scientific Software Installations (EESSI, pronounced as "easy") is a
brand new collaboration between different European HPC sites & industry partners, with the common goal
to set up a shared repository of scientific software installations that can be used on a variety of systems,
regardless of which flavor/version of Linux distribution or processor architecture is used, or whether it's
a full size HPC cluster, a cloud environment or a personal workstation.

The concept is heavily inspired by the Compute Canada software stack, which was presented at PEARC'19
under the title "Providing a Unified Software Environment for Canada’s National Advanced Computing Centers".

It consists of three layers:

* a distributed filesystem layer leveraging the established CernVM-FS technology (https://cernvm.cern.ch/portal/filesystem);
* a compatibility layer using Gentoo Prefix (https://wiki.gentoo.org/wiki/Project:Prefix) to install a limited set of "system" packages;
* a software layer hosting scientific software installations that were built for different processor architectures,
  where archspec (https://github.com/archspec/archspec), EasyBuild (https://easybuilders.github.io/easybuild)
  and Lmod (https://github.com/TACC/Lmod) are leveraged;

In this talk, we will present how the EESSI project grew out of a need for more collaboration to tackle the 
challenges in the changing landscape of scientific software and HPC system architectures. The project structure 
will be explained in more detail, covering the motivation for the layered approach and the choice of tools,
as well as the lessons learned from the Compute Canada approach. Finally, we will outline the goals we have in 
mind and how we plan to achieve them going forward.
  • topics to cover in presentation:
    • provides opportunity to work together with scientific software developers to get their software properly installed across European HPC sites;
    • uniform environment for scientists, to simplify hopping between systems & sites;
    • broad pool of expertise to leverage in European HPC community;

5. Getting Dell HPC involved

  • Jaco (Dell) proposed to add people from Dell HPC to the projects
  • European people expressed interest, so did Dell team at Texas
  • for now, only add European people
  • try to recruit people who could actively help getting things off the ground
  • Dell HPC could help code optimization & with leveraging the software stack for benchmarking purposes

6. Need for introduction to Git & GitHub?

  • several people expressed interest
  • Kenneth will set up a doodle to plan a training early Sept'20

7. Who does what in coming weeks?

CVMFS layer

  • Bob (UG):
    • document procedure to set up Stratum-0/Stratum-1/proxy/mount
    • Ansible playbook for configuration
  • Thomas (UiO) & Patrick (TUE): test documented procedures
  • Henk-Jan (UG): look into secure workflow for adding software to the master repository on Stratum-0

Compatibility layer

  • Peter (VUA) & Bob (RUG) & Patrick (TUE): setup of Gentoo Prefix

Software layer

  • Kenneth (HPC-UGent), Caspar (SurfSARA), Alan (JSC)
    • filesystem structure in /cvmfs/
    • EasyBuild configuration
    • keeping track of which software to install
    • own easyconfigs repository vs leveraging central easyconfigs repository
    • workflow for proposing software to be added to repository
    • automation & testing

Docs

Website

Next meeting

  • Thu July 2nd 2020, 3pm CEST
Clone this wiki locally