AWS meeting 2023 01 11

EESSI/AWS sync meeting (2023-01-11)

Attending

• Brendan Bouffler (AWS)
• Matt Vaughn (AWS)
• Angel Pizarro (AWS)
• Kenneth Hoste (HPC-UGent - EESSI)
• Alan O'Cais (CECAM - EESSI)
• Henk-Jan Zilverberg (Univ. of Groningen, NL - EESSI)

Notes

• project proposal doc @ https://docs.google.com/document/d/1CHG9fCh2LkfJ-EI8J-_Wr5NpHL5iwm8Wu6syfK9h7-c/edit#heading=h.r7nb1sbhbecm

  • $3.5k may not be ambitious enough, based on what we know now
    • if we would have to make estimates today, they would probably be higher (partially due to having some tooling in place now, like build-and-deploy bot)
  • questions by AWS
    • benefit for AWS
    • can we find an AWS customer that can give feedback on this
    • use of CVMFS
      • EC2 + EBS is "bad architecture" / expensive
      • other "cloud distribution mechanisms" like CloudFront and S3 should be looked into it
      • S3 as backend is already supported by CVMFS, see https://cvmfs.readthedocs.io/en/stable/cpt-repo.html#sct-s3storagesetup
    • community engagement @ ISC/SC?
      • BoF session @ ISC'23
      • EESSI is a (small) part of EasyBuild tutorial @ ISC'22 (also submitted for ISC'23)
      • could fit in Getting Scientific Software Installed BoF sessions (cfr. previous ISC/SC years)
    • how to measure success
      • supported software
      • how many customers are using it
        • this is difficult to measure, due to serving via HTTP by CVMFS (no detailed stats on # users, etc) - best proxy is maybe outgoing bandwidth of CVMFS mirror servers
      • another proxy: support tickets
      • usage could be extracted from S3 logs
      • productivity of build infrastructure - how much stuff was built via AWS vs some other way
    • integration with ParallelCluster
      • getting access to EESSI is basically just a matter of installing+configuring CVMFS
      • see script at https://github.com/EESSI/eessi-demo/blob/main/scripts/install_cvmfs_x86_64_Ubuntu.sh

• urgent need for additional sponsored AWS credits

  • currently only $2.3k left, just enough for Jan'23 with current activities
    • currently burning (only) $1k-1.5k/month
    • we would like to be able to increase that significantly now...
    • see also https://github.com/EESSI/meetings/blob/main/meetings/EESSI_meeting_20230105.pdf (slide )
    • => let's start with quarter of estimates required credits (3x $3.5k)
  • dedicated manpower available now since 1 Jan 2023 thanks to start of EuroHPC Centre-of-Excellence project "MultiXscale"
  • build-and-deploy bot (https://github.com/EESSI/eessi-bot-software-layer) for EESSI is now capable enough to start using it on large scale (build new software layer for next EESSI pilot version)

• interest in HPC Tech Shorts episode on EESSI @ AWS?

  • demo of how quickly we can get GROMACS running on a fresh VM (Linux distro doesn't matter)
  • GPU support not in place yet, but in the works
  • MPI support requires libfabric work (cfr. project doc)
    • need to know the questions to ask them
    • see Alan's issue https://github.com/ofiwg/libfabric/issues/7790

• action points (Alan/Kenneth):

  • flesh out list of success measures in shared docs
  • pass EESSI account ID to Brendan for additional credits
  • technical questions w.r.t. libfabric