Release v1.40.0

[tpdownes]

What's Changed

Important

All HPC VM images based upon CentOS 7 have been deprecated. This means that
referring to the "hpc-centos-7" family in the "cloud-hpc-image-public"
project will fail. We recommend migrating to the "hpc-rocky-linux-8" family
that is the new default throughout the Toolkit. If CentOS 7 is truly needed,
the final HPC CentOS 7 image can be used by its name: "hpc-centos-7-v20240712".

Key New Features 🎉

• GKE A3 High blueprint and GKE A3 Mega blueprint with aautomated GPU networking performance enhancements
• Add enable-maintenance-reservation flag in slurm to control reservation for scheduled maintenance by @harshthakkar01 in Add enable-maintenance-reservation flag in slurm to control reservation for scheduled maintenance #2987
• adding documentation for versioned blueprint feature by @RachaelSTamakloe in adding documentation for versioned blueprint feature #3055
• adding unit test for version blueprint caching mechanism by @RachaelSTamakloe in adding unit test for version blueprint caching mechanism #3052

New Modules 🧱

• implement kubectl-apply module by @sharabiani in implement kubectl-apply module #2980

Module Improvements 🔨

• Default to zonal bulkInsert by @mr0re1 in Default to zonal bulkInsert #3005
• Add machine type availability checks by @annuay-google in Add machine type availability checks #3003
• add support for enabling tcpx/o in a3 and a3mega vm, provide script for injecting rxdm sidecar and other required components into user workload by @chengcongdu in add support for enabling tcpx/o in a3 and a3mega vm, provide script for injecting rxdm sidecar and other required components into user workload #3012
• support ghpc_stage function in kubectl-apply module by @sharabiani in support ghpc_stage function in kubectl-apply module #3036
• Validate Reservations in GKE Blueprints by @arajmane-g in Validate Reservations in GKE Blueprints #3024
• Fix multivpc missing region by @wiktorn in Fix multivpc missing region #3046
• Add initial_node_count support to gke-node-pool by @sharabiani in Add initial_node_count support to gke-node-pool #3068

Improvements 🛠

• Update gVNIC driver in A3 Mega solution by @tpdownes in Update gVNIC driver in A3 Mega solution #2957
• Implement udev-based approach to mounting aperture devices by @tpdownes in Implement udev-based approach to mounting aperture devices #2955
• Update Debian 12 image in A3 Mega solution by @tpdownes in Update Debian 12 image in A3 Mega solution #2958
• adding module cache to prevent repeated module downloads during modul… by @RachaelSTamakloe in adding module cache to prevent repeated module downloads during modul… #3010
• add additional vpc validation for a3/a3mega machine by @chengcongdu in add additional vpc validation for a3/a3mega machine #3049
• Adds option to allow Kueue/Jobset to be installed on a GKE cluster via blueprints by @ankitkinra in Adds option to allow Kueue/Jobset to be installed on a GKE cluster via blueprints #3017
• update readme for gpudirect by @chengcongdu in update readme for gpudirect #3059

Deprecations 💤

• SlurmGCP V6. Remove CentOS7 image support. by @mr0re1 in SlurmGCP V6. Remove CentOS7 image support. #3038
• removing deprecated spack setup variables by @RachaelSTamakloe in removing deprecated spack setup variables #3040
• removing deprecated ramble setup variables by @RachaelSTamakloe in removing deprecated ramble setup variables #3041

Version Updates ⏫

• Update NeMo 23.11 to 24.07 by @akiki-liang0 in Update NeMo 23.11 to 24.07 #3090

Bug fixes 🐞

• Retry mounting daos container by @harshthakkar01 in Retry mounting daos container #3045
• add argparse dependency to cloud build by @chengcongdu in add argparse dependency to cloud build #3057
• Allow users to provide a commit hash instead of git tag for Spack and Ramble installations by @rohitramu in Allow users to provide a commit hash instead of git tag for Spack and Ramble installations #3073
• resolving error when var.initial_node_count is null by @RachaelSTamakloe in resolving error when var.initial_node_count is null #3081
• A3 High blueprint prolog solution updates by @tpdownes in A3 High blueprint prolog solution updates #3088

Other changes

• NeMo readme instructions for preloading gpt2 tokenizer by @koallison in NeMo readme instructions for preloading gpt2 tokenizer #3075

New Contributors

• @koallison made their first contribution in NeMo readme instructions for preloading gpt2 tokenizer #3075
• @akiki-liang0 made their first contribution in Update NeMo 23.11 to 24.07 #3090

Full Changelog: v1.39.0...v1.40.0

NOTES FOR REVIEWER

The observed integration test failures are being addressed on develop:

• Chrome Remote Desktop: increase resilience of apt operations #3093 addresses PR-test-hpc-slurm-chromedesktop-v5-legacy
  • the error is intermittent and likely affects all releases of the example; the fix on develop will propagate forward through our normal development process
• Move a3-megagpu-8g tests to us-west4-a #3086 addresses PR-test-ml-a3-megagpu-slurm
  • our internal capacity for testing moved; the error is not customer-facing

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #