Adding GROMACS #61

pearce8 · 2023-11-29T14:17:04Z

Working to add a new problem definition for GROMACS. Will upstream to Ramble at a later point.

repo/gromacs/application.py

pszi1ard · 2023-12-01T21:21:50Z

repo/gromacs/application.py

+    figure_of_merit('Nanosecs per day', log_file=log_str,
+                    fom_regex=r'Performance:\s+' +
+                              r'(?P<ns_per_day>[0-9]+\.[0-9]+).*',
+                    group_name='ns_per_day', units='ns/day')


Note that this is the only FOM that really matters.

@pszi1ard Do you think it would be less confusing if we remove the rest of the FOMs?

pszi1ard · 2023-12-01T21:23:56Z

experiments/gromacs/openmp/ramble.yaml

+            processes_per_node: ['8', '4']
+            n_nodes: ['1', '2']
+            threads_per_node_core: ['2', '4']
+            omp_num_threads: '{threads_per_node_core} * {n_nodes}'


Can you explain what this means? Isn't omp_num_threads here the same as OMP_NUM_THREADS, i.e. the threads per MPI rank?

@pszi1ard OMP_NUM_THREADS (set using the value of omp_num_threads) is the threads per MPI rank. For this particular experiment config we scale omp_num_threads with n_nodes - for 1 node we use 2 and 4 threads, while for 2 nodes we use 4 and 8 threads. In both cases the total number of MPI ranks is 8, but since the latter case uses 2 nodes, we can assign the additional cores to more omp threads. This can be changed per the experiment's requirements

OK, I see. That's a atypical way of scaling to multiple nodes in MD/GROMACS. I suggest to using a fixed OMP_NUM_THREADS and scale the number of ranks instead. On CPU-only systems typically small values of OMP_NUM_THREADS are best (1-3) until the case starts running out of parallelism (limiting domain decomposition) and increasing the OMP_NUM_THREADS can help to keep the number of ranks fixed while still using more resources.

On GPU-accelerated machines, the decomposition, hence the rank count is determined by the number of GPU devices (typically 1 or 2 ranks per device).

Co-authored-by: Szilárd Páll <pall.szilard@gmail.com>

pearce8 · 2023-12-18T23:50:07Z

experiments/gromacs/openmp/ramble.yaml

+        packages:
+        - lapack
+        - default-mpi
+        - fftw-omp


@rfhaque Can we use spack to build fftw instead?

pearce8 · 2024-02-16T04:13:08Z

configs/LLNL-Sierra-IBM-power9-V100-Infiniband/spack.yaml

@@ -6,12 +6,16 @@
 spack:
  packages:
    default-compiler:
-      spack_spec: xl@16.1.1-2022.08.19-cuda{default_cuda_version}
+      spack_spec: clang@16.0.6-cuda{default_cuda_version}


xl should be the default compiler on sierra. Please create compiler-clang to use for Gromacs if you must.

scheibelp

Comments starting with "minor" or "for a later PR" aren't important for this.

Let me know if you disagree with any of these or the requests seem unclear.

scheibelp · 2024-02-17T00:29:13Z

configs/LLNL-Pascal-Penguin-broadwell-P100-OmniPath/auxiliary_software_files/packages.yaml

@@ -32,8 +32,12 @@ packages:
  mvapich2:
    buildable: false
    externals:
-    - spec: mvapich2@2.3.7%intel@2021.6.0
+    - spec: mvapich2@2.3.7-gcc-11.2.1


I think you might mean

mvapich2@2.3.7%gcc@11.2.1

vs

mvapich2@2.3.7-gcc-11.2.1

in the former case, Spack will know that mvapich2 was compiled with gcc

scheibelp · 2024-02-17T00:32:13Z

configs/LLNL-Pascal-Penguin-broadwell-P100-OmniPath/auxiliary_software_files/packages.yaml

+    externals:
+    - spec: intel-oneapi-mkl@2022.1.0
+      prefix: /usr/tce/packages/mkl/mkl-2022.1.0
+    buildable: false


I think it will be cleaner to reorganize

blas: externals: - spec: intel-oneapi-mkl@2022.1.0 prefix: /usr/tce/packages/mkl/mkl-2022.1.0 buildable: false lapack: externals: - spec: intel-oneapi-mkl@2022.1.0 prefix: /usr/tce/packages/mkl/mkl-2022.1.0 buildable: false intel-oneapi-mkl: externals: - spec: intel-oneapi-mkl@2022.1.0 prefix: /usr/tce/packages/mkl/mkl-2022.1.0/ buildable: false

as

blas: buildable: false lapack: buildable: false intel-oneapi-mkl: externals: - spec: intel-oneapi-mkl@2022.1.0 prefix: /usr/tce/packages/mkl/mkl-2022.1.0/ buildable: false

intel-oneapi-mkl provides both blas and lapack, so this reduces redundancy.

scheibelp · 2024-02-17T00:35:35Z

configs/LLNL-Sierra-IBM-power9-V100-Infiniband/auxiliary_software_files/compilers.yaml

+    flags:
+      cflags: -g -O2
+      cxxflags: -g -O2 -std=c++17
+      fflags: ''


(minor) you can remove this line entirely if you want.

scheibelp · 2024-02-17T00:37:57Z

configs/LLNL-Sierra-IBM-power9-V100-Infiniband/auxiliary_software_files/compilers.yaml

@@ -65,3 +65,19 @@ compilers:
    modules: []


(for another PR) I noticed when looking above that there are 3 xl/cuda compiler combos - is that still necessary?

scheibelp · 2024-02-17T00:48:01Z

configs/LLNL-Sierra-IBM-power9-V100-Infiniband/auxiliary_software_files/packages.yaml

@@ -70,4 +75,8 @@ packages:
      prefix: /usr/tce/packages/spectrum-mpi/spectrum-mpi-rolling-release-xl-2022.08.19-cuda-10.1.243
      extra_attributes:
        ldflags: "-lmpiprofilesupport -lmpi_ibm_usempi -lmpi_ibm_mpifh -lmpi_ibm"
+    - spec: spectrum-mpi@2022.08.19-clang16.0.6-cuda-11.8.0


If you want a build of a package with clang to use this, then you'd want to make the compiler association explicit using % like spectrum-mpi@2022.08.19%clang16.0.6-cuda-11.8.0

(question, no change required) Do you know if it's strictly required to use/link an MPI compiled w/clang when building dependents with clang?

repo/gromacs/application.py

scheibelp · 2024-02-17T01:11:30Z

repo/gromacs/application.py

+
+
+class Gromacs(SpackApplication):
+    '''Define a Gromacs application'''


(minor) Are comments required? This doesn't add much by itself.

scheibelp · 2024-02-17T01:15:37Z

repo/gromacs/package.py

-            options.append("-DGMX_LAPACK_USER={0}".format(self.spec["lapack"].libs.joined(";")))
+        target = self.spec.target
+        if "+cuda" in self.spec and target.family == "ppc64le":
+            options.append("-DGMX_EXTERNAL_LAPACK:BOOL=OFF")


This is to disable the use of external lapack on a specific system if I understand: LLNL-Sierra-IBM-power9-V100-Infiniband/spack.yaml. I think it would be useful to add an explicit comment about that so it's easier to correct in the future.

scheibelp · 2024-02-17T01:16:08Z

repo/cusolver/package.py

+from spack.package import *
+
+
+class Cusolver(Package):


Does anything use this? Is it still needed?

scheibelp · 2024-02-17T01:22:31Z

configs/LLNL-Sierra-IBM-power9-V100-Infiniband/auxiliary_software_files/packages.yaml

+    externals:
+    - spec: fftw@3.3.10
+      prefix: /usr/tcetmp/packages/fftw/fftw-3.3.10-xl-2023.06.28
+    buildable: false
  lapack:


While this PR disables use of external blas/lapack, I think it would be worth reorganizing it:

blas: buildable: false lapack: buildable: false netlib-lapack: externals: - spec: netlib-lapack@3.6.0 prefix: /usr/tcetmp/packages/lapack/lapack-3.11.0-gcc-11.2.1/

(this config worked for me when using external blas/lapack and also avoids any potential issues with listing lapack-xl as an external, which isn't a Spack package).

pearce8 · 2024-02-20T07:10:07Z

configs/LLNL-Tioga-HPECray-zen3-MI250X-Slingshot/auxiliary_software_files/packages.yaml

@@ -193,6 +193,13 @@ packages:
      prefix: /opt/cray/pe/libsci/23.05.1.4/gnu/10.3/x86_64/
  lapack:
    buildable: false
+  hypre:


Please remove hypre from this list. All variants are already set to amdgpu_target=gfx90a

…ure-gromacs

pszi1ard · 2024-02-21T20:10:52Z

configs/LLNL-Tioga-HPECray-zen3-MI250X-Slingshot/auxiliary_software_files/compilers.yaml

@rfhaque you need rocmcc and you should compile both ACPP/hipsycl and GROMACS using that
ref pszi1ard@2398aba#diff-fd7d81e7edf97a63d331611675196dd0ab63b3ccc6f6dee144bd01eb36438fa2R113

* remove version locking from experiment config; achieve this with a mixin package * add needed externals, proper dependency reference for mixin * also align rocmcc version w/hip version

Create gromacs/application.py

acce1d6

pearce8 self-assigned this Nov 29, 2023

Add gromacs openmp config (#62)

720b176

pszi1ard reviewed Dec 1, 2023

View reviewed changes

pszi1ard mentioned this pull request Dec 1, 2023

Add gromacs openmp config #62

Merged

rfhaque and others added 5 commits December 1, 2023 14:24

Merge remote-tracking branch 'origin/develop' into feature-gromacs

6b70948

Update repo/gromacs/application.py

06c2b29

Co-authored-by: Szilárd Páll <pall.szilard@gmail.com>

Update repo/gromacs/application.py

0f98c77

Co-authored-by: Szilárd Páll <pall.szilard@gmail.com>

Merge branch 'develop' into feature-gromacs

68e7d28

Merge branch 'develop' into feature-gromacs

1679d15

pearce8 assigned scheibelp Dec 14, 2023

pearce8 added 2 commits December 18, 2023 17:48

copyright

f5f723a

copyright

e73ec42

pearce8 commented Dec 18, 2023

View reviewed changes

copyright

fa765a6

pearce8 added this to the v0.1.1 milestone Dec 19, 2023

pearce8 added feature New feature or request experiment New or modified experiment and removed feature New feature or request labels Dec 19, 2023

Merging develop

aea9180

github-actions bot added configs New or modified system config application New or modified application labels Jan 8, 2024

pearce8 added 3 commits January 15, 2024 18:47

Merge branch 'develop' into feature-gromacs

aeb966c

Merge branch 'develop' into feature-gromacs

05dc3b2

Merge branch 'develop' into feature-gromacs

c1e2112

pearce8 assigned rfhaque Jan 22, 2024

Riyaz Haque and others added 4 commits January 30, 2024 12:53

Draft ADAC experiment setup

992421a

Merge branch 'develop' into feature-gromacs

7eba856

Merge branch 'develop' into feature-gromacs

1762eea

Merge remote-tracking branch 'origin/develop' into feature-gromacs

4347564

Riyaz Haque added 4 commits February 10, 2024 15:33

CUDA experiment for LLNL-Sierra-IBM-power9-V100-Infiniband

0738ba5

CUDA config for LLNL-Pascal-Penguin-broadwell-P100-OmniPat

9f13277

Fix target name

32f84ee

CUDA config for lassen

ee940e6

pearce8 commented Feb 16, 2024

View reviewed changes

Riyaz Haque added 2 commits February 16, 2024 16:15

Fix cuda config

c065093

Set Openmp thread affinity to cores

b8ca5a2

scheibelp requested changes Feb 17, 2024

View reviewed changes

Riyaz Haque and others added 2 commits February 16, 2024 18:06

Fix cuda compiler for gromacs

5f334fe

Merge branch 'develop' into feature-gromacs

5b695ca

pearce8 commented Feb 20, 2024

View reviewed changes

Riyaz Haque added 2 commits February 21, 2024 11:21

gromacs rocm config

72ff192

Merge branch 'feature-gromacs' of github.com:LLNL/benchpark into feat…

078c4a5

…ure-gromacs

pszi1ard reviewed Feb 21, 2024

View reviewed changes

Riyaz Haque added 4 commits February 22, 2024 12:26

gromacs rocm config

5de5699

Remove spack_setup

3fbc244

Merge with develop

379bac6

Merge remote-tracking branch 'origin/develop' into feature-gromacs

89ff45a

scheibelp mentioned this pull request Feb 27, 2024

ROCm packages: externals consistency #150

Merged

scheibelp and others added 3 commits March 4, 2024 13:07

ROCm packages: externals consistency (#150)

2d4f924

* remove version locking from experiment config; achieve this with a mixin package * add needed externals, proper dependency reference for mixin * also align rocmcc version w/hip version

Merge remote-tracking branch 'origin/develop' into feature-gromacs

5bd8821

Merge with develop

90ccdd9

pearce8 mentioned this pull request Jun 21, 2024

gromacs #197

Merged

8 tasks

rfhaque closed this in #197 Jun 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding GROMACS #61

Adding GROMACS #61

pearce8 commented Nov 29, 2023

pszi1ard Dec 1, 2023

pearce8 Dec 5, 2023 •

edited

Loading

pszi1ard Dec 1, 2023

rfhaque Dec 1, 2023 •

edited

Loading

pszi1ard Dec 1, 2023

pearce8 Dec 18, 2023

pearce8 Feb 16, 2024

scheibelp left a comment

scheibelp Feb 17, 2024

scheibelp Feb 17, 2024

scheibelp Feb 17, 2024

scheibelp Feb 17, 2024

scheibelp Feb 17, 2024

scheibelp Feb 17, 2024

scheibelp Feb 17, 2024

scheibelp Feb 17, 2024

scheibelp Feb 17, 2024

pearce8 Feb 20, 2024 •

edited

Loading

pszi1ard Feb 21, 2024



		class Gromacs(SpackApplication):
		'''Define a Gromacs application'''

Adding GROMACS #61

Adding GROMACS #61

Conversation

pearce8 commented Nov 29, 2023

Choose a reason for hiding this comment

pearce8 Dec 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rfhaque Dec 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scheibelp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pearce8 Feb 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pearce8 Dec 5, 2023 •

edited

Loading

rfhaque Dec 1, 2023 •

edited

Loading

pearce8 Feb 20, 2024 •

edited

Loading