OpenACC/CUDA performance on juwels-booster

[goord]

I have tested the STORM test case on juwels-booster, a machine featuring 2 24-core AMD EPYC-ROME procesors and 4 NVIDIA A100 GPU cards per node. I have used 288 MPI tasks (6 nodes) and the ParaStation MPI implementation. The results are

build	total runtime [s]	tracer runtime [s]
intel	456	215
intel+cuda	434	192
gnu	444	194
gnu+cuda	415	165
pgi	483	183
pgi+openacc	463	163

I have tuned both the CUDA and OpenACC versions to use 64 threads per block, to fit the 47 vertical levels.

[koldunovn]

Looks good! If only pgi would be as effective as intel for the rest of the code :) @dsidoren Maybe we could think of adding some artificial additional tracers for testing, to imitate BGC model, to show the benefit of using GPU better?

[koldunovn]

I can provide 23M mesh for testing if you wish :)

[goord]

I would appreciate that, do you have an ftp site or something to download from?

[koldunovn]

I will put it online now. The question is - do you want me to do the partitioning (if do, so for how many cores), or you can do it yourself?

[goord]

I can do it myself, I need a multiple of 48, what do you recommend? Also, which time step should I use?

[koldunovn]

You can download it from here:
https://swift.dkrz.de/v1/dkrz_c719fbc3-98ea-446c-8e01-356dac22ed90/xoak/R4.2_mesh.tar.bz2?temp_url_sig=f060ac88fc0015aabbb2b3a6a423763e03d10f99&temp_url_expires=2021-02-02T13:44:35Z

Usually we don't do more than 300-400 nodes per core, so something like 75000 would be the maximum. I would go for 1 minute (1440 per day), should be stable. Note, that realistic sea ice at this resolution would require more EVP subcycles (evp_rheol_steps), but to be comparable with other setups, initially I would keep it the same.

[dguibert]

a machine featuring 2 24-core AMD EPYC-ROME procesors and 4 NVIDIA A100 GPU cards per node. I have used 288 MPI tasks (6 nodes) and the ParaStation MPI implementation

How many gpus are used in OpenACC and CUDA?
Do you set CUDA_VISIBLE_DEVICES to enforce spreading the MPI process to the GPU? if yes, do you ensure that the MPI processes access their closest GPU on the socket?

[goord]

Hi @dguibert I use all 4 available GPU's per node, but I haven't inserted any code to ensure the affinity you mention. I'm really struggling to get the CUDA-aware MPI going BTW, I basically have given up on that