Hard coded compilation flags

[mandresm]

Hi all,

Trying to get some of the AWI coupled models ported into Levante I found that, for some of the components, it is not possible to define the compilation flags without editing the CMake files. One of those components is FESOM. These poses 2 problems:

• ESM-Tools can not pass to the model the compilation flags depending on the machine
• There are many branches of FESOM around, which means adding and if condition in the CMake files on all of these branches, to handle Levante's special requirements will be painful. That's what I did in this commit in a branch for AWICM3: e4d7946

That's a strategy copied from Kai Himstedt: If the environment variable CPU_MODEL=AMD_EPYC_ZEN3 is set, then the -march and -mtune flags needed for Levante are added to the list of compilation flags. Otherwise, compilation flags remain a default.

Alternatively, we could do the same but instead of defining a CPU_MODEL defining directly FESOM_FFLAGS, and if this variable existS in the environment, then CMake should use that instead of the default compilation flags in FESOM.

In my view, no matter what we do, we are gonna have to change the CMakeLists.txt in a number of branches (same goes for ECHAM), but since we will have to do it anyway, I think it is now a good time to discuss what is the best strategy to follow.

@dsidoren @koldunovn @hegish @JanStreffing @pgierz

[koldunovn]

I am not at all CMake expert, so don't know what is the best solution, but can say that setting -march=core-avx2 -mtune=core-avx2 on levante makes model run 2 times faster, so some solution is certainly needed :)
I would go for FESOM_FFLAGS as this is something easy to set in the shell file.

[hegish]

Technically it is not a problem to pass compiler flags to cmake fro the command line. See also here https://cmake.org/cmake/help/latest/envvar/FFLAGS.html?highlight=fflags
The problem with this approach is, this applies to every lib and exe we build from this cmake call. I think we had the compiler flags explicitly set in the CMakeLists.txt to have a well tested configuration. You can break many things with the wrong flags. So you can not just pass the same flags to everything we build.
@mandresm, what is it you want to change in the current setup? And for which cmake targets (i.e. libs, exe)?

[mandresm]

Thanks for the link to the documentation in CMake. I am no expert on CMake. Similarly to what @koldunovn mentioned above, these -march=core-avx2 -mtune=core-avx2 flags are needed in Levante for running FESOM fast. In my commit I have modify the following line in CMakeLists.txt to that end (similarly to what @dsidoren has done):

target_compile_options(${PROJECT_NAME} PRIVATE -r8 -i4 -march=core-avx2 -mtune=core-avx2 -fp-model precise -no-prec-div -no-prec-sqrt -fast-transcendentals -ip -init=zero -no-wrap-margin)

My question to you would be what approach do you recommend to add these 2 flags for the compilation of FESOM, only in Levante?

• Some type of evaluation in the CMakeLists to catch if it's running in Levante as done here: e4d7946
• pass compiler flags to cmake through ESM-Tools, when it finds that is running in Levante
• other solutions...

[koldunovn]

Can we have something like FESOM_model that will be passed only to the model compilation, and maybe FESOM_partitioner that will be passed for partitioner compilation, and some sane defaults if those environmental variables are not present?

[hegish]

If this is so critical, we should also add these flags to the other parts, e.g. the solver, right?
Is there a reason you add the flags to the intel compiler section? I thought levante is AMD?
Which compiler are you using on levante?

[mandresm]

Yes, that would certainly help. I just don't know how to best do that (specially taking into account that this is a change that will have to be applied for multiple branches of FESOM). Would someone from FESOM do it?

[koldunovn]

I did a PR #338 that can serve as a base. With this settings fesom compiles and run core experiments on levante.

[hegish]

There are no flags yet in the PR, right? I will try to do that tomorrow.

[mandresm]

Thanks to both!

[koldunovn]

No, no flags. The optimal (for now), would be

if(${CMAKE_Fortran_COMPILER_ID} STREQUAL  Intel )
	   target_compile_options(${PROJECT_NAME} PRIVATE -r8 -i4 -march=core-avx2 -mtune=core-avx2 -fp-model precise -no-prec-div -no-prec-sqrt -fimf-use-svml -
ip -init=zero -no-wrap-margin)

So I removed -xHost as it conflicts with march. Things might change, of course, and we might not only have to add this two flags fro levante.

[hegish]

Should I add my commits to the levante_update branch?

[hegish]

do you know if the -march=core-avx2 -mtune=core-avx2 flags are valid for the C compiler as well? I also want to build parms this way.

the flags seem to work for parms as well

[hegish]

I have updated https://github.com/FESOM/fesom2/tree/levante_update, let me know what you think.

[mandresm]

Thanks a lot @hegish . By looking at your changes I think I understand that target_compile_options can be used to expand the compiling flag list, right? This is something I was wondering how to do so I've learnt a bit :)

I just have one question, I thought you where going to make the additional flags be dependent on whether it was an AMD processor or not, but instead you are checking against the FESOM_PLATFORM_STRATEGY export variable. Would you clarify why in the end you chose to go for this strategy instead?

[hegish]

I could not automatically distinguish the CPU type from a typical Intel CPU. For example uname -m just says x86_64 on levante. Which is also the variable cmake sets for its CMAKE_HOST_SYSTEM_PROCESSOR variable. So we seem to have to explicitly tell cmake to do a special "levante" build.
To do this I utilized the env.sh script, where we already have each target platform hardcoded. No need to add another variable for each env script.
One can always overwrite this with e.g.
cmake -DFESOM_PLATFORM_STRATEGY=obscure.platform ../src

[mandresm]

Okay, thanks for the insight!

[hegish]

Should we use this approach for the other branches as well? @mandresm, could you provide me with a list of fesom branches you have to deal with?

[pgierz]

The AWI-ESM branches are all the ones that (I know of) we support. I will take care of the main one today, for recom and wiso @mbutzin and @mwerner-awi will need to have a look.

[pgierz]

OK, the main AWI-ESM changes are in #345. I will test it "in real life" as soon as Levante is back online if it compiles, but perhaps @hegish & @mandresm you can already have a look at what I would change.

[mwerner-awi]

The AWI-ESM branches are all the ones that (I know of) we support. I will take care of the main one today, for recom and wiso @mbutzin and @mwerner-awi will need to have a look.

I don't understand all the technical details you are discussing here. But for the wiso part of AWIESM, no extra compiler flags are used. Everything is set via namelist entries.

[pgierz]

Thanks @mwerner-awi. I will then assume that if the standard AWI-ESM works, the wiso variant will work as well and then check off both boxes on @mandresm list once I can confirm that it compiles correctly.

[pgierz]

And additional info in case it went under in the techno-babble: this has to do with a potentially significant model speedup using optimised compiler flags suggested by DKRZ.