Start: PerformanceHints After Init

[ax3l]

Start a helper routine that gives performance hints after initialization of the simulation.

Example output (amr.n_cell = 96 32 32 and 4 MPI ranks with default blocking factors of 32^3):

MPI initialized with 4 MPI processes
MPI initialized with thread support level 3
OMP initialized with 1 OMP threads
AMReX (21.01-31-g2536f1ae9f7b) initialized
ndiags 1
Level 0: dt = 1.90502406e-15 ; dx = 6.25e-07 ; dy = 1.875e-06 ; dz = 2.125e-06

Grids Summary:
  Level 0   3 grids  98304 cells  100 % of domain
            smallest grid: 32 x 32 x 32  biggest grid: 32 x 32 x 32

  Writing plotfile diags/diag100000

[Warning] [Performance] Too many resources / too little work!
  It looks like you requested more compute resources than there are total number of boxes of cells available (3). You started with (4) MPI ranks, so (1) rank(s) will have no work.
  More information:
  https://warpx.readthedocs.io/en/latest/running_cpp/parallelization.html

STEP 1 starts ...
STEP 1 ends. TIME = 1.90502406e-15 DT = 1.90502406e-15
Walltime = 0.017794147 s; This step = 0.017794103 s; Avg. per step = 0.017794147 s

[mrowan137]

Thanks, looks good!

[dpgrote]

The question was raised in Tuesday morning's meeting about whether work is done by level or all levels at once. If it is by level, this should use the number of boxes only on level 0. Correct?

[ax3l]

This is checking all levels right now, assuming that we can distribute refined levels to otherwise idle ranks that don't have a box from level 0 to crunch on.

[dpgrote]

That was my question, is that a good assumption? Did someone check it?

[RemiLehe]

Yes, I think that this is a good assumption, at least for the starting configuration (i.e. without explicit load-balancing).
For instance, using the above example, let us assume that we have 4 MPI rank (numbered from 0 to 3) and 3 boxes on level 0 + 1 box on level 1.
In that case, amrex will create the 3 boxes from level 0 on ranks 0,1,2. It will then create the box for level 1, and automatically place it on level 1 on rank 3, because amrex uses hints on the memory usage of each MPI ranks, when allocating new boxes.
At least that is my understanding, and I did check this a few months ago. But maybe @WeiqunZhang can confirm.

[dpgrote]

Could this be printed in color? You can use the escape sequences that set the color.

[ax3l]

Hehe, good question.
In practice, escape sequences are sometimes terminal/platform dependent and we might want to add options for this (enable, disable), because not all users know to open files with escape sequences with less -R for instance.

Some terminal settings also highlight keywords like error or warning by default.

Then there is the question of light vs. dark background and good color schemes.

One could use a library like {fmt} for reasonable color support and excellent pythonic formatting. Worth considering because colors are great fun, but I would not do it in this PR yet.

[dpgrote]

Trying to think of a way to make the message more noticeable since it will quickly get lost in the output.

[RemiLehe]

@dpgrote I agree: this is a recurring discussion that we have (e.g. this was also discussed for unused parameters).

Alternatives that were mentioned before include aborting immediately (a bit exaggerated in this particular case), printing the message at each timestep (maybe an issue if we have many such warnings...) and including the message in a separate file (but the user may not necessarily see this).
Any other suggestion is definitely welcome, since the above ones are not ideal.

That being said, it is probably a larger issue than that particular PR, so I think it is fine to merge this one in its current state.