Particle balance optimisation

[conradtchan]

Type of PR:
modification to existing code

Description:
In the existing implementation of balancedomains, recv_part is called after checking if a particle belongs to a task to check if a particle is waiting to be received. Checking if a particle belongs to a task is an integer comparison, which is fast relative to performing MPI_Test within recv_part.

These repeated MPI_Test calls present significant overhead and do not need to be called so frequently. This problem is made worse when MPI_THREAD_MULTIPLE is enabled (namely for upcoming optimisation #310), because MPI calls have an additional small overhead.

In this PR, the loop is changed to only call recv_part every check_interval iterations. check_interval is dynamically adjusted based on the frequency of actual receives. The frequency is dependent on the number of cells that have remote neighbours, which is problem dependent. check_interval is initialised as "check every 2 loop iterations", but adjusted to poll at no more than 4x the frequency of actual receives. check_interval is a saved variable across calls of balancedomains, to improve rate convergence, relying on the assumption that the particle distribution does not change drastically between calls.

Testing:
Performance benchmarks were performed for 2 problems, timing the balancedomains routine in seconds, and as a percentage of total run time.

Disc setup with 2M particles, 1 node, 4 MPI tasks, 4 OMP threads per task

old: 16.81s (18.03%)
new: 0.62s (0.79%)
~27x speedup

Common envelope setup (self-gravity) with 2M particles, 2 nodes, 8 MPI tasks per node, 4 OMP threads per task

old: 0.07min (0.90%)
new: 0.01min (0.10%)
~9x speedup

The performance improvement for the self-gravity problem is not as significant because a larger fraction of particles need to be sent. This increases the polling frequency. The limit of maximum polling frequency is equivalent to the old implementation.

Did you run the bots? yes