rho_fp from Python charge density deposition edge cells not properly set

[roelof-groenewald]

PR #2710 added functionality to deposit the charge density for a given species (at a given level) into the rho_fp multifab. After that PR was merged I discovered that the cells around the simulation domain edge are not properly set with this deposition call, as is shown in the figure below:

Adding the call warpx.SyncRho(); to the end of the new function WarpXWrapper.cpp::warpx_depositChargeDensity(), fixes this issue for the present case as is shown below:

My concern is that the function warpx_depositChargeDensity() specifically deposits the charge density in a given level. For the above simulation this doesn't matter since there is just 1 level in the simulation, but if there were more I suspect the warpx.SyncRho(); call would introduce other problems seeing as the charge density was not deposited in other levels.

An temporary fix would be to add the warpx.SyncRho(); call to warpx_depositChargeDensity() as well as a check that max_level == 0 otherwise print a warning saying the function is not appropriate to use.

[roelof-groenewald]

@dpgrote and @ax3l any thoughts on this issue and the suggested fix?

[ax3l]

Or alternatively, we could expose to Python SyncRho and call it after deposition explicitly (from Python)?
That way, we would not change the semantics between C++ and Python calls.

I am wondering right now why we don't do the same sync in the ES solver before calculating fields from the charge...

WarpX/Source/FieldSolver/ElectrostaticSolver.cpp

Lines 115 to 129 in 9610779

	pc.DepositCharge(rho, local, reset, do_rz_volume_scaling);
	// Get the particle beta vector
	bool const local_average = false; // Average across all MPI ranks
	std::array<Real, 3> beta = pc.meanParticleVelocity(local_average);
	for (Real& beta_comp : beta) beta_comp /= PhysConst::c; // Normalize
	// Compute the potential phi, by solving the Poisson equation
	computePhi( rho, phi, beta, pc.self_fields_required_precision,
	pc.self_fields_absolute_tolerance, pc.self_fields_max_iters,
	pc.self_fields_verbosity );
	// Compute the corresponding electric and magnetic field, from the potential phi
	computeE( Efield_fp, phi, beta );
	computeB( Bfield_fp, phi, beta );

Update: clarified on Slack - this overload of the deposition handles boundaries. The other one we use in Python currently does not.

The the EM solver loop, we always pair Deposition and Sync calls:
https://github.com/ECP-WarpX/WarpX/blob/9610779cf94427709a6a77530f545d06dfcbbc3a/Source/Evolve/WarpXEvolve.cpp