WIP: allow add_srcdata to place sources on non-owned points

[mawc2019]

This PR continues @stevengj 's work in #1547. At present, it still fails when too many processes are involved, which can be illustrated by the example with 7 processes. The error is unknown src_time_id (missing registration?).

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[stevengj]

Note: numcomm[i,j] here is stored at numcomm[i*P + j] (this is "row major order").

[mawc2019]

In fix_boundary_sources.cpp, the value of srcpts[0].src_time_id in the source process immediately after MPI_Send(srcpts, N, mpi_srcpt_info, pdest, psrc*P + pdest, mycomm) at Line 164 is sometimes not the same as that in the destination process immediately after MPI_Recv(srcpts, N, mpi_srcpt_info, psrc, psrc*P + pdest, mycomm, &status) at Line 129. The value at MPI_Send is a reasonable number such as 1, but the value at MPI_Recv seems a crazy number such as 139732466008065.

Furthermore, in the two places, src_time *srctime = lookup_src_time(srcpts[0].src_time_id) gives different srctime, which is not NULL at MPI_Send in the source process but is NULL at MPI_Recv in the destination process. The null srctime in the destination process causes the error later. Occasionally the error does not appear even though the same code is run.

[stevengj]

Went over the code with @mawc2019 today, and found a typo in the srcpt_info_offsets definition which was causing the problem noted above.

[mawc2019]

The error message of the online check is as follows, but that c++ file already has #include <algorithm>.

../../../src/fix_boundary_sources.cpp: In member function ‘void meep::fields::fix_boundary_sources()’:
../../../src/fix_boundary_sources.cpp:81:8: error: ‘sort’ is not a member of ‘std’; did you mean ‘sqrt’?
   81 |   std::sort(boundarysources.begin(), boundarysources.end(), compare);
      |        ^~~~

There is no error when it is checked offline.

[stevengj]

See my suggested patch above.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 73.58%. Comparing base (55f881f) to head (a01c76d).
Report is 266 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1959      +/-   ##
==========================================
+ Coverage   73.57%   73.58%   +0.01%     
==========================================
  Files          17       17              
  Lines        4896     4898       +2     
==========================================
+ Hits         3602     3604       +2     
  Misses       1294     1294              

Files with missing lines	Coverage Δ
python/adjoint/objective.py	92.07% <100.00%> (ø)
python/simulation.py	77.00% <100.00%> (+<0.01%)	⬆️
python/source.py	95.45% <100.00%> (+0.03%)	⬆️

[stevengj]

Note that this only affects code that use Fourier fields yee_grid=false.