Start Embedded Boundary development #1641

WeiqunZhang · 2021-01-20T01:20:26Z

Modify the build system for embedded boundary (EB) support. Currently EB is
a compile time option that must be explicitly enabled. For GNU Make, one
can enable it with USE_EB=TRUE, whereas for CMake, -DAMReX_EB=ON. Later
we could decide to enable EB by default with all regular geometry.

Add a simple geometry initialization function, WarpX::InitEB. By default,
the geometry is all regular. A few basic types such as box, cylinder,
plane, sphere, etc. are supported via ParmParse runtime parameters. See
amrex/Src/EB/AMReX_EB2.cpp for more details. Later, we could build more
complex geometry using constructive solid geometry (CSG). (
https://en.wikipedia.org/wiki/Constructive_solid_geometry ) See
amrex/Tutorials/EB/GeometryGeneration for an example of CSG. There is
also a STL ( https://en.wikipedia.org/wiki/STL_(file_format) ) approach
under development in AMReX.

Add a new member, m_factory, to WarpX class. This object can be used to
obtain geometry information such as whether a cell is cut, edge centroids,
etc. Currently we are not using these factories to build MultiFabs for
field data to embed the geometry information into the data containers. We
could do that later if it is needed or it makes things more convenient.
Nevertheless, this should be sufficient to start the EB development of the
field solver.

It's not clear to me yet how many ghost cells are needed for the geometry
information. It's currently one, and can be adjusted.

In the future, when particle and embedded boundary interaction is
considered, we can use amrex::FillSignedDistance function to obtain signed
distance function on the nodes. With that information, one should be able
to determine where and when a particle collides with the embedded boundary.

Modify the build system for embedded boundary (EB) support. Currently EB is a compile time option that must be explicitly enabled. For GNU Make, one can enable it with `USE_EB=TRUE`, whereas for CMake, `-DAMReX_EB=ON`. Later we could decide to enable EB by default with all regular geometry. Add a simple geometry initialization function, WarpX::InitEB. By default, the geometry is all regular. A few basic types such as box, cylinder, plane, sphere, etc. are supported via ParmParse runtime parameters. See `amrex/Src/EB/AMReX_EB2.cpp` for more details. Later, we could build more complex geometry using constructive solid geometry (CSG). ( https://en.wikipedia.org/wiki/Constructive_solid_geometry ) See `amrex/Tutorials/EB/GeometryGeneration` for an example of CSG. There is also a STL ( https://en.wikipedia.org/wiki/STL_(file_format) ) approach under development in AMReX. Add a new member, m_factory, to WarpX class. This object can be used to obtain geometry information such as whether a cell is cut, edge centroids, etc. Currently we are not using these factories to build MultiFabs for field data to embed the geometry information into the data containers. We could do that later if it is needed or it makes things more convenient. Nevertheless, this should be sufficient to start the EB development of the field solver. It's not clear to me yet how many ghost cells are needed for the geometry information. It's currently one, and can be adjusted. In the future, when particle and embedded boundary interaction is considered, we can use `amrex::FillSignedDistance` function to obtain signed distance function on the nodes. With that information, one should be able to determine where and when a particle collides with the embedded boundary.

for embedded boundaries

ax3l

Added additional user-facing CMake logic and some small cleaning :)

Explicit includes are most robust to make sure defines are set with future refactorings.

ax3l · 2021-01-26T16:19:44Z

Source/WarpX.H

+    // Factory for field data
+    amrex::Vector<std::unique_ptr<amrex::FabFactory<amrex::FArrayBox> > > m_factory;
+
+    amrex::FabFactory<amrex::FArrayBox> const& Factory (int lev) const noexcept {


Factory is a very general term in programming for a function that constructs classes.
As a general WarpX::Factory method it seems a bit technical/non-descript. Do you have a suggestion how we could make it more descriptive for the domain scientists?

E.g. fieldFactory or make_fields and make_EBfields or so?

I like. fieldFactory. I will make the change.

ax3l · 2021-01-26T16:21:43Z

I don't understand why the PSATD runtime tests crash now... They run on GNUmake and I thought my include re-ordering should not have caused this. But I guess I am wrong on that? Will trigger the PSATD CI again, maybe it was a hickup.

ax3l · 2021-01-27T00:32:02Z

Restarting CI worked :)

ax3l

Thank you, all aboard! 🚢

…rpX#1624) * Fix bug in momentum-conserving interpolation function * Reset checksum benchmark for CI test momentum-conserving-gather Unused Params: Check after Step 0 (ECP-WarpX#1596) An early check for unused parameters after the first step. Add Superparticle version of getParticlePosition. (ECP-WarpX#1640) * Add Superparticle version of GetParticlePosition. * move unpack_particle to a free function and rename * no longer need SuperPType * Update Source/Particles/Pusher/GetAndSetPosition.H * remove templating from get_particle_position * remove template * Add missing include Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Distribution mapping and cost plotting (ECP-WarpX#1444) * Distribution mapping and cost plotting Cleanup Cleanup Cleanup EOL Unused import * Cost initializes to 0.0 * plot slices of 3D * WIP * WIP docs * docs * docs * docs * docs * docs * docs * docs * docs * EOL SpeciesProperties: C, N, O, Cu (ECP-WarpX#1638) * SpeciesProperties: C, N, O, Cu Add more ionic species as pre-defined particle species. * Cu: Add to Ionization Energies Fix 2D: ParticleHistogram (ECP-WarpX#1635) Fix out-of-bounds access in particle histogram in non-3D. Fix FilterFunctor in non-3D (ECP-WarpX#1633) Fix an out-of-bounds access to positions in 2D & RZ for filter functors. openPMD: ionizationLevel (ECP-WarpX#1622) * openPMD: ionizationLevel Write out the ionizationLevel with openPMD. * openPMD Record Repetitions: Use Lambdas openPMD: Particle Filter (Container) (ECP-WarpX#1632) * openPMD: Particle Filter (Container) A fresh implementation of particle filters for openPMD diagnostics using the generalized `PhysicalParticleContainer` approach that we also use in plotfiles 🎉 * Ion Example: Filter & Coarsen Test diagnostics filter & coarsen functions in CI. Test reduced diagnostics (histograms). Define: _OPENMP -> AMREX_USE_OMP (ECP-WarpX#1520) * Define: _OPENMP -> AMREX_USE_OMP Replace the define check of `_OPENMP` with the explicit backend control of `AMREX_USE_OMP` for parallel constructs. Doing so avoids that we accidentially turn on OpenMP, e.g. if a dependency pulls it in for linear algebra, I/O, etc. This can led to confusion if the user explicitly requested a serial build. Also, we might want to use OpenMP functionality here and there for auxiliary functions w/o having to use the AMReX OpenMP backend, i.e. because we compile for GPUs. * Add missing amrex::Gpu::notInLaunchRegion Start: PerformanceHints After Init (ECP-WarpX#1630) Start a helper routine that gives performance hints after initialization of the simulation. Fix number of guard cells in PML with 2D PSATD (ECP-WarpX#1648) Remove unused lines about particle filters (ECP-WarpX#1646) Add parameter for default galilean velocity (ECP-WarpX#1097) * Add parameter for default galilean velocity * Apply suggestions from code review Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> * Fix bug in constructor of PhysicalParticleContainer * Use new input parameter in CI test Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Co-authored-by: Edoardo Zoni <ezoni@lbl.gov> Cleaned up field diagnostic in picmi interface (ECP-WarpX#1649) Use pinned memory for tmp particles in diags. (ECP-WarpX#1644) * Use pinned memory for tmp particles in diags. * openPMD: pinned memory (tmp particles) Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Replace an always false test with an abort (ECP-WarpX#1655) * replace always false test with abort * fix bug [mini-PR] Broken link to scripts (ECP-WarpX#1651) CI: Ubuntu-latest -> 20.04 (ECP-WarpX#1654) Start Embedded Boundary development (ECP-WarpX#1641) * Start Embedded Boundary development Modify the build system for embedded boundary (EB) support. Currently EB is a compile time option that must be explicitly enabled. For GNU Make, one can enable it with `USE_EB=TRUE`, whereas for CMake, `-DAMReX_EB=ON`. Later we could decide to enable EB by default with all regular geometry. Add a simple geometry initialization function, WarpX::InitEB. By default, the geometry is all regular. A few basic types such as box, cylinder, plane, sphere, etc. are supported via ParmParse runtime parameters. See `amrex/Src/EB/AMReX_EB2.cpp` for more details. Later, we could build more complex geometry using constructive solid geometry (CSG). ( https://en.wikipedia.org/wiki/Constructive_solid_geometry ) See `amrex/Tutorials/EB/GeometryGeneration` for an example of CSG. There is also a STL ( https://en.wikipedia.org/wiki/STL_(file_format) ) approach under development in AMReX. Add a new member, m_factory, to WarpX class. This object can be used to obtain geometry information such as whether a cell is cut, edge centroids, etc. Currently we are not using these factories to build MultiFabs for field data to embed the geometry information into the data containers. We could do that later if it is needed or it makes things more convenient. Nevertheless, this should be sufficient to start the EB development of the field solver. It's not clear to me yet how many ghost cells are needed for the geometry information. It's currently one, and can be adjusted. In the future, when particle and embedded boundary interaction is considered, we can use `amrex::FillSignedDistance` function to obtain signed distance function on the nodes. With that information, one should be able to determine where and when a particle collides with the embedded boundary. * resize factory vector * CMake: Require AMReX_EB for embedded boundaries * Add AMReX_Config.H et al. Explicit includes are most robust to make sure defines are set with future refactorings. * rename Factory fieldFactory Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Add filter for particle histogram (ECP-WarpX#1643) * Add filter for particle histogram * Update benchmarks * Update benchmark again * Use serialize_ics in test + proper usage of RandomEngine * Laser-Ion Acc. (2D3V): Test Hist Filter Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Take time step into account to compute guard cells for J and rho (ECP-WarpX#1607) * Use IntVect for ng_J and ng_rho * Compute guard cells for J and rho based on dt * Reset some CI benchmarks * Fix rebase commit * Add back +1 cell for rho: fix remaining out-of-bound accesses * Simplify ASSERTS using new interface of amrex::numParticlesOutOfRange Modern setup.py: CMake-Driven, Multi-Dimensional (ECP-WarpX#1647) This adds a new, project-centric setup.py file. With this file, all dimensions (2D, 3D, RZ) of WarpX can be built and packaged at once, using the CMake build logic. Build & install: ```bash pip wheel -v . pip install *whl ``` Updates for electrostatic solver (ECP-WarpX#1604) * Fix electrostatic solver with momentum conservation * Fix electrostatic, adding call to FillBoundaryAux * For electrostatic, removed unneeded extra calls to UpdateAux and FillBoundary * For electrostatic, calculate fields at the end of the time step * Updated ElectrostaticSphere analysis script to use fields from end of time step Reset broken benchmark for CI test initial_distribution (ECP-WarpX#1661) make sure we redefine the tmp particle tiles when we load balance. (ECP-WarpX#1658) [mini-PR] Add particle tiling documentation (ECP-WarpX#1665) * tiling wip * do_tiling documentation * Update Docs/source/running_cpp/parameters.rst Co-authored-by: Luca Fedeli <luca.fedeli@for.unipi.it> Co-authored-by: Luca Fedeli <luca.fedeli@for.unipi.it> fix inconsistent formatting (ECP-WarpX#1666) parameter.rst fixes (ECP-WarpX#1663) requirements.txt: fix overspecification (ECP-WarpX#1668) I accidentally added the patch-level for version-compatible matching This removes the patch-level for `~=` matching for the packages that have a >=1 major version already. scipy 1.5+ (ECP-WarpX#1670) Sufficient since it was last release in december and works well. The 3.5 release series still builds wheels for Python 3.6 for Ubuntu oldstable (18.04). Remove redundant "do_qed" option in inputfile (ECP-WarpX#1667) * removed redundant do_qed option in inputfile * fixed bug remove semicolons (ECP-WarpX#1662) fixTypo for pml has particles parameter for docs (ECP-WarpX#1671) load balance efficiency reduced diagnostic eol minor eol

…rpX#1624) * Fix bug in momentum-conserving interpolation function * Reset checksum benchmark for CI test momentum-conserving-gather Unused Params: Check after Step 0 (ECP-WarpX#1596) An early check for unused parameters after the first step. Add Superparticle version of getParticlePosition. (ECP-WarpX#1640) * Add Superparticle version of GetParticlePosition. * move unpack_particle to a free function and rename * no longer need SuperPType * Update Source/Particles/Pusher/GetAndSetPosition.H * remove templating from get_particle_position * remove template * Add missing include Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Distribution mapping and cost plotting (ECP-WarpX#1444) * Distribution mapping and cost plotting Cleanup Cleanup Cleanup EOL Unused import * Cost initializes to 0.0 * plot slices of 3D * WIP * WIP docs * docs * docs * docs * docs * docs * docs * docs * docs * EOL SpeciesProperties: C, N, O, Cu (ECP-WarpX#1638) * SpeciesProperties: C, N, O, Cu Add more ionic species as pre-defined particle species. * Cu: Add to Ionization Energies Fix 2D: ParticleHistogram (ECP-WarpX#1635) Fix out-of-bounds access in particle histogram in non-3D. Fix FilterFunctor in non-3D (ECP-WarpX#1633) Fix an out-of-bounds access to positions in 2D & RZ for filter functors. openPMD: ionizationLevel (ECP-WarpX#1622) * openPMD: ionizationLevel Write out the ionizationLevel with openPMD. * openPMD Record Repetitions: Use Lambdas openPMD: Particle Filter (Container) (ECP-WarpX#1632) * openPMD: Particle Filter (Container) A fresh implementation of particle filters for openPMD diagnostics using the generalized `PhysicalParticleContainer` approach that we also use in plotfiles 🎉 * Ion Example: Filter & Coarsen Test diagnostics filter & coarsen functions in CI. Test reduced diagnostics (histograms). Define: _OPENMP -> AMREX_USE_OMP (ECP-WarpX#1520) * Define: _OPENMP -> AMREX_USE_OMP Replace the define check of `_OPENMP` with the explicit backend control of `AMREX_USE_OMP` for parallel constructs. Doing so avoids that we accidentially turn on OpenMP, e.g. if a dependency pulls it in for linear algebra, I/O, etc. This can led to confusion if the user explicitly requested a serial build. Also, we might want to use OpenMP functionality here and there for auxiliary functions w/o having to use the AMReX OpenMP backend, i.e. because we compile for GPUs. * Add missing amrex::Gpu::notInLaunchRegion Start: PerformanceHints After Init (ECP-WarpX#1630) Start a helper routine that gives performance hints after initialization of the simulation. Fix number of guard cells in PML with 2D PSATD (ECP-WarpX#1648) Remove unused lines about particle filters (ECP-WarpX#1646) Add parameter for default galilean velocity (ECP-WarpX#1097) * Add parameter for default galilean velocity * Apply suggestions from code review Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> * Fix bug in constructor of PhysicalParticleContainer * Use new input parameter in CI test Co-authored-by: Edoardo Zoni <59625522+EZoni@users.noreply.github.com> Co-authored-by: Edoardo Zoni <ezoni@lbl.gov> Cleaned up field diagnostic in picmi interface (ECP-WarpX#1649) Use pinned memory for tmp particles in diags. (ECP-WarpX#1644) * Use pinned memory for tmp particles in diags. * openPMD: pinned memory (tmp particles) Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Replace an always false test with an abort (ECP-WarpX#1655) * replace always false test with abort * fix bug [mini-PR] Broken link to scripts (ECP-WarpX#1651) CI: Ubuntu-latest -> 20.04 (ECP-WarpX#1654) Start Embedded Boundary development (ECP-WarpX#1641) * Start Embedded Boundary development Modify the build system for embedded boundary (EB) support. Currently EB is a compile time option that must be explicitly enabled. For GNU Make, one can enable it with `USE_EB=TRUE`, whereas for CMake, `-DAMReX_EB=ON`. Later we could decide to enable EB by default with all regular geometry. Add a simple geometry initialization function, WarpX::InitEB. By default, the geometry is all regular. A few basic types such as box, cylinder, plane, sphere, etc. are supported via ParmParse runtime parameters. See `amrex/Src/EB/AMReX_EB2.cpp` for more details. Later, we could build more complex geometry using constructive solid geometry (CSG). ( https://en.wikipedia.org/wiki/Constructive_solid_geometry ) See `amrex/Tutorials/EB/GeometryGeneration` for an example of CSG. There is also a STL ( https://en.wikipedia.org/wiki/STL_(file_format) ) approach under development in AMReX. Add a new member, m_factory, to WarpX class. This object can be used to obtain geometry information such as whether a cell is cut, edge centroids, etc. Currently we are not using these factories to build MultiFabs for field data to embed the geometry information into the data containers. We could do that later if it is needed or it makes things more convenient. Nevertheless, this should be sufficient to start the EB development of the field solver. It's not clear to me yet how many ghost cells are needed for the geometry information. It's currently one, and can be adjusted. In the future, when particle and embedded boundary interaction is considered, we can use `amrex::FillSignedDistance` function to obtain signed distance function on the nodes. With that information, one should be able to determine where and when a particle collides with the embedded boundary. * resize factory vector * CMake: Require AMReX_EB for embedded boundaries * Add AMReX_Config.H et al. Explicit includes are most robust to make sure defines are set with future refactorings. * rename Factory fieldFactory Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Add filter for particle histogram (ECP-WarpX#1643) * Add filter for particle histogram * Update benchmarks * Update benchmark again * Use serialize_ics in test + proper usage of RandomEngine * Laser-Ion Acc. (2D3V): Test Hist Filter Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja> Take time step into account to compute guard cells for J and rho (ECP-WarpX#1607) * Use IntVect for ng_J and ng_rho * Compute guard cells for J and rho based on dt * Reset some CI benchmarks * Fix rebase commit * Add back +1 cell for rho: fix remaining out-of-bound accesses * Simplify ASSERTS using new interface of amrex::numParticlesOutOfRange Modern setup.py: CMake-Driven, Multi-Dimensional (ECP-WarpX#1647) This adds a new, project-centric setup.py file. With this file, all dimensions (2D, 3D, RZ) of WarpX can be built and packaged at once, using the CMake build logic. Build & install: ```bash pip wheel -v . pip install *whl ``` Updates for electrostatic solver (ECP-WarpX#1604) * Fix electrostatic solver with momentum conservation * Fix electrostatic, adding call to FillBoundaryAux * For electrostatic, removed unneeded extra calls to UpdateAux and FillBoundary * For electrostatic, calculate fields at the end of the time step * Updated ElectrostaticSphere analysis script to use fields from end of time step Reset broken benchmark for CI test initial_distribution (ECP-WarpX#1661) make sure we redefine the tmp particle tiles when we load balance. (ECP-WarpX#1658) [mini-PR] Add particle tiling documentation (ECP-WarpX#1665) * tiling wip * do_tiling documentation * Update Docs/source/running_cpp/parameters.rst Co-authored-by: Luca Fedeli <luca.fedeli@for.unipi.it> Co-authored-by: Luca Fedeli <luca.fedeli@for.unipi.it> fix inconsistent formatting (ECP-WarpX#1666) parameter.rst fixes (ECP-WarpX#1663) requirements.txt: fix overspecification (ECP-WarpX#1668) I accidentally added the patch-level for version-compatible matching This removes the patch-level for `~=` matching for the packages that have a >=1 major version already. scipy 1.5+ (ECP-WarpX#1670) Sufficient since it was last release in december and works well. The 3.5 release series still builds wheels for Python 3.6 for Ubuntu oldstable (18.04). Remove redundant "do_qed" option in inputfile (ECP-WarpX#1667) * removed redundant do_qed option in inputfile * fixed bug remove semicolons (ECP-WarpX#1662) fixTypo for pml has particles parameter for docs (ECP-WarpX#1671) Move getCosts Always synchronize nodal points of PML MultiFabs (ECP-WarpX#1669) * Always synchronize nodal points of PML MultiFabs * Reset benchmark for CI test pml_x_psatd * Use new separate functions NodalSyncPML minor

WeiqunZhang requested review from atmyers and RevathiJambunathan January 20, 2021 01:20

ax3l self-requested a review January 21, 2021 19:52

WeiqunZhang added 3 commits January 22, 2021 09:44

Merge branch 'development' into eb

6cb890a

resize factory vector

075b0f8

Merge branch 'development' into eb

64e8c53

atmyers approved these changes Jan 25, 2021

View reviewed changes

ax3l added component: core Core WarpX functionality enhancement New feature or request labels Jan 25, 2021

ax3l force-pushed the eb branch 4 times, most recently from b000bd4 to ab8ad9e Compare January 25, 2021 07:16

CMake: Require AMReX_EB

2608c92

for embedded boundaries

ax3l force-pushed the eb branch from ab8ad9e to e255ae6 Compare January 25, 2021 07:17

ax3l approved these changes Jan 25, 2021

View reviewed changes

Add AMReX_Config.H et al.

518256c

Explicit includes are most robust to make sure defines are set with future refactorings.

ax3l force-pushed the eb branch from e255ae6 to 518256c Compare January 25, 2021 21:30

ax3l self-assigned this Jan 25, 2021

Merge branch 'development' into eb

8ee2990

ax3l reviewed Jan 26, 2021

View reviewed changes

rename Factory fieldFactory

bf3159a

ax3l approved these changes Jan 27, 2021

View reviewed changes

RevathiJambunathan approved these changes Jan 27, 2021

View reviewed changes

ax3l merged commit 75931e8 into ECP-WarpX:development Jan 27, 2021

RemiLehe mentioned this pull request Feb 3, 2021

Add embedded boundaries in the electrostatic solver #1675

Closed

lgiacome mentioned this pull request Apr 7, 2021

Staircased embedded boundaries in the YEE solver #1881

Merged

peterscherpelz mentioned this pull request Apr 20, 2021

Conductor shapes and particle scraping #1901

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start Embedded Boundary development #1641

Start Embedded Boundary development #1641

WeiqunZhang commented Jan 20, 2021

ax3l left a comment

ax3l Jan 26, 2021 •

edited

Loading

WeiqunZhang Jan 27, 2021

ax3l commented Jan 26, 2021 •

edited

Loading

ax3l commented Jan 27, 2021

ax3l left a comment

Start Embedded Boundary development #1641

Start Embedded Boundary development #1641

Conversation

WeiqunZhang commented Jan 20, 2021

ax3l left a comment

Choose a reason for hiding this comment

ax3l Jan 26, 2021 • edited Loading

Choose a reason for hiding this comment

WeiqunZhang Jan 27, 2021

Choose a reason for hiding this comment

ax3l commented Jan 26, 2021 • edited Loading

ax3l commented Jan 27, 2021

ax3l left a comment

Choose a reason for hiding this comment

ax3l Jan 26, 2021 •

edited

Loading

ax3l commented Jan 26, 2021 •

edited

Loading