Merge remote-tracking branch 'upstream/development' into AllocateFewe…

…rGuardCells
NeilZaim · Aug 9, 2022 · 93f2d04 · 93f2d04
2 parents eb23f1e + b13e7f5
commit 93f2d04
Show file tree

Hide file tree

Showing 12 changed files with 168 additions and 49 deletions.
diff --git a/.github/workflows/cuda.yml b/.github/workflows/cuda.yml
@@ -106,7 +106,7 @@ jobs:
         which nvcc || echo "nvcc not in PATH!"
 
         git clone https://github.com/AMReX-Codes/amrex.git ../amrex
-        cd amrex && git checkout --detach 22.08 && cd -
+        cd amrex && git checkout --detach 103db6ebe2b570910ac4dbd7d6611e59d80f1a0b && cd -
         make COMP=gcc QED=FALSE USE_MPI=TRUE USE_GPU=TRUE USE_OMP=FALSE USE_PSATD=TRUE USE_CCACHE=TRUE -j 2
 
   build_nvhpc21-11-nvcc:

diff --git a/Docs/requirements.txt b/Docs/requirements.txt
@@ -8,6 +8,12 @@ breathe
 # docutils 0.17 breaks HTML tags & RTD theme
 # https://github.com/sphinx-doc/sphinx/issues/9001
 docutils<=0.16
+
+# PICMI API docs
+# note: keep in sync with version in ../requirements.txt
+picmistandard==0.0.19
+# for development against an unreleased PICMI version, use:
+#picmistandard @ git+https://github.com/picmi-standard/picmi.git#subdirectory=PICMI_Python
 pygments
 recommonmark
 sphinx>=2.0

diff --git a/Docs/source/usage/python.rst b/Docs/source/usage/python.rst
@@ -3,47 +3,128 @@
 Python (PICMI)
 ==============
 
-WarpX uses the PICMI standard for its Python input files.
-See `PICMI information and source code <https://github.com/picmi-standard/picmi>`__.
-
-WarpX can be run in one of two modes. It can run as a preprocessor, using the
-Python input file to generate an input file to be used by the C++ version, or
-it can be run directly from Python.
-
-In either mode, if using a `virtual environment <https://docs.python.org/3/tutorial/venv.html>`__, be sure to activate it before compiling and running WarpX.
+WarpX uses the `PICMI standard <https://github.com/picmi-standard/picmi>`__ for its Python input files.
+Python version 3.6 or newer is required.
 
 Example input files can be found in :ref:`the examples section <usage-examples>`.
 The examples support running in both modes by commenting and uncommenting the appropriate lines.
 
-WarpX requires Python version 3.6 or newer.
+.. _usage-picmi-parameters:
 
+Parameters
+----------
 
-Using Python input as a preprocessor
-------------------------------------
+Simulation and grid setup
+^^^^^^^^^^^^^^^^^^^^^^^^^
 
-In this case, only the pure Python version needs to be installed, as described :ref:`here <developers-gnumake-python>`.
+The `Simulation` object is the central object in a PICMI script.
+It defines the simulation time, field solver, registered species, etc.
 
-In order to run a new simulation:
+.. autoclass:: picmistandard.PICMI_Simulation
+    :members: step, add_species, add_laser, write_input_file
 
-* Create a **new directory**, where the simulation will be run.
+Field solvers define the updates of electric and magnetic fields.
 
-* Add a **Python script** in the directory.
+.. autoclass:: picmistandard.PICMI_ElectromagneticSolver
 
-The input file should have the line like ``sim.write_input_file(file_name = 'inputs_from_PICMI')``
-which runs the preprocessor, generating the AMReX inputs file.
+.. autoclass:: picmistandard.PICMI_ElectrostaticSolver
 
-* **Run** the script with Python:
+Grid define the geometry and discretization.
 
-.. code-block:: bash
+.. autoclass:: picmistandard.PICMI_Cartesian3DGrid
 
-   python <python_script>
+.. autoclass:: picmistandard.PICMI_Cartesian2DGrid
 
-where ``<python_script>`` is the name of the script.
-This creates the WarpX input file that you can run as normal with the WarpX executable.
+.. autoclass:: picmistandard.PICMI_Cartesian1DGrid
+
+.. autoclass:: picmistandard.PICMI_CylindricalGrid
+
+For convenience, the PICMI interface defines the following constants,
+which can be used directly inside any PICMI script. The values are in SI units.
+
+- ``picmi.constants.c``: The speed of light in vacuum.
+- ``picmi.constants.ep0``: The vacuum permittivity :math:`\epsilon_0`
+- ``picmi.constants.mu0``: The vacuum permeability :math:`\mu_0`
+- ``picmi.constants.q_e``: The elementary charge (absolute value of the charge of an electron).
+- ``picmi.constants.m_e``: The electron mass
+- ``picmi.constants.m_p``: The proton mass
+
+Additionally to self-consistent fields from the field solver, external fields can be applied.
+
+.. autoclass:: picmistandard.PICMI_ConstantAppliedField
+
+.. autoclass:: picmistandard.PICMI_AnalyticAppliedField
+
+.. autoclass:: picmistandard.PICMI_Mirror
+
+Diagnostics can be used to output data.
+
+.. autoclass:: picmistandard.PICMI_ParticleDiagnostic
+
+.. autoclass:: picmistandard.PICMI_FieldDiagnostic
+
+.. autoclass:: picmistandard.PICMI_ElectrostaticFieldDiagnostic
+
+Lab-frame diagnostics diagnostics are used when running boosted-frame simulations.
+
+.. autoclass:: picmistandard.PICMI_LabFrameParticleDiagnostic
+
+.. autoclass:: picmistandard.PICMI_LabFrameFieldDiagnostic
+
+Particles
+^^^^^^^^^
+
+Species objects are a collection of particles with similar properties.
+For instance, background plasma electrons, background plasma ions and an externally injected beam could each be their own particle species.
+
+.. autoclass:: picmistandard.PICMI_Species
+
+.. autoclass:: picmistandard.PICMI_MultiSpecies
+
+Particle distributions can be used for to initialize particles in a particle species.
+
+.. autoclass:: picmistandard.PICMI_GaussianBunchDistribution
+
+.. autoclass:: picmistandard.PICMI_UniformDistribution
+
+.. autoclass:: picmistandard.PICMI_AnalyticDistribution
+
+.. autoclass:: picmistandard.PICMI_ParticleListDistribution
+
+Particle layouts determine how to microscopically place macro particles in a grid cell.
+
+.. autoclass:: picmistandard.PICMI_GriddedLayout
+
+.. autoclass:: picmistandard.PICMI_PseudoRandomLayout
+
+Lasers
+^^^^^^
+
+Laser profiles can be used to initialize laser pulses in the simulation.
+
+.. autoclass:: picmistandard.PICMI_GaussianLaser
+
+.. autoclass:: picmistandard.PICMI_AnalyticLaser
+
+Laser injectors control where to initialize laser pulses on the simulation grid.
+
+.. autoclass:: picmistandard.PICMI_LaserAntenna
+
+
+.. _usage-picmi-run:
+
+Running
+-------
+
+WarpX can be run in one of two modes. It can run as a preprocessor, using the
+Python input file to generate an input file to be used by the C++ version, or
+it can be run directly from Python.
+
+In either mode, if using a `virtual environment <https://docs.python.org/3/tutorial/venv.html>`__, be sure to activate it before compiling and running WarpX.
 
 
 Running WarpX directly from Python
-----------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 For this, a full Python installation of WarpX is required, as described in :ref:`the install documentation <install-users>` (:ref:`developers <install-developers>`).
 
@@ -63,3 +144,26 @@ The input file should have the line ``sim.step()`` which runs the simulation.
 
 where ``<n_ranks>`` is the number of MPI ranks used, and ``<python_script>``
 is the name of the script.
+
+Using Python input as a preprocessor
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In this case, only the pure Python version needs to be installed, as described :ref:`here <developers-gnumake-python>`.
+
+In order to run a new simulation:
+
+* Create a **new directory**, where the simulation will be run.
+
+* Add a **Python script** in the directory.
+
+The input file should have the line like ``sim.write_input_file(file_name = 'inputs_from_PICMI')``
+which runs the preprocessor, generating the AMReX inputs file.
+
+* **Run** the script with Python:
+
+.. code-block:: bash
+
+   python <python_script>
+
+where ``<python_script>`` is the name of the script.
+This creates the WarpX input file that you can run as normal with the WarpX executable.
diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py
@@ -61,23 +61,26 @@ def init(self, kw):
                     self.charge = '-q_e'
                 else:
                     self.charge = self.charge_state*constants.q_e
-            # Match a string of the format '#nXx', with the '#n' optional isotope number.
-            m = re.match(r'(?P<iso>#[\d+])*(?P<sym>[A-Za-z]+)', self.particle_type)
-            if m is not None:
-                element = periodictable.elements.symbol(m['sym'])
-                if m['iso'] is not None:
-                    element = element[m['iso'][1:]]
-                if self.charge_state is not None:
-                    assert self.charge_state <= element.number, Exception('%s charge state not valid'%self.particle_type)
-                    try:
-                        element = element.ion[self.charge_state]
-                    except ValueError:
-                        # Note that not all valid charge states are defined in elements,
-                        # so this value error can be ignored.
-                        pass
-                self.element = element
-                if self.mass is None:
-                    self.mass = element.mass*periodictable.constants.atomic_mass_constant
+            if self.particle_type is not None:
+                # Match a string of the format '#nXx', with the '#n' optional isotope number.
+                m = re.match(r'(?P<iso>#[\d+])*(?P<sym>[A-Za-z]+)', self.particle_type)
+                if m is not None:
+                    element = periodictable.elements.symbol(m['sym'])
+                    if m['iso'] is not None:
+                        element = element[m['iso'][1:]]
+                    if self.charge_state is not None:
+                        assert self.charge_state <= element.number, Exception('%s charge state not valid'%self.particle_type)
+                        try:
+                            element = element.ion[self.charge_state]
+                        except ValueError:
+                            # Note that not all valid charge states are defined in elements,
+                            # so this value error can be ignored.
+                            pass
+                    self.element = element
+                    if self.mass is None:
+                        self.mass = element.mass*periodictable.constants.atomic_mass_constant
+                else:
+                    raise Exception('The species "particle_type" is not known')
 
         self.boost_adjust_transverse_positions = kw.pop('warpx_boost_adjust_transverse_positions', None)
 

diff --git a/Regression/WarpX-GPU-tests.ini b/Regression/WarpX-GPU-tests.ini
@@ -60,7 +60,7 @@ emailBody = Check https://ccse.lbl.gov/pub/GpuRegressionTesting/WarpX/ for more
 
 [AMReX]
 dir = /home/regtester/git/amrex/
-branch = 22.08
+branch = 103db6ebe2b570910ac4dbd7d6611e59d80f1a0b
 
 [source]
 dir = /home/regtester/git/WarpX

diff --git a/Regression/WarpX-tests.ini b/Regression/WarpX-tests.ini
@@ -59,7 +59,7 @@ emailBody = Check https://ccse.lbl.gov/pub/RegressionTesting/WarpX/ for more det
 
 [AMReX]
 dir = /home/regtester/AMReX_RegTesting/amrex/
-branch = 22.08
+branch = 103db6ebe2b570910ac4dbd7d6611e59d80f1a0b
 
 [source]
 dir = /home/regtester/AMReX_RegTesting/warpx

diff --git a/Source/Diagnostics/BTDiagnostics.cpp b/Source/Diagnostics/BTDiagnostics.cpp
@@ -744,10 +744,6 @@ BTDiagnostics::Flush (int i_buffer)
                 vrefratio.push_back(m_particles_buffer[i_buffer][0]->GetParGDB()->refRatio(lev));
             }
         }
-        for (int isp = 0; isp < m_particles_buffer.at(i_buffer).size(); ++isp) {
-            // BTD output is single level. Setting particle geometry, dmap, boxarray to level0
-            m_particles_buffer[i_buffer][isp]->SetParGDB(vgeom[0], vdmap[0], vba[0]);
-        }
         // Redistribute particles in the lab frame box arrays that correspond to the buffer
         // Prior to redistribute, increase buffer box and Box in ParticleBoxArray by 1 index in the
         // lo and hi-end, so particles can be binned in the boxes correctly.
@@ -761,6 +757,10 @@ BTDiagnostics::Flush (int i_buffer)
         amrex::BoxArray buffer_ba( particle_buffer_box );
         buffer_ba.maxSize(m_max_box_size*2);
         m_particles_buffer[i_buffer][0]->SetParticleBoxArray(0, buffer_ba);
+        for (int isp = 0; isp < m_particles_buffer.at(i_buffer).size(); ++isp) {
+            // BTD output is single level. Setting particle geometry, dmap, boxarray to level0
+            m_particles_buffer[i_buffer][isp]->SetParGDB(vgeom[0], vdmap[0], buffer_ba);
+        }
     }
 
     RedistributeParticleBuffer(i_buffer);

diff --git a/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.cpp b/Source/Diagnostics/ComputeDiagFunctors/BackTransformParticleFunctor.cpp
@@ -145,6 +145,7 @@ BackTransformParticleFunctor::operator () (PinnedMemoryParticleContainer& pc_dst
                    if (Flag[i] == 1) GetParticleLorentzTransform(dst_data, src_data, i,
                                                                  old_size + IndexLocation[i]);
                 });
+                amrex::Gpu::synchronize();
             }
         }
     }

diff --git a/Source/EmbeddedBoundary/ParticleScraper.H b/Source/EmbeddedBoundary/ParticleScraper.H
@@ -156,10 +156,14 @@ scrapeParticles (PC& pc, const amrex::Vector<const amrex::MultiFab*>& distance_t
             auto& tile = pti.GetParticleTile();
             auto ptd = tile.getParticleTileData();
             const auto np = tile.numParticles();
+            amrex::Particle<0,0> * const particles = tile.GetArrayOfStructs()().data();
             auto phi = (*distance_to_eb[lev])[pti].array();  // signed distance function
             amrex::ParallelForRNG( np,
             [=] AMREX_GPU_DEVICE (const int ip, amrex::RandomEngine const& engine) noexcept
             {
+                // skip particles that are already flagged for removal
+                if (particles[ip].id() < 0) return;
+
                 amrex::ParticleReal xp, yp, zp;
                 getPosition(ip, xp, yp, zp);
 

diff --git a/cmake/dependencies/AMReX.cmake b/cmake/dependencies/AMReX.cmake
@@ -235,7 +235,7 @@ set(WarpX_amrex_src ""
 set(WarpX_amrex_repo "https://github.com/AMReX-Codes/amrex.git"
     CACHE STRING
     "Repository URI to pull and build AMReX from if(WarpX_amrex_internal)")
-set(WarpX_amrex_branch "22.08"
+set(WarpX_amrex_branch "103db6ebe2b570910ac4dbd7d6611e59d80f1a0b"
     CACHE STRING
     "Repository branch for WarpX_amrex_repo if(WarpX_amrex_internal)")
 

diff --git a/requirements.txt b/requirements.txt
@@ -3,6 +3,7 @@ numpy~=1.15
 periodictable~=1.5
 
 # PICMI
+# note: don't forget to update the version in Docs/requirements.txt, too
 picmistandard==0.0.19
 #   for development against an unreleased PICMI version, use:
 #picmistandard @ git+https://github.com/picmi-standard/picmi.git#subdirectory=PICMI_Python

diff --git a/run_test.sh b/run_test.sh
@@ -71,7 +71,7 @@ python3 -m pip install --upgrade -r warpx/Regression/requirements.txt
 
 # Clone AMReX and warpx-data
 git clone https://github.com/AMReX-Codes/amrex.git
-cd amrex && git checkout --detach 22.08 && cd -
+cd amrex && git checkout --detach 103db6ebe2b570910ac4dbd7d6611e59d80f1a0b && cd -
 # warpx-data contains various required data sets
 git clone --depth 1 https://github.com/ECP-WarpX/warpx-data.git