Issue looping through ts.get_field()

[ashwynsam]

I ran a WarpX simulation with 3000 time steps and I outputted data at every 25 time steps. I want to create a plot of ion density at a particular spatial point vs. time. I first get a 2D slice of the ion density. From this 2D array, I want to collect the data at just the (50,50) cell (this is an arbitrary point). I append this data into a list. Here is the code I used to do this:

# this loop is to get the ion density at the (50, 50) cell as a function of time 
for i in range(0,3001,25): 
    rho_protons, rho_protons_info = ts.get_field( iteration=i, field='rho_protons', slice_across = 'z', plot=False )
    density_at_point.append(rho_protons[50][50]) #50, 50 here since is arbitrary 

I initialized ts as :

ts = OpenPMDTimeSeries('/scratch/users/amsam/soliton_det/potentialknob/diags/diag1')

However, when I try to run the above for loop, I get the following error:

  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
  File "/tmp/ipykernel_33023/2770327891.py", line 4, in <module>
    rho_protons, rho_protons_info = ts.get_field( iteration=i, field='rho_protons', slice_across = 'z', plot=False )
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/openpmd_viewer/openpmd_timeseries/main.py", line 503, in get_field
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/openpmd_viewer/openpmd_timeseries/data_reader/data_reader.py", line 190, in read_field_cartesian
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/openpmd_viewer/openpmd_timeseries/data_reader/io_reader/field_reader.py", line 109, in read_field_cartesian
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/openpmd_viewer/openpmd_timeseries/data_reader/io_reader/utilities.py", line 124, in get_data
RuntimeError: [Wed Jun 21 17:48:01 2023] [ADIOS2 EXCEPTION] <Toolkit> <transport::file::FilePOSIX> <CheckFile> : couldn't open file /scratch/users/amsam/soliton_det/potentialknob/diags/diag1/openpmd_000975.bp/data.0, in call to POSIX open: errno = 24: Too many open files
: iostream error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 1997, in showtraceback
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/IPython/core/ultratb.py", line 1112, in structured_traceback
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/IPython/core/ultratb.py", line 1006, in structured_traceback
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/IPython/core/ultratb.py", line 859, in structured_traceback
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/IPython/core/ultratb.py", line 793, in format_exception_as_a_whole
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/IPython/core/ultratb.py", line 838, in get_records
  File "/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/pygments/styles/__init__.py", line 82, in get_style_by_name
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1002, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 945, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1439, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1411, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1548, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1591, in _fill_cache
OSError: [Errno 24] Too many open files: '/home/users/amsam/miniconda3/envs/ashwyn/lib/python3.10/site-packages/pygments/styles' 

To debug, I printed out the data at that point in the above loop and I noticed that it runs fine for 39 iterations and then throws the above error. I get this same error when I try to create plots of the 2D data slices at every 25 time steps in my for loop (I am only showing this simpler for loop above to demonstrate this error). Please let me know if you have any insights into how to fix this.

Thank you,
Ashwyn

[RemiLehe]

Thanks a lot for reporting this issue @ashwynsam !
Could you also let us know on which platform or computer cluster this was observed?

@ax3l From the Traceback and the error message

[ADIOS2 EXCEPTION] <Toolkit> <transport::file::FilePOSIX> <CheckFile> : couldn't open file /scratch/users/amsam/soliton_det/potentialknob/diags/diag1/openpmd_000975.bp/data.0, in call to POSIX open: errno = 24: Too many open files

it looks like the error is raised directly by ADIOS2. Have you observed a similar error in the past, on some specific platforms? Also: this seems related to this discussion but I am not sure what the fix is.

[ashwynsam]

@RemiLehe I am running this on Stanford's Sherlock computer (https://www.sherlock.stanford.edu/) .

[ax3l]

Thanks for the ping!

This looks like too many open files. Can you print what ulimit -Sn and ulimit -Hn output?

You can increase the limit temporarily with ulimit -n ... to the 2nd value in the command.

But there is a better way. openPMD supports multiple ways to encode in files how to store multiple steps. For convenience, but bad for performance, many codes (incl. WarpX) default to create a new file (with subfiles) per step, which we call file based encoding in openPMD.

An alternative is group based and especially for ADIOS2 variable based encoding, which keeps the numbers of files constant and is also faster. The correspond WarpX option you can try is <diag_name>.openpmd_encoding = v

You will need to rerun your simulation for that.

https://warpx.readthedocs.io/en/latest/usage/parameters.html#full-diagnostics

[ashwynsam]

@ax3l , I get 256 when I run both those commands.

Thanks for the suggestions, I will look into these.

[ashwynsam]

@ax3l , I re ran the sim with <diag_name>.openpmd_encoding = v and this generated 1 data folder as opposed to 1 for every time step. However, I now don't know how to access the data from this file. Previously, I would do something like :
rho_ions, rho_ions_info = ts.get_field( iteration=10, field='rho_ions', plot=True )
But when I do this, I get this error:

OpenPMDException                          Traceback (most recent call last)
Cell In [5], line 1
----> 1 rho_ions, rho_ions_info = ts.get_field( iteration=10, field='rho_ions', plot=True )

File ~/miniconda3/envs/ashwyn/lib/python3.10/site-packages/openpmd_viewer/openpmd_timeseries/main.py:488, in OpenPMDTimeSeries.get_field(self, field, coord, t, iteration, m, theta, slice_across, slice_relative_position, plot, plot_range, **kw)
    482         raise OpenPMDException(
    483             "The requested mode '%s' is not available.\n"
    484             "The available modes are: \n - %s" % (m, mode_list))
    486 # Find the output that corresponds to the requested time/iteration
    487 # (Modifies self._current_i, self.current_iteration and self.current_t)
--> 488 self._find_output(t, iteration)
    489 # Get the corresponding iteration
    490 iteration = self.iterations[self._current_i]

File ~/miniconda3/envs/ashwyn/lib/python3.10/site-packages/openpmd_viewer/openpmd_timeseries/main.py:629, in OpenPMDTimeSeries._find_output(self, t, iteration)
    627     else:
    628         iter_list = '\n - '.join([str(it) for it in self.iterations])
--> 629         raise OpenPMDException(
    630               "The requested iteration '%s' is not available.\nThe "
    631               "available iterations are: \n - %s\n" % (iteration, iter_list))
    632 else:
    633     pass  # self._current_i retains its previous value

OpenPMDException: The requested iteration '10' is not available.
The available iterations are: 
 - 0

Suggesting that I can only access data from the 0th step when I use ts.get_field()

Can you help me with accessing the data ?

[RemiLehe]

@ax3l More specifically, when using the above workflow (with variable-based encoding), it seems that we are not using the right openPMD-api interface within openPMD-viewer:

from openpmd_viewer import OpenPMDTimeSeries
ts = OpenPMDTimeSeries('./diags/openPMD/')

results in:

The opened Series uses variable-based encoding, but is being accessed by
READ_ONLY mode which operates in random-access manner.
Random-access is (currently) unsupported by variable-based encoding
and some iterations may not be found by this access mode.
Consider using Access::READ_LINEAR and Series::readIterations().

@ax3l Do you know how to fix this?