MemoryError

[camines]

Dear all,

I'm presenting two memory errors,

First saving, but selecting few variables :
#o.run(steps=(24*365*11), time_step=3600, time_step_output=3600*24, outfile='Tons_run.nc',export_variables=['ID', 'age_seconds', 'origin_marker','lon','lat'])

Error:
MemoryError: Unable to allocate 6.92 GiB for an array with shape (48678, 4016) and data type [('ID', '<i4'), ('status', '<i4'), ('moving', '<i4'), ('age_seconds', '<f4'), ('origin_marker', '<i2'), ('lon', '<f4'), ('lat', '<f4'), ('z', '<f4'), ('wind_drift_factor', '<f4'), ('terminal_velocity', '<f4')]

Second just runnung:
o.run()

Error:
MemoryError: Unable to allocate 9.54 GiB for an array with shape (48678, 26302) and data type float64

This model was running with less seeded particles,

Thanks for your time!

[knutfrode]

Hi,

This seems quite strange: the array to be initialized should not have more time steps than 100 (export_buffer_length)
Which version of OpenDrift are you using?

Can you post more of the error message, including at which line this is happening?
(Not with just o.run(), but the full specification that you listed first)

[camines]

Hello,

Thanks for your fast reply :)

Imnot sure how to see the version of OpenDrift.

The code was working with that number of steps normally. I changed and the different of particles was from ca. 9,300 to 49,000 particles.
The number of steps always for me have been the same, I haven't changed it.

Somehow it runs, and save a large .nc file, but gets an error.

This the complete error :

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
<ipython-input-23-6ee243b3c5e5> in <module>
      1 #Here the steps are 11 years in one hour per step of 3600 sec
      2 #o.run()
----> 3 o.run(steps=(24*365*11), time_step=3600, time_step_output=3600*24, outfile='Tons_run.nc',export_variables=['ID', 'age_seconds', 'origin_marker','lon','lat'])
      4 #o.run(outfile='PlastDrift3.nc', time_step= 3600, time_step_output=3600*6, duration=timedelta(weeks=480))
      5 #'x_sea_water_velocity','y_sea_water_velocity','sea_surface_wave_stokes_drift_x_velocity','sea_surface_wave_stokes_drift_y_velocity','x_wind','y_wind'

~/opendrift/opendrift/models/basemodel.py in run(self, time_step, steps, time_step_output, duration, end_time, outfile, export_variables, export_buffer_length, stop_on_error)
   2541             if hasattr(self, 'environment_profiles'):
   2542                 del self.environment_profiles
-> 2543             self.io_import_file(outfile)
   2544 
   2545         self.timer_end('cleaning up')

~/opendrift/opendrift/export/io_netcdf.py in import_file(self, filename, times, elements)
    293 
    294     # Import dataset (history)
--> 295     self.history = np.ma.array(
    296         np.zeros([num_elements, self.steps_output]),
    297         dtype=history_dtype, mask=[True])

~/.conda/envs/opendrift/lib/python3.9/site-packages/numpy/ma/core.py in array(data, dtype, copy, order, mask, fill_value, keep_mask, hard_mask, shrink, subok, ndmin)
   6554 
   6555     """
-> 6556     return MaskedArray(data, mask=mask, dtype=dtype, copy=copy,
   6557                        subok=subok, keep_mask=keep_mask,
   6558                        hard_mask=hard_mask, fill_value=fill_value,

~/.conda/envs/opendrift/lib/python3.9/site-packages/numpy/ma/core.py in __new__(cls, data, mask, dtype, copy, subok, ndmin, fill_value, keep_mask, hard_mask, shrink, order)
   2823         """
   2824         # Process data.
-> 2825         _data = np.array(data, dtype=dtype, copy=copy,
   2826                          order=order, subok=True, ndmin=ndmin)
   2827         _baseclass = getattr(data, '_baseclass', type(_data))

MemoryError: Unable to allocate 6.92 GiB for an array with shape (48678, 4016) and data type [('ID', '<i4'), ('status', '<i4'), ('moving', '<i4'), ('age_seconds', '<f4'), ('origin_marker', '<i2'), ('lon', '<f4'), ('lat', '<f4'), ('z', '<f4'), ('wind_drift_factor', '<f4'), ('terminal_velocity', '<f4')]

Even before, with less particles, it was running well, also adding to the output file this variables:

[x_sea_water_velocity','y_sea_water_velocity','sea_surface_wave_stokes_drift_x_velocity','sea_surface_wave_stokes_drift_y_velocity','x_wind','y_wind']

Way more

Thanks for your time!!! :)

[knutfrode]

Ok, it seems that the simulation has been successfully completed and saved to file, but is simply too large to be imported back into memory.
It is however possible to load a subset of the particles or the time steps:

>>> import opendrift
>>> o = opendrift-open(<netCDF-file>, elements=range(0, 1000))
... or
>>> o = opendrift-open(<netCDF-file>, times=range(0, 1000))

An alternative is to use open_xarray to calculate and plot density fields instead of showing individual elements, like in this example:
https://opendrift.github.io/gallery/example_river_runoff.html

[camines]

Hello again!

Hope everything is going great

Thanks for your answer.

As you mention, it generates an NetCDF file, where I can extract the data of the run and make some analyisis. But in terms on using OpenDrift to plot, I still can not.

I tried to open the file as you mentioned with the elements, but its just too few to have an idea of the run.
I try with a range of time, but I get an error, and I think at the end would not be a representative of the run, even if I plot the last time steps. I try as well with the elements to do something like :o = opendrift.open(File, elements=range(0, 49000, 100)), but not work

The error:
TypeError: Input arrays must have same length. Lengths given: [12444, 12444, 12444, 12444, 12444, 48678]

At the end, taking a look to the river runoff example, I think could be the best way to plot it in density fields, and at the end will give me a density perspective, if would let me open a large output. but when I try to open it, I have this error:

I open the file with:
o = opendrift.open_xarray(File)

And I get this:

`TypeError Traceback (most recent call last)
in
----> 1 o = opendrift.open_xarray(File)

~/opendrift/opendrift/init.py in open_xarray(filename, analysis_file, chunks)
114 o = cls()
115 o.analysis_file = analysis_file
--> 116 o.io_import_file_xarray(filename, chunks=chunks)
117
118

~/opendrift/opendrift/export/io_netcdf.py in import_file_xarray(self, filename, chunks)
237
238 # Read some saved parameters
--> 239 if os.path.exists(self.analysis_file):
240 self.af = xr.open_dataset(self.analysis_file)
241 self.lonmin = self.af.lonmin

~/.conda/envs/opendrift/lib/python3.9/genericpath.py in exists(path)
17 """Test whether a path exists. Returns False for broken symbolic links"""
18 try:
---> 19 os.stat(path)
20 except (OSError, ValueError):
21 return False

TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType
`

Thanks for your time and help !! :)

[knutfrode]

I wonder if your netCDF-file may be incomplete, e.g. due to the simulation ending without being able to postprocess the file. Or maybe this importing does not work since you subset the output with export_variables(?)
Could you check if opendrift.open(file, elements=.... ) works with another OpenDrift output netCDF-file than the one in question?
Also you could run pytest from your main opendrift-folder, to check that all tests are passing (including testing of this type of partial import).
Also, it could help to update to the latest version of OpenDrift, in case you have an older version.

Anyway, as you say, plotting only a subset of either particles or timestep may be a limited visualization, so it might be better to show density fields, which at least is based on all particles and all time steps.
I don't know why also opendrift.open_xarray() does not work for you, but it could be the same reason as mentioned above.

It could be useful to see the ncdump of the actual netCDF-file.

[camines]

Hello again,

Sorry for having you for so long in this. I really really appreciate your time and help :)

The weird thing is that with less particles was working well.

With opendrift.open(file, elements=.... ) it works, even the large .nc file. But it doesent show me much, that's why I'm trying with the densities.
The opendrift.open(file, times=.... ) gives me error, even with an old small output file.

The opendrift.open_xarray() gives me the error I specify before, even with older small files. I'm just using import opendrift, maybe I need something additional?

Today I have something very weird. The model run, and I was just waiting for the output .nc file, and actually run until the end without the Memory error. I just added the Horizontal diffusivity variable. When I tried to plot, the Kernel die. Now even with just few particles, on that specific script, it just dont work

`Time:
Start: 2010-01-01 00:00:00
Present: 2010-01-02 00:00:00
Calculation steps: 46 * 1:00:00 - total time: 1 day, 0:00:00
Output steps: 2 * 1 day, 0:00:00

'The simulation stopped before requested end time was reached.'`

Model: PlastDrift (OpenDrift version 1.5.6) I have this one, how can I update it? :)

I check as you recommend the pytest, and I have a lot of errors and warnings. Would you recommend to update, or reinstall opendrift?
I send you an image of the end

---

A parallel question. It is possible to plot a vector plot without the particles?
Some like an average of the forcings?

I did it for animations, and for plots, by background=['x_sea_water_velocity', 'y_sea_water_velocity'] but is there a way to do it just the forcing?
I tried with show_elements=False, show_trajectories=False, but it didn't worked

Thanks again for your help and time :)

[knutfrode]

Ok, your version is slightly outdated, and also out of sync with the 3rd party requirements. See here for how to update: https://opendrift.github.io/install.html Basically it should be (from OpenDrift folder): git pull conda env update -f environment.yml pip install -e .

…

On Tue, Feb 1, 2022, 23:28 camines ***@***.***> wrote: Hello again, Sorry for having you for so long in this. I really really appreciate your time and help :) The weird thing is that with less particles was working well. With opendrift.open(file, elements=.... ) it works, even the large .nc file. But it doesent show me much, that's why I'm trying with the densities. The opendrift.open(file, times=.... ) gives me error, even with an old small output file. The opendrift.open_xarray() gives me the error I specify before, even with older small files. I'm just using import opendrift, maybe I need something additional? Today I have something very weird. The model run, and I was just waiting for the output .nc file, and actually run until the end without the Memory error. I just added the Horizontal diffusivity variable. When I tried to plot, the Kernel die. Now even with just few particles, on that specific script, it just dont work `Time: Start: 2010-01-01 00:00:00 Present: 2010-01-02 00:00:00 Calculation steps: 46 * 1:00:00 - total time: 1 day, 0:00:00 Output steps: 2 * 1 day, 0:00:00 'The simulation stopped before requested end time was reached.'` Model: PlastDrift (OpenDrift version 1.5.6) I have this one, how can I update it? :) I check as you recommend the pytest, and I have a lot of errors and warnings. Would you recommend to update, or reinstall opendrift? I send you an image of the end ------------------------------ A parallel question. It is possible to plot a vector plot without the particles? Some like an average of the forcings? I did it for animations, and for plots, by background=['x_sea_water_velocity', 'y_sea_water_velocity'] but is there a way to do it just the forcing? I tried with show_elements=False, show_trajectories=False, but it didn't worked Thanks again for your help and time :) [image: pytest] <https://user-images.githubusercontent.com/64263250/152061716-9824b49b-ccdc-4929-92ac-515ebbef7eaf.png> — Reply to this email directly, view it on GitHub <#845 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABH25I2KPUHHZMV6OAAOZA3UZBNANANCNFSM5M325LEQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[camines]

Dear Knut,

hope everything its fine with you.

It actually work the update in the way that it finish the run. It will not plot because as I understand it is too big, by killing the Kernel. I think I can manage to have some plots using just certain data by your recommendation of o = opendrift-open(<netCDF-file>, elements=[n1, ..., n1+x])

As well, now as it ends the run, I think the NetCDF is complete, so actually the o=opendrift.open_xarray(File) is running well. Now Ill try to do some diffusivity plots.

Thanks a lot for your help and time :)

---

I'm still wondering if its possible to have some forcing plots, without the particles.
I plot forgings vectors before with background=['x_sea_water_velocity', 'y_sea_water_velocity'] but is there a way to do it just the forcing?
I tried with show_elements=False, show_trajectories=False but it didn't worked

Thanks again for your help!!