You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I just pushed a demo notebook for tile-processing accross multiple device using dask and zarr.
It actually work well but with a major issue, it does not seems that the garbage collector is working in the dask system. Hence, each tile is being push in the device memory but not released when finished processing. In the example the data fit in my GPUs but this example would fail on a very large data, which kind of fail the reason why one would use such approach.
The text was updated successfully, but these errors were encountered:
StRigaud
changed the title
python garbage collector when tile processing with multiple device
python garbage collector fail when tile processing with multiple device
Nov 4, 2024
I just pushed a demo notebook for tile-processing accross multiple device using dask and zarr.
It actually work well but with a major issue, it does not seems that the garbage collector is working in the dask system. Hence, each tile is being push in the device memory but not released when finished processing. In the example the data fit in my GPUs but this example would fail on a very large data, which kind of fail the reason why one would use such approach.
The text was updated successfully, but these errors were encountered: