[Bug]: Problem when writing to csv from N>1 nodes

[ClaudiaComito]

What happened?

test_io.test_save_csv hangs when processes are distributed over several nodes. Works fine if all processes are on the same node.

Code snippet triggering the error

# On HDFML
salloc --account=haf --nodes=2 --time=00:30:00 --gres=gpu:4
ml GCC OpenMPI PyTorch torchvision mpi4py HDF5 netCDF 
srun -N 2 --ntasks-per-node=1 python -m unittest -vf   heat/core/tests/test_io.py

Error message or erroneous outcome

test_save_csv (heat.core.tests.test_io.TestIO) ... test_save_csv (heat.core.tests.test_io.TestIO) ...

Version

1.2.x

Python version

3.9

PyTorch version

1.11

[bhagemeier]

Hi Claudia, this surprises me, as it has of course been tested successfully in the past. I'll look into it.

[ClaudiaComito]

This persists after bug fix #1058 . Would you like to look into this @mtar or (when you're back) @JuanPedroGHM ?

[mtar]

The IO unit tests use temporary files. Have you set one of the TMPDIR, TEMP or TMP environment variables to a directory that can be accessed by all nodes?

[ClaudiaComito]

thanks @mtar that was it.