Failed tiles in v1.1.0, perhaps due to multiprocessing or related issues

[oegorov]

Some tiles are not reduced, perhaps due to some issues with the multiprocessing. Not sure though...
So far, I found only one such tile. I'll update this post if I find more.

MJD=60423, TILE_ID=1043109, EXP=16812

Reduction stopped at the flux calibration using the science field stars
I successfully reduced this tile on the local machine with the latest lvmdrp version from the master branch and the updated calibration/lvmcore files.

2024-10-31 17:06:58,469 - INFO - Fluxcal /uufs/chpc.utah.edu/common/home/sdss50/sdsswork/lvm/spectro/redux/1.1.0/1043XX/1043109/60423/ancillary/lvm-hobject-b-00016812.fits: elapsed time: 10.2725 seconds using 479.44 MB of memory (peak)
2024-10-31 17:06:58,472 - ERROR - Failed to reduce science frame mjd 60423 exposure 16812: [Errno 39] Directory not empty: '/scratch/general/nfs1/u6045701/pbs/lvm_cluster_run/50eded78-97af-11ef-9aa1-2cea7ff48ea8/temp_20241031_230656'
Traceback (most recent call last):
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/lvmdrp/1.1.0/python/lvmdrp/main.py", line 1789, in run_drp
    science_reduction(expnum, use_longterm_cals=True,
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/lvmdrp/1.1.0/python/lvmdrp/main.py", line 1616, in science_reduction
    fluxcal_sci_ifu_stars(hsci_path, GAIA_CACHE_DIR=MASTERS_DIR+'/gaia_cache')
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/lvmdrp/1.1.0/python/lvmdrp/functions/fluxCalMethod.py", line 568, in fluxcal_sci_ifu_stars
    rss, res_sci = science_sensitivity(rss, res_sci, ext, GAIA_CACHE_DIR, NSCI_MAX=NSCI_MAX, plot=plot)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/lvmdrp/1.1.0/python/lvmdrp/functions/fluxCalMethod.py", line 352, in science_sensitivity
    r, calibrated_spectra, sampling = fluxcal.get_XP_spectra(expnum, ra, dec, plot=False, lim_mag=13.5,
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/git/sdss/lvmdrp/1.1.0/python/lvmdrp/core/fluxcal.py", line 136, in get_XP_spectra
    calibrated_spectra, _ = gaiaxpy.calibrate(ids, truncation=False, save_file=False)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.11_lvmdrp/lib/python3.11/site-packages/gaiaxpy/calibrator/calibrator.py", line 64, in calibrate
    return _calibrate(input_object, sampling, truncation, output_path, output_file, output_format, save_file,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.11_lvmdrp/lib/python3.11/site-packages/gaiaxpy/calibrator/calibrator.py", line 90, in _calibrate
    password=password).read()
                       ^^^^^^
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.11_lvmdrp/lib/python3.11/site-packages/gaiaxpy/input_reader/input_reader.py", line 49, in read
    disable_info=self.disable_info).read()
                                    ^^^^^^
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.11_lvmdrp/lib/python3.11/site-packages/gaiaxpy/input_reader/list_reader.py", line 41, in read
    result = gaia.load_data(ids=sources, format='csv', data_release=_data_release, data_structure='raw',
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.11_lvmdrp/lib/python3.11/site-packages/astroquery/gaia/core.py", line 316, in load_data
    shutil.rmtree(path)
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.11_lvmdrp/lib/python3.11/shutil.py", line 763, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/uufs/chpc.utah.edu/common/home/sdss50/software/pkg/miniconda/3.11_lvmdrp/lib/python3.11/shutil.py", line 761, in rmtree
    os.rmdir(path, dir_fd=dir_fd)
OSError: [Errno 39] Directory not empty: '/scratch/general/nfs1/u6045701/pbs/lvm_cluster_run/50eded78-97af-11ef-9aa1-2cea7ff48ea8/temp_20241031_230656'
2024-10-31 17:06:58,550 - INFO - Reduction EXPNUM 16812: elapsed time: 2515.9578 seconds using 0.00 MB of memory (peak)

[ndrory]

I can confirm that at Utah this tile reduces fine in isolation

[ndrory]

This is caused by astroquery.gaia using now.strftime("%Y%m%d_%H%M%S") to generate temporary filenames, which are not unique in our multiprocessing environment. I've submitted an issue in astroquery astropy/astroquery#3128. We still need a temporary workaround.

[ndrory]

This is fixed upstream with PR astropy/astroquery#3130 in astroquery, and our dependency was updated