read_spice_l2_fits fails because of aligned axes on wide-slit SPICE L2 files

[ebuchlin]

Description

Opening a SPICE L2 file corresponding to a raster taken with the wide (30") slit (solo_L2_spice-w-ras_*.fits files) with read_spice_l2_fits fails: there is a "ValueError: Aligned axes are not all of same length." exception, while we expect these files to open with no error.

Steps to Reproduce

• Download solo_L2_spice-w-ras_20201117T180612_V08_33554567-000.fits
• Open it with sunraster:

from sunraster.instr.spice import read_spice_l2_fits
r = read_spice_l2_fits('solo_L2_spice-w-ras_20201117T180612_V08_33554567-000.fits')

The trace is:

/usr/local/lib/python3.9/dist-packages/sunraster/instr/spice.py in read_spice_l2_fits(filenames, windows, memmap, read_dumbbells)
    117     else:
    118         aligned_axes = None
--> 119     return NDCollection(window_sequences, aligned_axes=aligned_axes)
    120 
    121 

/usr/local/lib/python3.9/dist-packages/ndcube/ndcollection.py in __init__(self, key_data_pairs, aligned_axes, meta, **kwargs)
     57             # Sanitize aligned axes unless hidden kwarg indicates not to.
     58             if sanitize_inputs:
---> 59                 aligned_axes = collection_utils._sanitize_aligned_axes(keys, data, aligned_axes)
     60             else:
     61                 aligned_axes = dict(zip(keys, aligned_axes))

/usr/local/lib/python3.9/dist-packages/ndcube/utils/collection.py in _sanitize_aligned_axes(keys, data, aligned_axes)
     19     else:
     20         # Else, sanitize user-supplied aligned axes.
---> 21         sanitized_axes = _sanitize_user_aligned_axes(data, aligned_axes)
     22 
     23     return dict(zip(keys, sanitized_axes))

/usr/local/lib/python3.9/dist-packages/ndcube/utils/collection.py in _sanitize_user_aligned_axes(data, aligned_axes)
    104                             for j in range(n_aligned_axes)])
    105     if check_dimensions != {1}:
--> 106         raise ValueError("Aligned axes are not all of same length.")
    107 
    108     return aligned_axes

ValueError: Aligned axes are not all of same length.

This seems to be because the detection of aligned axes fails to exclude the spectral axis in the case of wide-slit observations, maybe because of the phys_type != ("em.wl",) condition on spice.py line 115.

In the case of wide slit observations, there is of course some spatial X information on the wavelength axis, but we still have CTYPE3 = 'WAVE ', so I guess that the corresponding physical type should still be em.wl. first_sequence.array_axis_physical_types is [('time',), ('custom:pos.helioprojective.lon', 'custom:pos.helioprojective.lat', 'em.wl'), ('custom:pos.helioprojective.lon', 'custom:pos.helioprojective.lat'), ('custom:pos.helioprojective.lon', 'custom:pos.helioprojective.lat', 'time')] for this file.

Also, please note that in current version of the SPICE pipeline, contiguous windows are merged on the wavelength axis, and the merged windows are duplicated in the L2 FITS file (retaining the relevant metadata of each parent window), but this does not seem to be the root of the current issue.

System Details

• Python packages:

sunraster==0.3.0
ndcube==2.0.1
sunpy==3.1.2
astropy==5.0

• Python 3.9.5
• OS information: Ubuntu 21.04 (issue confirmed by someone else with Windows)

[DanRyanIrish]

Hi @ebuchlin. Thanks for raising this bug. From your description above, it seems that array axes associated with wavelength should not be aligned, even if they are also associated with other coordinates, e.g. spatial X. (The exception of course, if the data come from the same spectral window.) If my understanding is correct, then this bug should be fixed with the following change to spice.py line 114-116:

aligned_axes = tuple(i for i, phys_type in enumerate(first_sequence.array_axis_physical_types) if "em.wl" in set(phys_type))

This will exclude ALL array axes associated with wavelength, not just those ONLY associated with wavelength as before. Do you agree with this fix?

Also, please note that in current version of the SPICE pipeline, contiguous windows are merged on the wavelength axis, and the merged windows are duplicated in the L2 FITS file (retaining the relevant metadata of each parent window), but this does not seem to be the root of the current issue.

I don't understand whether this has consequences for this bugfix. But it does, could you elaborate?

[ebuchlin]

Hi @ebuchlin. Thanks for raising this bug. From your description above, it seems that array axes associated with wavelength should not be aligned, even if they are also associated with other coordinates, e.g. spatial X. (The exception of course, if the data come from the same spectral window.) If my understanding is correct, then this bug should be fixed with the following change to spice.py line 114-116:

aligned_axes = tuple(i for i, phys_type in enumerate(first_sequence.array_axis_physical_types) if "em.wl" in set(phys_type))

This will exclude ALL array axes associated with wavelength, not just those ONLY associated with wavelength as before. Do you agree with this fix?

Hi @DanRyanIrish, yes this works in this case (but with not in instead of in), and it seems reasonable to exclude from aligned axes all array axes associated with wavelength (there should be only one in SPICE L2 files).

However, we have now found an other case showing that this way of determining aligned axes is still insufficient. SPICE has the capability to bin in the y direction with different factors for different windows. The unbinned axes are then aligned, but the binned axes are not (unless there is some way to consider axes with a different number of pixels as aligned). I don't know yet what should be done in this case. I can provide an example file and/or open another issue for this.

Also, please note that in current version of the SPICE pipeline, contiguous windows are merged on the wavelength axis, and the merged windows are duplicated in the L2 FITS file (retaining the relevant metadata of each parent window), but this does not seem to be the root of the current issue.

I don't understand whether this has consequences for this bugfix. But it does, could you elaborate?

No, I don't expect any consequence on this bug, but it was one of the hypotheses I considered, so I thought that it was interesting to be aware of that.

[nabobalis]

The test file loads with #204

If you find other files that are broken, please do either reopen this issue or create a new one please!

This is in the sunraster release 0.4.2, if you update to that pip install -U sunraster (you will have to wait about 30 minutes before it shows up on PyPi for install).