Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: test_cubeviz_aperphot_cube_orig_flux fails with RuntimeError: NULL error object in wcslib #2805

Closed
pllim opened this issue Apr 16, 2024 · 7 comments · Fixed by #2809
Closed
Labels

Comments

@pllim
Copy link
Contributor

pllim commented Apr 16, 2024

Example log: https://github.com/spacetelescope/jdaviz/actions/runs/8706207800/job/23883214225

I started noticing this after merging #2802 but that PR had all green CI checks. I can also reproduce this error locally though it was initially green but only started failing after I reinstalled some packages, so the green was probably due to an outdated cache somewhere. This test does not use any remote data. It is not immediately clear to me why it is suddenly failing and why the error is all the way in wcslib.

The test:

def test_cubeviz_aperphot_cube_orig_flux(cubeviz_helper, image_cube_hdu_obj_microns):

The offending call:

# Need this to make it available for photometry data drop-down.
cubeviz_helper.app.add_data_to_viewer("uncert-viewer", "test[FLUX] collapsed")

The data used:

jdaviz/jdaviz/conftest.py

Lines 102 to 135 in 620ccfe

def image_cube_hdu_obj_microns():
# Basic rectangle ramp for aperture photometry test.
a = np.zeros((8, 9, 10)).astype(np.float32) # (nz, ny, nx)
for i in range(8):
a[i, :5, :3] = i + 1
flux_hdu = fits.ImageHDU(a)
flux_hdu.name = 'FLUX'
uncert_hdu = fits.ImageHDU(np.zeros((8, 9, 10)).astype(np.float32))
uncert_hdu.name = 'ERR'
mask_hdu = fits.ImageHDU(np.ones((8, 9, 10)).astype(np.uint16))
mask_hdu.name = 'MASK'
wcs = {
'WCSAXES': 3, 'CRPIX1': 38.0, 'CRPIX2': 38.0, 'CRPIX3': 1.0,
'CRVAL1': 205.4384, 'CRVAL2': 27.004754, 'CRVAL3': 4.890499866509344,
'CTYPE1': 'RA---TAN', 'CTYPE2': 'DEC--TAN', 'CTYPE3': 'WAVE',
'CUNIT1': 'deg', 'CUNIT2': 'deg', 'CUNIT3': 'um',
'CDELT1': 3.61111097865634E-05, 'CDELT2': 3.61111097865634E-05, 'CDELT3': 0.001000000047497451, # noqa
'PC1_1 ': -1.0, 'PC1_2 ': 0.0, 'PC1_3 ': 0,
'PC2_1 ': 0.0, 'PC2_2 ': 1.0, 'PC2_3 ': 0,
'PC3_1 ': 0, 'PC3_2 ': 0, 'PC3_3 ': 1,
'DISPAXIS': 2, 'VELOSYS': -2538.02,
'SPECSYS': 'BARYCENT', 'RADESYS': 'ICRS', 'EQUINOX': 2000.0,
'LONPOLE': 180.0, 'LATPOLE': 27.004754,
'MJDREFI': 0.0, 'MJDREFF': 0.0, 'DATE-OBS': '2014-03-30'}
flux_hdu.header.update(wcs)
flux_hdu.header['BUNIT'] = '1E-17 erg*s^-1*cm^-2*Angstrom^-1'
uncert_hdu.header['BUNIT'] = '1E-17 erg*s^-1*cm^-2*Angstrom^-1'
return fits.HDUList([fits.PrimaryHDU(), flux_hdu, uncert_hdu, mask_hdu])

_____________________ test_cubeviz_aperphot_cube_orig_flux _____________________

cubeviz_helper = <jdaviz.configs.cubeviz.helper.Cubeviz object at 0x7fcef5e5cfb0>
image_cube_hdu_obj_microns = [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x7fceeb071820>, <astropy.io.fits.hdu.image.ImageHDU object at 0x7fce...py.io.fits.hdu.image.ImageHDU object at 0x7fceeb073530>, <astropy.io.fits.hdu.image.ImageHDU object at 0x7fceeb073b60>]

    def test_cubeviz_aperphot_cube_orig_flux(cubeviz_helper, image_cube_hdu_obj_microns):
        cubeviz_helper.load_data(image_cube_hdu_obj_microns, data_label="test")
        flux_unit = u.Unit("1E-17 erg*s^-1*cm^-2*Angstrom^-1")
    
        aper = RectanglePixelRegion(center=PixCoord(x=1, y=2), width=3, height=5)
        cubeviz_helper.load_regions(aper)
    
        # Make sure MASK is not an option even when shown in viewer.
        cubeviz_helper.app.add_data_to_viewer("flux-viewer", "test[MASK]", visible=True)
    
        plg = cubeviz_helper.plugins["Aperture Photometry"]._obj
        assert plg.dataset.labels == ["test[FLUX]", "test[ERR]"]
        assert plg.cube_slice == "4.894e+00 um"
    
        plg.dataset_selected = "test[FLUX]"
        plg.aperture_selected = "Subset 1"
        plg.vue_do_aper_phot()
        row = cubeviz_helper.get_aperture_photometry_results()[0]
    
        # Basically, we should recover the input rectangle here.
        assert_allclose(row["xcenter"], 1 * u.pix)
        assert_allclose(row["ycenter"], 2 * u.pix)
        sky = row["sky_center"]
        assert_allclose(sky.ra.deg, 205.43985906934287)
        assert_allclose(sky.dec.deg, 27.003490103642033)
        assert_allclose(row["sum"], 75 * flux_unit)  # 3 (w) x 5 (h) x 5 (v)
        assert_allclose(row["sum_aper_area"], 15 * (u.pix * u.pix))  # 3 (w) x 5 (h)
        assert_allclose(row["mean"], 5 * flux_unit)
        assert_quantity_allclose(row["slice_wave"], 4.894499866699333 * u.um)
    
        # Move slider and make sure it recomputes for a new slice automatically.
        cube_slice_plg = cubeviz_helper.plugins["Slice"]._obj
        cube_slice_plg.vue_goto_first()
        plg.vue_do_aper_phot()
        row = cubeviz_helper.get_aperture_photometry_results()[1]
    
        # Same rectangle but different slice value.
        assert_allclose(row["xcenter"], 1 * u.pix)
        assert_allclose(row["ycenter"], 2 * u.pix)
        sky = row["sky_center"]
        assert_allclose(sky.ra.deg, 205.43985906934287)
        assert_allclose(sky.dec.deg, 27.003490103642033)
        assert_allclose(row["sum"], 15 * flux_unit)  # 3 (w) x 5 (h) x 1 (v)
        assert_allclose(row["sum_aper_area"], 15 * (u.pix * u.pix))  # 3 (w) x 5 (h)
        assert_allclose(row["mean"], 1 * flux_unit)
        assert_quantity_allclose(row["slice_wave"], 4.8904998665093435 * u.um)
    
        # We continue on with test_cubeviz_aperphot_generated_2d_collapse here
        # because we want to make sure the result would append properly between 3D and 2D.
        collapse_plg = cubeviz_helper.plugins["Collapse"]._obj
        collapse_plg.vue_collapse()
    
        # Need this to make it available for photometry data drop-down.
>       cubeviz_helper.app.add_data_to_viewer("uncert-viewer", "test[FLUX] collapsed")

../../.tox/py312-test-devdeps/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_cubeviz_aperphot.py:63: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/jdaviz/app.py:1498: in add_data_to_viewer
    self.set_data_visibility(viewer_item, data_label, visible=visible, replace=clear_other_data)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/jdaviz/app.py:2032: in set_data_visibility
    viewer.add_data(data, percentile=95, color=viewer.color_cycler())
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue_jupyter/view.py:118: in add_data
    result = super().add_data(data)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/common/viewer.py:209: in add_data
    layer = get_layer_artist_from_registry(data, self) or self.get_data_layer_artist(data)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue_jupyter/bqplot/image/viewer.py:101: in get_data_layer_artist
    return self.get_layer_artist(cls, layer=layer, layer_state=layer_state)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue_jupyter/view.py:143: in get_layer_artist
    return cls(self, self.state, layer=layer, layer_state=layer_state)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue_jupyter/bqplot/image/layer_artist.py:21: in __init__
    super().__init__(view, *args, **kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/image/layer_artist.py:68: in __init__
    super(ImageLayerArtist, self).__init__(axes, viewer_state,
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/image/layer_artist.py:31: in __init__
    super(BaseImageLayerArtist, self).__init__(axes, viewer_state,
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/matplotlib/layer_artist.py:18: in __init__
    super(MatplotlibLayerArtist, self).__init__(viewer_state,
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/common/layer_artist.py:25: in __init__
    self.state = layer_state or self._layer_state_cls(viewer_state=viewer_state,
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue_jupyter/bqplot/image/state.py:34: in __init__
    super(BqplotImageLayerState, self).__init__(*args, **kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/image/state.py:550: in __init__
    self._update_attribute()
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/image/state.py:559: in _update_attribute
    self.attribute_att_helper.set_multiple_data([self.layer])
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/core/data_combo_helper.py:322: in set_multiple_data
    self.refresh()
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/core/data_combo_helper.py:379: in refresh
    self.choices = choices
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/core/data_combo_helper.py:84: in choices
    with delay_callback(self.state, self.selection_property):
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/echo/core.py:538: in __exit__
    p.notify(*args)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/utils/matplotlib.py:170: in wrapper
    return func(*args, **kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/matplotlib/state.py:38: in notify
    super(DeferredDrawSelectionCallbackProperty, self).notify(*args, **kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/echo/core.py:125: in notify
    cback(new)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/image/state.py:608: in _update_attribute_display_unit_choices
    self.attribute_display_unit = component.units
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/echo/core.py:263: in __setattr__
    self._notify_global(**{attribute: value})
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/utils/matplotlib.py:170: in wrapper
    return func(*args, **kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/viewers/matplotlib/state.py:319: in _notify_global
    super(MatplotlibLayerState, self)._notify_global(*args, **kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/echo/core.py:258: in _notify_global
    callback(**kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/core/state_objects.py:206: in _update_values
    self.update_values(**properties)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/core/state_objects.py:349: in update_values
    lower, upper = converter.to_unit(self.data,
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/core/units.py:31: in to_unit
    return self.converter_helper.to_unit(data, cid, values, original_units, target_units)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/jdaviz/app.py:98: in to_unit
    spec = data.get_object(cls=Spectrum1D)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue/core/data.py:298: in get_object
    return handler.to_object(self, **kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/glue_astronomy/translators/spectrum1d.py:287: in to_object
    return Spectrum1D(**data_kwargs, **kwargs)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/specutils/spectra/spectrum1d.py:277: in __init__
    if hasattr(self.wcs, "spectral"):
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/astropy/wcs/wcs.py:3488: in spectral
    return self.sub([WCSSUB_SPECTRAL])  # Defined by C-ext
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/astropy/wcs/wcs.py:656: in sub
    copy = self.deepcopy()
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/astropy/wcs/wcs.py:653: in deepcopy
    return copy.deepcopy(self)
/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/copy.py:143: in deepcopy
    y = copier(memo)
../../.tox/py312-test-devdeps/lib/python3.12/site-packages/astropy/wcs/wcs.py:626: in __deepcopy__
    deepcopy(self.wcs, memo),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

x =        flag: 137
      naxis: 0
      crpix: 0x5632efcd0720
            
         pc: 0x5632efcbf970
      cdelt: 0x56...(remainder unused)
    isGrism: 0
        err: 0x0
     spxX2P: 0x0
     spxP2S: 0x0
     spxS2P: 0x0
     spxP2X: 0x0

memo = {}, _nil = []

    def deepcopy(x, memo=None, _nil=[]):
        """Deep copy operation on arbitrary Python objects.
    
        See the module's __doc__ string for more info.
        """
    
        if memo is None:
            memo = {}
    
        d = id(x)
        y = memo.get(d, _nil)
        if y is not _nil:
            return y
    
        cls = type(x)
    
        copier = _deepcopy_dispatch.get(cls)
        if copier is not None:
            y = copier(x, memo)
        else:
            if issubclass(cls, type):
                y = _deepcopy_atomic(x, memo)
            else:
                copier = getattr(x, "__deepcopy__", None)
                if copier is not None:
>                   y = copier(memo)
E                   RuntimeError: NULL error object in wcslib

/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/copy.py:143: RuntimeError
=========================== short test summary info ============================
FAILED ../../.tox/py312-test-devdeps/lib/python3.12/site-packages/jdaviz/configs/cubeviz/plugins/tests/test_cubeviz_aperphot.py::test_cubeviz_aperphot_cube_orig_flux - RuntimeError: NULL error object in wcslib

🐱

@rosteen
Copy link
Collaborator

rosteen commented Apr 16, 2024

A quick search turned up the same error getting solved in 2014 in astropy, don't see how that could be related though

@rosteen
Copy link
Collaborator

rosteen commented Apr 16, 2024

Here's a diff between the pip freeze in a passing recent test and a failing one (left is good, right is bad):

2d1
< appnope==0.1.4
20c19
< backports.tarfile==1.0.0
---
> backports.tarfile==1.1.0
26c25
< casa-formats-io==0.2.2
---
> casa-formats-io==0.2.3
31a31
> colorama==0.4.6
45c45
< fast-histogram==0.12
---
> fast-histogram==0.13
52c52
< glue-core==1.18.0
---
> glue-core==1.19.0
67d66
< importlib_resources==6.4.0
75c74
< ipython==8.18.1
---
> ipython==8.23.0
86c85
< jdaviz @ file:///Users/runner/work/jdaviz/jdaviz/.tox/.tmp/package/1/jdaviz-3.10.dev23%2Bg9d85904a.tar.gz#sha256=2d299235d9a012050d187bf708ad2346416b1c7550e6ff9b633e7aaf63f613a0
---
> jdaviz @ file:///D:/a/jdaviz/jdaviz/.tox/.tmp/package/1/jdaviz-3.10.dev17%2Bg5c858bb7.tar.gz#sha256=6b07c579ef2dfcb0e8854034dcec55ada248a56391ee93112b61673ed022506e
120c119
< networkx==3.2.1
---
> networkx==3.3
131,132c130
< pexpect==4.9.0
< photutils==1.11.0
---
> photutils==1.12.0
139d136
< ptyprocess==0.7.0
161a159,161
> pywin32==306
> pywin32-ctypes==0.2.2
> pywinpty==2.0.13
168c168
< regions==0.8
---
> regions==0.9
173c173
< scikit-image==0.22.0
---
> scikit-image==0.23.1
185c185
< specutils==1.13.0
---
> specutils==1.14.0
187,188c187,188
< stdatamodels==1.10.0
< synphot==1.3.post0
---
> stdatamodels==1.10.1
> synphot==1.4.0
211d210
< y-py==0.5.4

@pllim
Copy link
Contributor Author

pllim commented Apr 16, 2024

The test case seems very broken on my machine but I need someone else to confirm. Please run this locally with Jdaviz from main.

  1. Does the image viewer show anything? It should but blank for me.
  2. Can you move the spectrum slice thingy by clicking it? It is not responsive for me.
  3. Does mouseover work on any viewer? It does not for me.

This test case used to work fine. Now it does not but I don't see any traceback in the notebook nor snackbar.

import numpy as np
from astropy.io import fits

from jdaviz import Cubeviz


def image_cube_hdu_obj_microns():
    # Basic rectangle ramp for aperture photometry test.
    a = np.zeros((8, 9, 10)).astype(np.float32)  # (nz, ny, nx)
    for i in range(8):
        a[i, :5, :3] = i + 1
    flux_hdu = fits.ImageHDU(a)
    flux_hdu.name = 'FLUX'

    uncert_hdu = fits.ImageHDU(np.zeros((8, 9, 10)).astype(np.float32))
    uncert_hdu.name = 'ERR'

    mask_hdu = fits.ImageHDU(np.ones((8, 9, 10)).astype(np.uint16))
    mask_hdu.name = 'MASK'

    wcs = {
        'WCSAXES': 3, 'CRPIX1': 38.0, 'CRPIX2': 38.0, 'CRPIX3': 1.0,
        'CRVAL1': 205.4384, 'CRVAL2': 27.004754, 'CRVAL3': 4.890499866509344,
        'CTYPE1': 'RA---TAN', 'CTYPE2': 'DEC--TAN', 'CTYPE3': 'WAVE',
        'CUNIT1': 'deg', 'CUNIT2': 'deg', 'CUNIT3': 'um',
        'CDELT1': 3.61111097865634E-05, 'CDELT2': 3.61111097865634E-05, 'CDELT3': 0.001000000047497451,  # noqa
        'PC1_1 ': -1.0, 'PC1_2 ': 0.0, 'PC1_3 ': 0,
        'PC2_1 ': 0.0, 'PC2_2 ': 1.0, 'PC2_3 ': 0,
        'PC3_1 ': 0, 'PC3_2 ': 0, 'PC3_3 ': 1,
        'DISPAXIS': 2, 'VELOSYS': -2538.02,
        'SPECSYS': 'BARYCENT', 'RADESYS': 'ICRS', 'EQUINOX': 2000.0,
        'LONPOLE': 180.0, 'LATPOLE': 27.004754,
        'MJDREFI': 0.0, 'MJDREFF': 0.0, 'DATE-OBS': '2014-03-30'}

    flux_hdu.header.update(wcs)
    flux_hdu.header['BUNIT'] = '1E-17 erg*s^-1*cm^-2*Angstrom^-1'

    uncert_hdu.header['BUNIT'] = '1E-17 erg*s^-1*cm^-2*Angstrom^-1'

    return fits.HDUList([fits.PrimaryHDU(), flux_hdu, uncert_hdu, mask_hdu])

cubeviz = Cubeviz()

cubeviz.load_data(image_cube_hdu_obj_microns(), data_label="test")

cubeviz.show()

@rosteen
Copy link
Collaborator

rosteen commented Apr 16, 2024

The test passes for me locally if I downgrade glue-core to 1.18.0 from 1.19.0.

@pllim
Copy link
Contributor Author

pllim commented Apr 16, 2024

argh https://github.com/glue-viz/glue/releases/tag/v1.19.0 was released 6 hours ago. Maybe you found the culprit!

@astrofrog
Copy link
Collaborator

The problematic commit is glue-viz/glue@9f7929d - will investigate more

@pllim
Copy link
Contributor Author

pllim commented Apr 16, 2024

Superseded by #2811

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants