Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QL spectra __add__ method not order independent #410

Open
nicHoch opened this issue Nov 11, 2024 · 0 comments
Open

QL spectra __add__ method not order independent #410

nicHoch opened this issue Nov 11, 2024 · 0 comments
Assignees
Labels
bug Something isn't working investigation investigation
Milestone

Comments

@nicHoch
Copy link
Collaborator

nicHoch commented Nov 11, 2024

the L0 processing step for generating QL spectra data is not order independent of the parent LB fits files.

in general the GenericProduct.add method should be symmetric a + b = b + a but this is not the case as the end2end testing shows the finial L0 product fits file is dependent of the order the parent LB files are processed.

needs fixing and investigation if this needs a reprocessing of existing FITS files.

example input:

solo_LB_stix-21-6-32.zip

example script to investigate:

import zipfile
import warnings
import urllib.request
from pprint import pformat
from pathlib import Path

import pytest

from astropy.io.fits.diff import FITSDiff

from stixcore.config.config import CONFIG
from stixcore.data.test import test_data
from stixcore.ephemeris.manager import Spice, SpiceKernelManager
from stixcore.idb.manager import IDBManager
from stixcore.io.RidLutManager import RidLutManager
from stixcore.processing.LBtoL0 import Level0
from stixcore.processing.pipeline import PipelineStatus
from stixcore.products.product import Product
from stixcore.soop.manager import SOOPManager
from stixcore.util.logging import get_logger
from stixcore.util.scripts.end2end_testing import end2end_pipeline

def test_e2e_21_6_32(out_dir):
    _spm = SpiceKernelManager(test_data.ephemeris.KERNELS_DIR)
    Spice.instance = Spice(_spm.get_latest_mk())

    # pinpoint the api files location
    CONFIG.set('SOOP', 'soop_files_download', str(test_data.soop.DIR))

    SOOPManager.instance = SOOPManager(test_data.soop.DIR, mock_api=True)

    idbpath = Path(__file__).parent.parent.parent / "data" / "idb"
    IDBManager.instance = IDBManager(idbpath)  # force_version="2.26.35")

    RidLutManager.instance = RidLutManager(Path(CONFIG.get('Publish', 'rid_lut_file')), update=True)

    PipelineStatus.log_setup()
    
    f1 = Path("/data/stix/out/test/e2e_21_6_32/solo_LB_stix-21-6-32_0678153600_V02.fits")
    f2 = Path("/data/stix/out/test/e2e_21_6_32/solo_LB_stix-21-6-32_0678240000_V02.fits")
 
    out_dir_o1 = out_dir / "o1"
    out_dir_o2 = out_dir / "o2"

    l0_proc_o1 = Level0(out_dir_o1, out_dir_o1)
    l0_files_o1 = l0_proc_o1.process_fits_files(files=[f2, f1])

    l0_proc_o2 = Level0(out_dir_o2, out_dir_o2)
    l0_files_o2 = l0_proc_o2.process_fits_files(files=[f1, f2])

    for fits_1 in l0_files_o1:
        # find corresponding original file
        fits_2 = next(ofits for ofits in l0_files_o2 if ofits.name == fits_1.name)
        diff = FITSDiff(fits_1, fits_2, atol=0.00001, rtol=0.00001,
                        ignore_keywords=['CHECKSUM', 'DATASUM', 'DATE',
                                         'VERS_SW', 'VERS_CFG', 'HISTORY'])
        if not diff.identical:
            print(diff.report())
            assert diff.identical

@nicHoch nicHoch added this to the v1.5.0 milestone Nov 11, 2024
@nicHoch nicHoch added bug Something isn't working investigation investigation labels Nov 11, 2024
@nicHoch nicHoch modified the milestones: v1.5.0, v1.6.0 Nov 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working investigation investigation
Projects
None yet
Development

No branches or pull requests

2 participants