JP-3746: Improve background combination for NIRSpec MOS master background

[hayescr]

Resolves JP-3746

Closes #8788

This PR adds an option to do a pixel-by-pixel "sigma clipping" with a median and median absolute deviation on MOS background spectra to remove outliers and contaminated spectra when combining inputs into a single spectrum (in combine_1d). This PR also adds an option to perform a median filter on the 1D master background spectrum produced in both master_background and master_background MOS (by default this is off).

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

Attention: Patch coverage is 93.97590% with 5 lines in your changes missing coverage. Please review.

Project coverage is 64.52%. Comparing base (4d8e589) to head (72cb6f0).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...st/master_background/master_background_mos_step.py	44.44%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8932      +/-   ##
==========================================
- Coverage   64.52%   64.52%   -0.01%     
==========================================
  Files         375      375              
  Lines       38738    38739       +1     
==========================================
  Hits        24997    24997              
- Misses      13741    13742       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[melanieclarke]

This looks good to me. Thanks for all the extra test coverage for master_background_mos! That was very much needed.

Running some selected regtests locally, I see a significant improvement to the masterbg1d spectrum in test_nirspec_masterbackground.py::test_nirspec_mos_mbkg. I see no changes in the other NIRSpec master background tests, as expected.

I do see some new diffs in the test_niriss_wfss.py c1d products, due to flux=NaN being ignored instead of just DQ=1. I see the same diffs in regtests for #8961 (which sets DQ=1 for flux=NaN), so it seems fine to make that change here.

A few questions and suggestions below.

[melanieclarke]

Everything looks good to me now, other than the potential conflict with #8847. I will update from main and run the full set of regression tests.

Tests are running here:
https://github.com/spacetelescope/RegressionTests/actions/runs/11902214724

[melanieclarke]

Regression tests results look as expected.

There are some minor changes for MIRI master background tests and WFSS - both are due to the change to combine_1d to ignore NaN flux values. Changes are also as expected downstream of NIRSpec MOS master background in spec2 tests: background correction looks cleaner with new sigma clipping defaults.

Since this PR is ready, let's go ahead and get it in ahead of #8847. @penaguerrero - you'll need to rebase and deconflict.