Background subtraction in spectral extraction (cubeviz)

[kecnry]

Description

This pull request implements support for background subtraction in cubeviz's spectral extraction.

Since this has now gone far beyond "collapsing", this PR also takes this and the 4.0 major version bump as an excuse to rename collapse_to_spectrum(...) to extract(...) and then also adds a new extract_bg_spectrum(...) (using the same name as in specviz2d in case we ever do want to support extracting the background as well... we could consider just naming this to extract_background(...) if we don't want to account for that possibility down the road).

Screen.Recording.2024-05-07.at.1.49.12.PM.mov

TODO:

• live preview of background spectrum
• ability to export background spectrum independently (in same way as is done in specviz2d)
• determine what to do for sum case with different sized subsets (possibly by rescaling the background spectrum by the relative areas - *= aperture_area / background_area?)
• test coverage
• investigate glitchy disappearing of preview markers - this seems to have been happening before as well so might be better as a follow-up effort

Change log entry

• Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
  list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
  should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
• Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
• Do the proposed changes follow the STScI Style Guides?
• Are tests added/updated as required? If so, do they follow the STScI Style Guides?
• Are docs added/updated as required? If so, do they follow the STScI Style Guides?
• Did the CI pass? If not, are the failures related?
• Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
• After merge, any internal documentations need updating (e.g., JIRA, Innerspace)?

[codecov]

Codecov Report

Attention: Patch coverage is 94.44444% with 8 lines in your changes missing coverage. Please review.

Project coverage is 88.69%. Comparing base (1c5589e) to head (ba4d660).
Report is 154 commits behind head on main.

Files	Patch %	Lines
...plugins/spectral_extraction/spectral_extraction.py	92.38%	8 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2859   +/-   ##
=======================================
  Coverage   88.68%   88.69%           
=======================================
  Files         109      109           
  Lines       16996    17044   +48     
=======================================
+ Hits        15073    15117   +44     
- Misses       1923     1927    +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[camipacifici]

The workflow is nice and the orange highlights in the viewers are nice too, but it looks like it is inverting flux and background when subtracting.

If I set 'none' for the background, the extracted spectrum is not flipped.

[camipacifici]

Another thing I noticed:

• create subset 1 for source and subset 2 for background
• use spectral extraction plugin with subset 1 (not using subset 2 for background now...see above). i see the orange preview in the spectrum viewer.
• hit extract -> spectrum turns red and preview disappears
• modify subset 1 -> preview does not reappear following the newly defined subset 1
  I would have intuitively expect the preview to come back, but please let me know if this is intentional and we can discuss.
  Thank you!

[kecnry]

but it looks like it is inverting flux and background when subtracting.

Is it possible that there are negative surface brightnesses in either of those apertures?

I would have intuitively expect the preview to come back, but please let me know if this is intentional and we can discuss.

There are some known bugs with the preview disappearing that need to be addressed yet - but I'm quite sure that they are not introduced here and so probably don't need to block this PR (but maybe should block the next release).

[gibsongreen]

Everything looks good to me and is working as expected!

Unrelated to this PR, I was moving and updating subsets to check the auto-updating. If I have a spectral subset, the spectrum created matches. If I update the color of the spatial subset in plot options, the spectrum doesn't update. I think we talked about this with Patrick that this was the intended behavior, so leaving this here just to make sure!

[kecnry]

If I have a spectral subset, the spectrum created matches. If I update the color of the spatial subset in plot options, the spectrum doesn't update. I think we talked about this with Patrick that this was the intended behavior, so leaving this here just to make sure!

Yes, this was decided to be the desired behavior.

[bmorris3]

This works great when aperture_method == sum. When I try it with mean, I see an all-zero extraction. Do you see that on your end?

Renormalization feature missing in sum mode

I know this isn't in the ticket, but I think we should make a follow up ticket for one more feature. If you're using a background region, you almost always want to subtract that background such that the area of the source and background apertures don't influence the background measurement. When aperture_method == 'mean' this isn't a problem, but when aperture_method == 'sum', you essentially want to renormalize by the areas of each region, like:

(bg to subtract from the source's aperture extraction) = 
(bg sum) * (source region area) / (background region area)

so that the background gets applied correctly to each source aperture pixel. cc @camipacifici.

[bmorris3]

Should we specify the float size? I don't expect it matters much, but if it ever does matter, float promotion/demotion might be hard to trace back to this line in the future.

[kecnry]

I think I just migrated this over from your original implementation. What changes would you suggest here?

[bmorris3]

I think you can just drop the dtype kwarg.

[bmorris3]

Should we specify the float size? I don't expect it matters much, but if it ever does matter, float promotion/demotion might be hard to trace back to this line in the future.

[camipacifici]

@bmorris3 naively I thought that the background spectrum was subtracted per spaxel before extraction. If that is not the case, then yes, that normalization is necessary.

[kecnry]

@bmorris3 @camipacifici - normalization by aperture area is currently handled here. Is this different from what you had in mind?

[kecnry]

@camipacifici - I think I found and fixed the bug resulting in the unexpected negative spectrum (it was incorrectly still using the aperture mask but weighing by the relative area ratios). Please let me know if it seems to act as expected on your end now.

EDIT: looking into why this seems to have caused test failures in model fitting 🤔

[bmorris3]

Thanks, this is behaving as expected now. As for this comment, take it or leave it. We're unlikely to see a difference.