Switch exposure pipeline to use ModelLibrary instead of a list of models

[braingram]

The PR makes a few updates to the exposure pipeline. The major feature is to use a ModelLibrary instead of a list of models. This allows:

• removing the custom asn parsing in the pipeline
• the tweakreg (and other step) cal_step status to be correctly marked by stpipe when steps are skipped

This PR also fixes a bug uncovered while working on this PR (#1523) where an all saturated input resulted in:

• the pipeline skipping processing of all subsequence models in the input association
• failing to return an all zeroed ImageModel

Regtests: https://github.com/spacetelescope/RegressionTests/actions/runs/11861738024
show 2 expected failures.
The first:

AssertionError: 
  Diff report for:
      result file: /runner/_work/_temp/pytest_basetemp/popen-gw0/test_pipeline_suffix0/r0000101001001001001_0001_wfi01_star.asdf
          model type: ImageModel
      truth file: /runner/_work/_temp/pytest_basetemp/popen-gw0/test_pipeline_suffix0/truth/r0000101001001001001_0001_wfi01_star.asdf
          model type: ImageModel

{'dictionary_item_added': ["root['roman']['meta']['asn']", "root['roman']['meta']['exptype']", "root['roman']['meta']['group_id']"],
'values_changed': {"root['roman']['meta']['cal_step']['tweakreg']": {'new_value': 'SKIPPED',
'old_value': 'INCOMPLETE'}}}
assert False

is a byproduct of using ModelLibrary. The output files for elp now contain asn information and have tweakreg correctly marked as 'SKIPPED' for test_pipeline_suffix.

The second failure is due to this PR fixing #1523. test_all_saturated_against_truth fails since this PR restores the old behavior where an all saturated input produces a cal files with all zeros and all subsequent steps skipped. As described in #1523 several updates to the elp resulted in this behavior being lost and create_fully_saturated_zeroed_image (which was added in #541) being unused. With this PR create_fully_saturated_zeroed_image is once again used. To hopefully avoid a future regression I added some additional tests for the "all saturated" input file that:

• checks that the output is an ImageModel
• checks that the output arrays are all zeros

Finally I added a few unit tests that run the elp with all steps skipped and no reference files. These are relatively fast (good candidates for unit tests) and check that when provided an input model the pipeline returns a model etc (and not a list as it does on main).

Closes #1523

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. 24Q4_B15 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.associations.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.mosaic_pipeline.rst
• changes/<PR#>.patch_match.rst

steps

• changes/<PR#>.dq_init.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.jump_detection.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.flux.rst
• changes/<PR#>.source_detection.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.resample.rst
• changes/<PR#>.source_catalog.rst

[codecov]

Codecov Report

Attention: Patch coverage is 75.00000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 77.55%. Comparing base (d68a9ca) to head (b6f3799).
Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
romancal/pipeline/exposure_pipeline.py	75.00%	12 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1525      +/-   ##
==========================================
+ Coverage   76.81%   77.55%   +0.73%     
==========================================
  Files         118      118              
  Lines        7743     7721      -22     
==========================================
+ Hits         5948     5988      +40     
+ Misses       1795     1733      -62     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[schlafly]

Thanks. This looks good.

You also deleted some orphaned fully saturated exposure handling here. I think following your investigation we probably do want to keep and start using that code again, rather than deleting it?
#1523
Maybe just to say that deleting create_fully_saturated_zeroed_image needn't be part of this PR.

@ddavis-stsci , this refactors some of the pipeline handling to use ModelLibrary more uniformly, so you may want to weigh in here.

[braingram]

Converted to draft to attempt to also address #1523 and some other elp cleanup.

[ddavis-stsci]

Besides the all saturated case not working it seems fine.

[braingram]

Besides the all saturated case not working it seems fine.

Thanks for giving this a look. Unfortunately it was draft at the time and not yet finished. The major changes that occurred after your comment were:

• fixing the pipeline handling of all saturated input (this results in regression test differences that will need to be okified)
• adding unit tests for the elp input/output types

I re-requested your review in light of these changes.

[schlafly]

This looks good to me. I left some comments about the tweakreg / all saturated interaction.

Ultimately I think this case is rare enough and weird enough that probably we should be suspect of the whole exposure if any individual SCA is fully saturated, so I don't think it's a big deal. However, it surprised me a little to skip tweakreg for the whole array because of a single detector.

[schlafly]

Thanks Brett, approving!

[braingram]

Thanks! I rebased and started a new regtest run here:
https://github.com/spacetelescope/RegressionTests/actions/runs/12188643274
@schlafly feel free to merge this whenever the regtests finish and you're ready to test the new okify script (or let me know if you'd like me to do that).