rcal-936 Update skycell_asn docs and add skycell_asn as a script at install time #1471

ddavis-stsci · 2024-10-23T12:28:34Z

This PR updates the read the docs to add information about the skycell_asn script to generate associations based on sky cells in the patch table from INS.

The also updates the pyproject.toml file to install skycell_asn as script.

Tasks

request a review from someone specific, to avoid making the maintainers review every PR
add a build milestone, i.e. 24Q4_B15 (use the latest build if not sure)
Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
- write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
- [N/A] update or add relevant tests
- [N/A] update relevant docstrings and / or docs/ page
- [N/A] start a regression test and include a link to the running job (click here for instructions)
  - Do truth files need to be updated ("okified")?
    - after the reviewer has approved these changes, run okify_regtests to update the truth files
if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

changes/<PR#>.general.rst: infrastructure or miscellaneous change
changes/<PR#>.docs.rst
changes/<PR#>.stpipe.rst
changes/<PR#>.associations.rst
changes/<PR#>.scripts.rst
changes/<PR#>.mosaic_pipeline.rst
changes/<PR#>.patch_match.rst

steps

changes/<PR#>.dq_init.rst
changes/<PR#>.saturation.rst
changes/<PR#>.refpix.rst
changes/<PR#>.linearity.rst
changes/<PR#>.dark_current.rst
changes/<PR#>.jump_detection.rst
changes/<PR#>.ramp_fitting.rst
changes/<PR#>.assign_wcs.rst
changes/<PR#>.flatfield.rst
changes/<PR#>.photom.rst
changes/<PR#>.flux.rst
changes/<PR#>.source_detection.rst
changes/<PR#>.tweakreg.rst
changes/<PR#>.skymatch.rst
changes/<PR#>.outlier_detection.rst
changes/<PR#>.resample.rst
changes/<PR#>.source_catalog.rst

codecov · 2024-10-23T12:59:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.15%. Comparing base (0a530f2) to head (ab63e4e).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1471   +/-   ##
=======================================
  Coverage   76.15%   76.15%           
=======================================
  Files         115      115           
  Lines        7638     7638           
=======================================
  Hits         5817     5817           
  Misses       1821     1821

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

schlafly · 2024-10-23T14:38:07Z

docs/roman/associations/skycell_asn.rst

+To give a more concrete example we'll use the APT example program for the
+High Latitude Wide Angle Survey (hlwas). The program is 00991, the execution plan number is 01,
+there are 12 passes, 5 segments, 4 observations and a various number of visits.
+For this example we want to select a single filter, say F158, and that is segment 003.


I'm not sure I follow this example. When I look at program 991 and pass 1 segment 3, I see a lot of exposures in different filters. Specifically, each visit is a linegap 4 dither, and those are grouped into observations of 32 pointings in a mosaic, and those observations are grouped into a segment with the F106, F129, F158, F184, and GRISM elements.

So I think the HLWAS example program has:

12 passes (but note there are really only 2 passes of imaging, which are the ones we care about and the other passes are purely spectroscopic)

169 segments per spectroscopic pass (basically, a big mosaic covering a lot of the sky)

5 observations per segment (F106, F129, F158, F184, GRISM)

32 visits per observation (a smaller mosaic covering a patch)

4 exposures per visit (linegap 4 dither)
So to select out only F158 I think you need to get observation 3.

It does look to me like it's required that all exposures in the same "observation" use the same filter, which I hadn't appreciated before.

I've updated the numbers, I was trying to simplify this for the docs. Is there another program you know of that might be clearer?

The HLWAS makes sense because it's one people will have heard of. There are much smaller programs that one could use but then I just think they will be unknown to people. In general, I am not sure how well publicized the example APT programs are, but I did think it was useful to have a "worked example"; this was a reminder for me that the "execution plan" stuff is a weird detail that is pretty extraneous to our planning. It's possible we would want to ?? over that as well, but then I don't know what base name I would use, so I didn't suggest anything there.

The new version still selects segment 3, but I think you need observation 3. i.e., the names need to be more like r0099101001001003001_*_cal.asdf, likewise the base names.

schlafly · 2024-10-23T14:47:22Z

docs/roman/associations/skycell_asn.rst

+
+.. code-block:: text
+
+		skycell_asn r0099101???003*_*_cal.asdf  -o  r0099101001003001001 --product-type pass


I think the right example here is more like:
skycell_asn r0099101001???003*_*_cal.asdf -o r0099101001 --product-type pass
i.e., making the path specify both the execution plan and the pass (01001) and then using the three ??? to go over all segments before specifying observation 003 to get a particular filter. Then the output root should include the activity / pass information but nothing else, and the script figures out the filter?

good catch, I updated this from the visit and forgot to change that.

ddavis-stsci · 2024-10-23T15:31:08Z

updated segment to observation

…

On 10/23/24 11:07 AM, Eddie Schlafly wrote: ***@***.**** commented on this pull request. ------------------------------------------------------------------------ In docs/roman/associations/skycell_asn.rst <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/1471*discussion_r1812994823__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!1yJoUCc1sFm0Inmvx7xjuMkHP4-AN7oF_bwfnZZ5wwPvUpS9aXoowpbgTzwE3W352xeupx1M_BGu4KSgvnhAt_WL$>: > +and the Exposure_ID is a four digits designating the exposures + +eeee = Exposure number (within the visit) + +The detector is WFI01, WFI02, ... WFI18 + +The suffix indicates the processing level of the file and to create products based on the +skycells the suffix should generally be 'cal'. + +and the file name is constructed as, +rPPPPPCCAAASSSOOOVVV_eeee_detector_suffix.asdf + +To give a more concrete example we'll use the APT example program for the +High Latitude Wide Angle Survey (hlwas). The program is 00991, the execution plan number is 01, +there are 12 passes, 5 segments, 4 observations and a various number of visits. +For this example we want to select a single filter, say F158, and that is segment 003. The new version still selects segment 3, but I think you need observation 3. i.e., the names need to be more like r0099101001001003001_*_cal.asdf, likewise the base names. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/1471*discussion_r1812994823__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!1yJoUCc1sFm0Inmvx7xjuMkHP4-AN7oF_bwfnZZ5wwPvUpS9aXoowpbgTzwE3W352xeupx1M_BGu4KSgvnhAt_WL$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALXCXWK3V2J5Q737OP7YLR3Z463T7AVCNFSM6AAAAABQOWP2ISVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGOBZGEZDAOBRGA__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!1yJoUCc1sFm0Inmvx7xjuMkHP4-AN7oF_bwfnZZ5wwPvUpS9aXoowpbgTzwE3W352xeupx1M_BGu4KSgvoEKcYQ1$>. You are receiving this because you were assigned.Message ID: ***@***.***>

schlafly · 2024-10-23T15:43:50Z

docs/roman/associations/skycell_asn.rst

+that this is resampled 2-d imaging data. The release product name can be changed from the default
+by adding the optional argument --release-product <new name> to the command line.
+
+An analogous command to generate the pass level products, again setting segment to 003 to only select


Suggested change

An analogous command to generate the pass level products, again setting segment to 003 to only select

An analogous command to generate the pass level products, again setting observation to 003 to only select

schlafly

I marked one more case of segment -> observation in line, otherwise looks good.

for more information, see https://pre-commit.ci

rcal-936 Update docs and add skycell_asn as a script at install time

c6aa6dd

ddavis-stsci added documentation Improvements or additions to documentation associations labels Oct 23, 2024

ddavis-stsci added this to the 25Q1_B16 milestone Oct 23, 2024

ddavis-stsci self-assigned this Oct 23, 2024

ddavis-stsci requested a review from a team as a code owner October 23, 2024 12:28

github-actions bot added the dependencies Pull requests that update a dependency file label Oct 23, 2024

rcal-936 add changes

eb2e78d

ddavis-stsci force-pushed the rcal-936_dsd branch from b346ef3 to eb2e78d Compare October 23, 2024 12:32

schlafly reviewed Oct 23, 2024

View reviewed changes

rcal-936 Text updates based on comments

2008d3c

ddavis-stsci force-pushed the rcal-936_dsd branch from 9b55863 to 2008d3c Compare October 23, 2024 15:01

rcal-936 change segment to observation

2ebfe04

ddavis-stsci force-pushed the rcal-936_dsd branch from c3b22a1 to 2ebfe04 Compare October 23, 2024 15:29

schlafly reviewed Oct 23, 2024

View reviewed changes

schlafly approved these changes Oct 23, 2024

View reviewed changes

rcal-936 change segment to observation in last example

fea29ff

ddavis-stsci force-pushed the rcal-936_dsd branch from 93fc9e7 to fea29ff Compare October 23, 2024 15:49

[pre-commit.ci] auto fixes from pre-commit.com hooks

ab63e4e

for more information, see https://pre-commit.ci

ddavis-stsci merged commit e215f87 into spacetelescope:main Oct 23, 2024
31 checks passed

ddavis-stsci deleted the rcal-936_dsd branch October 23, 2024 16:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rcal-936 Update skycell_asn docs and add skycell_asn as a script at install time #1471

rcal-936 Update skycell_asn docs and add skycell_asn as a script at install time #1471

ddavis-stsci commented Oct 23, 2024

codecov bot commented Oct 23, 2024 •

edited

Loading

schlafly Oct 23, 2024

ddavis-stsci Oct 23, 2024

schlafly Oct 23, 2024

schlafly Oct 23, 2024

schlafly Oct 23, 2024

ddavis-stsci Oct 23, 2024

ddavis-stsci commented Oct 23, 2024 via email

schlafly Oct 23, 2024

schlafly left a comment


		.. code-block:: text

		skycell_asn r0099101???003__cal.asdf -o r0099101001003001001 --product-type pass

	An analogous command to generate the pass level products, again setting segment to 003 to only select
	An analogous command to generate the pass level products, again setting observation to 003 to only select

rcal-936 Update skycell_asn docs and add skycell_asn as a script at install time #1471

rcal-936 Update skycell_asn docs and add skycell_asn as a script at install time #1471

Conversation

ddavis-stsci commented Oct 23, 2024

Tasks

steps

codecov bot commented Oct 23, 2024 • edited Loading

Codecov Report

schlafly Oct 23, 2024

Choose a reason for hiding this comment

ddavis-stsci Oct 23, 2024

Choose a reason for hiding this comment

schlafly Oct 23, 2024

Choose a reason for hiding this comment

schlafly Oct 23, 2024

Choose a reason for hiding this comment

schlafly Oct 23, 2024

Choose a reason for hiding this comment

ddavis-stsci Oct 23, 2024

Choose a reason for hiding this comment

ddavis-stsci commented Oct 23, 2024 via email

schlafly Oct 23, 2024

Choose a reason for hiding this comment

schlafly left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 23, 2024 •

edited

Loading