SLITLESS FLAT

[debora-pe]

This a large work aimed at introducing the creation of slitless flat in the reduction workflow. Below is the summary of changes.

• slitless pixelflat are generated during the reduction and saved in pypeit/data/static_calib and can be archived for future uses

• added a script that show the archived pixelflat in ginga

• archived 3 new keck_hires slitless pixflat and 1 new keck_lris slitless pixelflat

• when multiple images are combined during the reduction, now there is the possibility (including a new parameter) to scale the images to the mean values

• improved frame typing and configuration setup for keck_hires and keck_lris

• fix a bug with argsort

• resolved the non_linear_counts TODOs

STILL MISSING: Documentation
Dev Suite PR: pypeit/PypeIt-development-suite#324

[debora-pe]

Tests pass (except the usual setup gui unrelated failures)

Test Summary
--------------------------------------------------------
--- PYTEST PYPEIT UNIT TESTS PASSED  256 passed, 68 warnings in 342.60s (0:05:42) ---
--- PYTEST UNIT TESTS FAILED  9 failed, 142 passed, 168 warnings, 1 error in 1680.99s (0:28:00) ---
--- PYTEST VET TESTS PASSED  61 passed, 97 warnings in 5193.25s (1:26:33) ---
--- PYPEIT DEVELOPMENT SUITE PASSED 242/242 TESTS  ---
Testing Started at 2024-06-27T02:57:22.928246
Testing Completed at 2024-06-27T18:24:55.315692
Total Time: 15:27:32.387446

[kbwestfall]

Great work! I have lots of minor comments and two major ones. The major ones are:

• Given that you've already done the work, I'm not sure it's worth going back on it, but there's a structural comment I wanted to bring up. To me, it would seem cleaner to have a separate script that builds the slitless pixel flats instead of trying to integrate the creation of these flats within the overall workflow of run_pypeit. There are some gymnastics you have to go through to get the latter to work that I think would be better separated out into more dedicated scripts. A dedicated script would also more cleanly separate the interaction with the cache. I.e., we could change to a mode where run_pypeit is only using slitless flats that can either be downloaded or installed in the local cache. Is a separate script something you thought through?
• I think we should create a new DataContainer subclass for these slitless flats.

The last thing I wanted to highlight is that I think we should be making more use of the get_file_path function for interaction with the cache. I.e., there are a few places in the code where you effectively repeat code that is the purpose of that function. The trouble is that you will often need to wrap it in a try/except block. If this becomes burdensome, we could alter that function to avoid that.

Let me know what you think!

[kbwestfall]

I'm not sure I understand the general application of this function. Given that it's only used for slitless flats right now, do we need a general function?

[debora-pe]

This is actually spectrograph dependent, this is why it's here. It is currently only used for keck_hires, but I tried to make it generic enough so that it could be used for other purposes (still spectrograph-specific). I have added more in the docstring.

[profxj]

and few comments to chew on. thx!

[profxj]

this should probably be its own stand-alone-method somewhere

[debora-pe]

I addressed all the minor-ish comments.
I have left:

• New datamodel for SlitlessFlats
• The cache stuff

[debora-pe]

This is actually spectrograph dependent, this is why it's here. It is currently only used for keck_hires, but I tried to make it generic enough so that it could be used for other purposes (still spectrograph-specific). I have added more in the docstring.

[debora-pe]

@kbwestfall @profxj I am done addressing the comments. Please take a look.

@kbwestfall I was not able to create a Datacontainer for SlitlessFlat (too complicated), but I re-organized the code and created a SlitlessFlat class that holds the code for crating the slitless flats.
I was also able to make adjustments to the cache for this PR, by adding a small change to get_file_path. Let me know what you think.

[debora-pe]

@kbwestfall @profxj I was finally able to run the tests (Google Drive is giving me hard time). ALL GOOD!

Test Summary
--------------------------------------------------------
--- PYTEST PYPEIT UNIT TESTS PASSED  256 passed, 66 warnings in 333.24s (0:05:33) ---
--- PYTEST UNIT TESTS FAILED  2 failed, 149 passed, 168 warnings in 1732.46s (0:28:52) ---
--- PYTEST VET TESTS FAILED  1 failed, 61 passed, 98 warnings in 5490.23s (1:31:30) ---
--- PYPEIT DEVELOPMENT SUITE FAILED 1/242 TESTS  ---
Failed tests:
    keck_hires/J0100+2802_N255Hr_RED_C2_ECH_0.74_XD_1.39_1x3 pypeit
Skipped tests:
Testing Started at 2024-08-22T16:11:52.047202
Testing Completed at 2024-08-23T10:03:18.531838
Total Time: 17:51:26.484636

NOTE:

• The 2 UNIT TESTS FAILED are due to a problem that my machine has with the setup_gui tests
• The reduction of keck_hires/J0100+2802_N255Hr_RED_C2_ECH_0.74_XD_1.39_1x3 didn't fail but just stopped. I re-run it without changing anything and run through fine

  Running tests on the following instruments:
      keck_hires
  
  Reducing data from keck_hires for the following setups:
      J0100+2802_N255Hr_RED_C2_ECH_0.74_XD_1.39_1x3
  
   0 passed/ 0 failed/ 0 skipped STARTED keck_hires/J0100+2802_N255Hr_RED_C2_ECH_0.74_XD_1.39_1x3 pypeit
   1 passed/ 0 failed/ 0 skipped PASSED  keck_hires/J0100+2802_N255Hr_RED_C2_ECH_0.74_XD_1.39_1x3 pypeit
    --- PYPEIT DEVELOPMENT SUITE PASSED 1/1 TESTS  ---
    
    Test Summary
    --------------------------------------------------------
    --- PYPEIT DEVELOPMENT SUITE PASSED 1/1 TESTS  ---
    Testing Started at 2024-08-23T11:17:30.449074
    Testing Completed at 2024-08-23T11:55:37.149999
    Total Time: 0:38:06.700925
  

• The 1 VET TESTS FAILED was due to the keck_hires missing reduction, I re-run it and all good

   pytest vet_tests/test_wavelengths.py::test_keck_hires
  ==================== test session starts =====================
  platform darwin -- Python 3.11.5, pytest-7.4.3, pluggy-1.3.0
  PyQt6 6.6.1 -- Qt runtime 6.6.1 -- Qt compiled 6.6.1
  rootdir: /Users/dpelliccia/PypeIt/PypeIt-development-suite
  plugins: astropy-0.11.0, hypothesis-6.92.0, qt-4.4.0, asdf-3.0.1, remotedata-0.4.1, cov-4.1.0, filter-subpackage-0.1.2, astropy-header-0.2.2, mock-3.12.0, doctestplus-1.1.0, arraydiff-0.6.1
  collected 1 item                                             
  
  vet_tests/test_wavelengths.py .                        [100%]
  
  ===================== 1 passed in 11.84s =====================
  

[kbwestfall]

Sorry for the slow response on this one! This looks good and I'm approving. Two things for future PRs:

• I'd like to help see if I can get the SlitlessFlat class to subclass from DataContainer.
• I worry a little about adding ~200MB in fits images, but these should not change often meaning that they shouldn't blow up the size of the repo. We should keep track of this and possibly remove them (and clean-up the commit history) if the repo gets too big.

[profxj]

excellent! great to have this