diff --git a/doc/calibrations/flat.rst b/doc/calibrations/flat.rst index d224400051..7bc244b8a8 100644 --- a/doc/calibrations/flat.rst +++ b/doc/calibrations/flat.rst @@ -51,7 +51,7 @@ spectrograph that we have not included example screen-shots. Raw Flat --------- +++++++++ This is the processed and combined ``pixelflat`` image. Despite the name, it is not completely raw. @@ -60,7 +60,7 @@ This image should look like any one of your input ``pixelflat`` images. Pixel Flat ----------- +++++++++++ This is the normalized to unity image which is used to correct for pixel-to-pixel variations across the detector. @@ -77,17 +77,32 @@ true if there is limited flux at these ends (e.g. the data goes below the atmospheric cutoff). Illumination Flat ------------------ ++++++++++++++++++ This image should also have most values near unity, but there will be vertical coherence. And the edges (left/right) may fall off well below unity. Flat Model ----------- +++++++++++ This image should largely resemble the `Raw Flat`_. +pypeit_show_pixflat +------------------- + +In addition to ``pypeit_chk_flats``, if a custom pixel flat is provided by the user, +another script ``pypeit_show_pixflat`` is available to inspect it. The script is called as follows: + +.. code-block:: console + + pypeit_show_pixflat PYPEIT_LRISb_pixflat_B600_2x2_17sep2009_specflip.fits.gz + +The script usage can be displayed by calling the script with the ``-h`` option: + +.. include:: ../help/pypeit_show_pixflat.rst + + Troubleshooting =============== diff --git a/doc/calibrations/flat_fielding.rst b/doc/calibrations/flat_fielding.rst index 904239b05b..134714c3db 100644 --- a/doc/calibrations/flat_fielding.rst +++ b/doc/calibrations/flat_fielding.rst @@ -204,7 +204,8 @@ need to provide the matching flat field images in your In short, if ``use_pixelflat=True`` for *any* of your frame types, at least one of the data files in the :ref:`pypeit_file` :ref:`data_block` must -be labelled as ``pixelflat`` (unless you `Feed a PixelFlat`_). +be labelled as ``pixelflat``, or ``slitless_pixflat`` +(unless you `Feed a PixelFlat`_). And, if ``use_illumflat=True`` for *any* of your frame types, at least one of the data files in the @@ -216,26 +217,47 @@ frames for the pixel and illumination corrections. This is supported, but we recommend that you set the ``trace`` frames to be the same as the ``illumflat`` frames. +.. _generate-pixflat: + +Generate a Slitless PixelFlat +----------------------------- + +If a set of ``slitless_pixflat`` frames are available in the +:ref:`data_block` of the :ref:`pypeit_file`, PypeIt will generate +a slitless pixel flat (unless you `Feed a PixelFlat`_ instead) +during the main :ref:`run-pypeit`, and will apply it to frame +types that have ``use_pixelflat=True``. +The slitless pixel flat is generated separately for each detector +(even in the case of a mosaic reduction) and it is stored in a FITS +file in the reduction directory, with one extension per detector. +In addition to saving the file in your reduction directory, +the constructed pixelflat is saved to the PypeIt cache (see ref:`data_installation`). +This allows you to use the file for both current and future reductions. +To use this file in future reductions, the user should add the slitless +pixel flat file name to the :ref:`pypeit_file` as shown in `Feed a PixelFlat`_. + +If you generate your own slitless pixel flat, and you think it is generally +applicable for your instrument, please consider sharing it with the PypeIt Developers. + + Feed a PixelFlat ---------------- If you have generated your own pixel flat (or were provided one) and it is trimmed and oriented following the expected :ref:`pypeit-orientation`, -then you may feed this into PypeIt. This is the recommended approach -at present for :ref:`lrisb`. +then you may feed this into PypeIt. -And you perform this by modifying the :ref:`parameter_block`: - -.. TODO: IS THIS STILL THE CORRECT APPROACH? WHAT DO PEOPLE DO IF THEY DON'T -.. HAVE THE DEV SUITE? +To use the available PixelFlat, you need to modify the :ref:`parameter_block` like, e.g.: .. code-block:: ini - [calibrations] - [[flatfield]] - pixelflat_file = /Users/joe/python/PypeIt-development-suite/CALIBS/PYPEIT_LRISb_pixflat_B600_2x2_17sep2009.fits.gz + [calibrations] + [[flatfield]] + pixelflat_file = PYPEIT_LRISb_pixflat_B600_2x2_17sep2009_specflip.fits.gz + +If any of the frames in the :ref:`data_block` are labelled as ``pixelflat``, or ``slitless_pixflat``, +the provided pixel flat file will still be used instead of generating a new one. -None of the frames in the :ref:`data_block` should be labelled as ``pixelflat``. Algorithms ---------- diff --git a/doc/calibrations/image_proc.rst b/doc/calibrations/image_proc.rst index f64d467908..b80caecf80 100644 --- a/doc/calibrations/image_proc.rst +++ b/doc/calibrations/image_proc.rst @@ -42,8 +42,9 @@ where: - the quantity :math:`C=N_{\rm frames}\ c/s\prime=c/s` is the number of electron counts excited by photons hitting the detector, - :math:`1/s=N_{\rm frames}/s\prime` is a factor that accounts for the number - of frames contributing to the electron counts, and the relative - throughput factors (see below) that can be measured from flat-field frames, + of frames contributing to the electron counts (`N_{\rm frames}`), and (`s\prime`) the relative + throughput factors (see below) that can be measured from flat-field frames plus a scaling factor + applied if the counts of each frame are scaled to the mean counts of all frames, - :math:`D` is the dark-current, i.e., the rate at which the detector generates thermal electrons, in e-/pixel/s, - :math:`N_{\rm bin}` is the number of pixels in a binned pixel, diff --git a/doc/dev/hiresconfig.rst b/doc/dev/hiresconfig.rst new file mode 100644 index 0000000000..52ad731b4b --- /dev/null +++ b/doc/dev/hiresconfig.rst @@ -0,0 +1,118 @@ +.. include:: ../include/links.rst + +.. _hires_config: + +Automated sorting of HIRES frames by instrument configuration +============================================================= + +Version History +--------------- + + +========= ================ =========== =========== +*Version* *Author* *Date* ``PypeIt`` +========= ================ =========== =========== +1.0 Debora Pelliccia 10 Aug 2024 1.16.1.dev +========= ================ =========== =========== + +---- + +Basics +------ + +To prepare for the data reduction, PypeIt, first, automatically associates fits +files to specific :ref:`frame_types` (see :ref:`hires_frames`) and, then, +collects groups of frames in unique instrument configurations (see below). This is performed +by the :ref:`pypeit_setup` script, which sorts the frames and writes a +:ref:`pypeit_file` for each unique configuration. See :ref:`setup_doc`. + + +HIRES configuration identification +--------------------------------- + +The HIRES instrument configurations are determined by the function +:func:`pypeit.metadata.PypeItMetaData.unique_configurations`, +which finds unique combinations of the following keywords: + +=============== ============ +``fitstbl`` key Header Key +=============== ============ +``dispname`` ``XDISPERS`` +``decker`` ``DECKNAME`` +``binning`` ``BINNING`` +``filter1`` ``FIL1NAME`` +``echangle`` ``ECHANGL`` +``xdangle`` ``XDANGL`` +=============== ============ + +The unique configurations are determined by collating the relevant metadata from the headers +of all frames found by a run of :ref:`pypeit_setup`, *except* those that are designated as +bias and slitless_pixflat frames. Bias and slitless_pixflat frames can have header data (e.g., ``filter1``) +that do not match the instrument configuration that an observer intended for their use. +Therefore, PypeIt uses the ``dispname`` and ``binning`` keys to match the bias and +slitless_pixflat frames to the configurations with frames taken with the same cross-disperser +and same binning. +Note that when using the ``echangle`` and ``xdangle`` keys to identify configurations, PypeIt +uses a relative tolerance of 1e-3 and absolute tolerance of 1e-2 for ``echangle``, and a relative +tolerance of 1e-2 for ``xdangle``, to account for small differences in the values of these angles. + +After that, :func:`pypeit.metadata.PypeItMetaData.set_configurations` associates each frame +to the relevant unique configuration ("setup"), by assigning a setup identifier +(e.g., A,B,C,D...) to every frames for which the values of the above keywords match the +values of the specific unique configuration. + +HIRES calibration groups +----------------------- + +PypeIt uses the concept of a "calibration group" to define a complete set of +calibration frames (e.g., arcs, flats) and the science frames to which these calibration +frames should be applied. + +By default, :ref:`pypeit_setup` uses the setup identifier to assign frames to a single +calibration group. Frames that are in the same calibration group will have the same PypeIt +keyword ``calib``. No automated procedure exists to do anything except this. +However, the user can edit the :ref:`pypeit_file` to, within a given configuration, assign +specific calibration frames to specific science frames using the data in the ``calib`` column +of the :ref:`data_block`. + +Testing +------- + +To test that PypeIt can successfully identify multiple +configurations among a set of files, we have added the +``test_setup_keck_hires_multiconfig()`` test to +``${PYPEIT_DEV}/unit_tests/test_setups.py``. + +Here is an example of how to run the test: + +.. code-block:: bash + + cd ${PYPEIT_DEV}/unit_tests + pytest test_setup.py::test_setup_keck_hires_multiconfig -W ignore + +The tests require that you have downloaded the PypeIt +:ref:`dev-suite` and defined the ``PYPEIT_DEV`` environmental +variable that points to the relevant directory. + +The algorithm for this test is as follows: + + 1. Collect the names of all files in selected HIRES directories. + + 2. Use :class:`~pypeit.pypeitsetup.PypeItSetup` to automatically + identify the configurations for these files. + + 3. Check that the code found two configurations and wrote the + pypeit files for each. + + 4. For each configuration: + + a. Read the pypeit file + + b. Check that the name for the setup is correct ('A' or 'B') + + c. Check that the calibration group is the same for all frames ('0' or '1') + + +Because these tests are now included in the PypeIt +:ref:`unit-tests`, these configuration checks are performed by the +developers for every new version of the code. diff --git a/doc/dev/hiresframes.rst b/doc/dev/hiresframes.rst new file mode 100644 index 0000000000..0d0dad6f41 --- /dev/null +++ b/doc/dev/hiresframes.rst @@ -0,0 +1,128 @@ +.. include:: ../include/links.rst + +.. _hires_frames: + +Automated typing of HIRES frames +================================ + +Version History +--------------- + + +========= ================ =========== =========== +*Version* *Author* *Date* ``PypeIt`` +========= ================ =========== =========== +1.0 Debora Pelliccia 10 Aug 2024 1.16.1.dev +========= ================ =========== =========== + +---- + +Basics +------ + +The general procedure used to assign frames a given type is described +here: :ref:`frame_types`. + +HIRES frame typing +----------------- + +The primary typing of HIRES frames is performed by +:func:`pypeit.spectrographs.keck_hires.KECKHIRESSpectrograph.check_frame_type`. +This function checks the values of various header keywords against a +set of criteria used to classify the frame type. +The header cards required for the frame-typing and their associated keyword in the +:class:`~pypeit.metadata.PypeItMetaData` object are: + +=============== ============ +``fitstbl`` key Header Key +=============== ============ +``exptime`` ``ELAPTIME`` +``hatch`` ``HATOPEN`` +``lampstat01`` See below +No key ``XCOVOPEN`` +No key ``AUTOSHUT`` +=============== ============ + +``lampstat01`` is defined using a combination of header keywords, which include +``LAMPCAT1``, ``LAMPCAT2``, ``LAMPQTZ2``, ``LAMPNAME``. If ``LAMPCAT1 = True`` or +``LAMPCAT2 = True``, ``lampstat01`` will be equal to ``'ThAr1'`` or ``'ThAr2'``, respectively. +If ``LAMPQTZ2 = True`` or ``LAMPNAME = 'quartz1'``, ``lampstat01`` will be equal to ``'on'``. + + +The criteria used to select each frame type are as follows: + +==================== ============ ============ ============ ====================================== ====================================================== +Frame ``hatch`` ``AUTOSHUT`` ``XCOVOPEN`` ``lampstat01`` ``exptime`` +==================== ============ ============ ============ ====================================== ====================================================== +``science`` ``True`` ``True`` ``True`` ``'off'`` ``>601s`` +``standard`` ``'open'`` ``True`` ``True`` ``'off'`` ``>1s`` & ``<600s`` +``bias`` ``False`` ``False`` ``True`` ``'off'`` ``<0.001s`` +``dark`` ``False`` ``True`` ``True`` ``'off'`` Not used +``slitless_pixflat`` ``False`` ``True`` ``False`` ``'off'`` ``<60s`` +``pixelflat`` ``False`` ``True`` ``True`` ``'on'`` ``<60s`` +``trace`` ``False`` ``True`` ``True`` ``'on'`` ``<60s`` +``illumflat`` ``False`` ``True`` ``True`` ``'on'`` ``<60s`` +``arc`` ``False`` ``True`` ``True`` ``'ThAr1'`` or ``'ThAr2'`` Not used +``tilt`` ``False`` ``True`` ``True`` ``'ThAr1'`` or ``'ThAr2'`` Not used +==================== ============ ============ ============ ====================================== ====================================================== + +Note that PypeIt employs commonly used value of ``exptime`` to distinguish frame type; +however, if needed, the user can specify a different value by +using the ``exprng`` parameter in the :ref:`pypeit_file`; see also :ref:`frame_types`. + +The ``science`` and ``standard`` frames have identical selection criteria, except for the +``exptime`` value. In order to better distinguish between the two types, the ``RA`` and ``DEC`` header +keywords are also used to assign the ``standard`` type to frames with ``RA`` and ``DEC`` values that are +within 10 arcmin of one of the standard stars available in PypeIt (see :ref:`standards`). + +The criteria used to select ``arc`` and ``tilt`` frames are identical; the same is true for +``pixelflat``, ``trace``, and ``illumflat`` frames. Note that if both ``pixelflat`` and +``slitless_pixflat`` frames are identified, the ``pixelflat`` assignment will be removed +so that the ``slitless_pixflat`` frames will be used for the flat fielding. + +Finally, note that a HIRES frame is never given a ``pinhole`` type. + + +Testing +------- + +To test that PypeIt can successfully identify HIRES framt types +among a set of files, we have added the +``test_hires()`` test to ``${PYPEIT_DEV}/unit_tests/test_frametype.py``. + +Here is an example of how to run the test: + +.. code-block:: bash + + cd ${PYPEIT_DEV}/unit_tests + pytest test_frametype.py::test_hires -W ignore + +The tests requires that you have downloaded the PypeIt +:ref:`dev-suite` and defined the ``PYPEIT_DEV`` environmental +variable that points to the relevant directory. The algorithm for +all these tests is the same and is as follows: + + 1. Find the directories in the :ref:`dev-suite` with Keck + HIRES data. + + 2. For each directory (i.e., instrument setup): + + a. Make sure there is a "by-hand" version of the pypeit file + for this setup where a human (one of the pypeit + developers) has ensured the frame types are correct. + + b. Effectively run :ref:`pypeit_setup` on each of the + instrument setups to construct a new pypeit file with the + automatically generated frame types. + + c. Read both the by-hand and automatically generated frame + types from these two pypeit files and check that they are + identical. This check is *only* performed for the + calibration frames, not any ``science`` or ``standard`` + frames. + +Because this test is now included in the ``PypeIt`` +:ref:`unit-tests`, this frame-typing check is performed by the +developers for every new version of the code. + + diff --git a/doc/dev/lrisconfig.rst b/doc/dev/lrisconfig.rst index 3e73cedcb5..aa0638ffe9 100644 --- a/doc/dev/lrisconfig.rst +++ b/doc/dev/lrisconfig.rst @@ -13,6 +13,7 @@ Version History *Version* *Author* *Date* ``PypeIt`` ========= ================ =========== =========== 1.0 Debora Pelliccia 6 Sep 2023 1.13.1.dev +1.1 Debora Pelliccia 10 Aug 2024 1.16.1.dev ========= ================ =========== =========== ---- @@ -50,12 +51,15 @@ which finds unique combinations of the following keywords: The unique configurations are determined by collating the relevant metadata from the headers of all frames found by a run of :ref:`pypeit_setup`, *except* those that are designated as -bias frames. The reason is that bias frames can have header data (e.g., ``dispangle``) +bias and slitless_pixflat frames. Bias frames can have header data (e.g., ``dispangle``) that do not match the instrument configuration that an observer intended for their use; e.g., the frames were taken before the instrument was fully configured for the night's observations. Therefore, PypeIt uses the ``dateobs``, ``binning``, ``amp`` keys to match the bias frames to the configurations with frames taken on the same date, with -the same binning and on the same amplifier. +the same binning and on the same amplifier. Similarly, slitless_pixflat frames are taken +without a slitmask (i.e., ``decker`` is different from the other frames), therefore +PypeIt uses the ``dateobs``, ``binning``, ``amp``, ``dispname``, ``dichroic`` keys to match +the slitless_pixflat frames to a specific configuration. After that, :func:`pypeit.metadata.PypeItMetaData.set_configurations` associates each frame to the relevant unique configuration ("setup"), by assigning a setup identifier diff --git a/doc/dev/lrisframes.rst b/doc/dev/lrisframes.rst index ef7c845273..ae52ee3c1c 100644 --- a/doc/dev/lrisframes.rst +++ b/doc/dev/lrisframes.rst @@ -13,6 +13,7 @@ Version History *Version* *Author* *Date* ``PypeIt`` ========= ================ =========== =========== 1.0 Debora Pelliccia 6 Sep 2023 1.13.1.dev +1.1 Debora Pelliccia 10 Aug 2024 1.16.1.dev ========= ================ =========== =========== ---- @@ -40,6 +41,7 @@ The header cards required for the frame-typing and their associated keyword in t =============== ====================================================== ``exptime`` ``ELAPTIME`` (``TELAPSE`` for ``keck_lris_red_mark4``) ``hatch`` ``TRAPDOOR`` +``decker`` ``SLITNAME`` ``lampstat01`` See below =============== ====================================================== @@ -51,32 +53,43 @@ of keywords used to define ``lampstat01`` varies depending on the available head The criteria used to select each frame type are as follows: -============= ============ ====================================== =========================================== -Frame ``hatch`` ``lampstat01`` ``exptime`` -============= ============ ====================================== =========================================== -``science`` ``'open'`` ``'off'`` ``>61s`` -``standard`` ``'open'`` ``'off'`` ``>1s`` & ``<61s`` -``bias`` ``'closed'`` ``'off'`` ``<0.001s`` -``pixelflat`` ``'closed'`` ``'Halogen' or '2H'`` ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE) -``pixelflat`` ``'open'`` ``'on'`` ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE) -``trace`` ``'closed'`` ``'Halogen'`` or ``'2H'`` ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE) -``trace`` ``'open'`` ``'on'`` ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE) -``illumflat`` ``'closed'`` ``'Halogen'`` or ``'2H'`` ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE) -``illumflat`` ``'open'`` ``'on'`` ``<60s``(LRIS RED) or ``<300s`` (LRIS BLUE) -``arc`` ``'closed'`` ``!= 'Halogen', '2H', 'on', 'off'`` Not used -``tilt`` ``'closed'`` ``!= 'Halogen', '2H', 'on', 'off'`` Not used -============= ============ ====================================== =========================================== +==================== ============ ====================================== ================= ====================================================== +Frame ``hatch`` ``lampstat01`` ``decker`` ``exptime`` +==================== ============ ====================================== ================= ====================================================== +``science`` ``'open'`` ``'off'`` ``!= 'GOH_LRIS'`` ``>61s`` +``standard`` ``'open'`` ``'off'`` ``!= 'GOH_LRIS'`` ``>1s`` & ``<61s`` (LRIS RED) or ``<900s`` (LRIS BLUE) +``bias`` ``'closed'`` ``'off'`` ``!= 'GOH_LRIS'`` ``<1s`` +``slitless_pixflat`` ``'open'`` ``'off'`` ``== 'direct'`` ``<60s`` +``pixelflat`` ``'closed'`` ``'Halogen' or '2H'`` ``!= 'GOH_LRIS'`` ``<60s`` (LRIS RED) or ``<300s`` (LRIS BLUE) +``pixelflat`` ``'open'`` ``'on'`` ``!= 'GOH_LRIS'`` ``<60s`` (LRIS RED) or ``<300s`` (LRIS BLUE) +``trace`` ``'closed'`` ``'Halogen'`` or ``'2H'`` ``!= 'GOH_LRIS'`` ``<60s`` (LRIS RED) or ``<300s`` (LRIS BLUE) +``trace`` ``'open'`` ``'on'`` ``!= 'GOH_LRIS'`` ``<60s`` (LRIS RED) or ``<300s`` (LRIS BLUE) +``illumflat`` ``'closed'`` ``'Halogen'`` or ``'2H'`` ``!= 'GOH_LRIS'`` ``<60s`` (LRIS RED) or ``<300s`` (LRIS BLUE) +``illumflat`` ``'open'`` ``'on'`` ``!= 'GOH_LRIS'`` ``<60s`` (LRIS RED) or ``<300s`` (LRIS BLUE) +``arc`` ``'closed'`` ``!= 'Halogen', '2H', 'on', 'off'`` ``!= 'GOH_LRIS'`` Not used +``tilt`` ``'closed'`` ``!= 'Halogen', '2H', 'on', 'off'`` ``!= 'GOH_LRIS'`` Not used +==================== ============ ====================================== ================= ====================================================== Note that PypeIt employs commonly used value of ``exptime`` to distinguish frame type; however, if needed, the user can specify a different value by using the ``exprng`` parameter in the :ref:`pypeit_file`; see also :ref:`frame_types`. +The ``science`` and ``standard`` frames have identical selection criteria, except for the +``exptime`` value. In order to better distinguish between the two types, the ``RA`` and ``DEC`` header +keywords are also used to assign the ``standard`` type to frames with ``RA`` and ``DEC`` values that are +within 10 arcmin of one of the standard stars available in PypeIt (see :ref:`standards`). + The criteria used to select ``arc`` and ``tilt`` frames are identical; the same is true for -``pixelflat``, ``trace``, and ``illumflat`` frames. However, it's important to note that +``pixelflat``, ``trace``, and ``illumflat`` frames. It's important to note that PypeIt is able to correctly assign the ``pixelflat``, ``trace``, and ``illumflat`` types -to the internal and dome flat frames, but the twilight flats will generally have the -``science`` or ``standard`` type. Therefore, the user should manually change their frame type -in the :ref:`pypeit_file`. +to the internal and dome flat frames, and it tries to do the same for the twilight flats, by selecting +frames that looks like ``science`` frames and include the following words in the ``OBJECT`` +or ``TARGNAME`` header keywords: 'sky', 'blank', 'twilight', 'twiflat', 'twi flat'. This way of +identifying twilight flats is not robust, therefore the user should always check the frame types assigned +and manually change them if needed in the :ref:`pypeit_file`. + +Note, also, that if both ``pixelflat`` and ``slitless_pixflat`` frames are identified, the ``pixelflat`` +assignment will be removed so that the ``slitless_pixflat`` frames will be used for the flat fielding. Finally, note that a LRIS frame is never given a ``pinhole`` or ``dark`` type. diff --git a/doc/frametype.rst b/doc/frametype.rst index 83387add34..148f9cf078 100644 --- a/doc/frametype.rst +++ b/doc/frametype.rst @@ -42,24 +42,25 @@ description can be listed as follows: More detailed descriptions are given in the table below. -================ ============================================================= -Frame Type Description -================ ============================================================= -``align`` Used to align spatial positions in multiple slits. This frame is particularly useful for slit-based IFU, such as Keck KCWI. -``arc`` Spectrum of one or more calibration arc lamps -``bias`` Bias frame; typically a 0s exposure with the shutter closed -``dark`` Dark frame; typically a >0s exposure to assess dark current (shutter closed) -``illumflat`` Spectrum taken to correct illumination profile of the slit(s). This is often the same as the trace flat (below). -``lampoffflats`` Spectrum taken to remove persistence from lamp on flat exposures and/or thermal emission from the telescope and dome. Usually this is an exposure using a flat with lamps OFF -``pinhole`` Spectrum taken through a pinhole slit (i.e. a very short slit length), and is used to define the centre if a slit (currently, this frame is only used for echelle data reduction). Often this is an exposure using a flat lamp, but one can in principle use a standard star frame too (or a science frame if the spectrum is uniform). -``pixelflat`` Spectrum taken to correct for pixel-to-pixel detector variations Often an exposure using a dome (recommended) or internal flat lamp, but for observations in the very blue, this may be on-sky -``science`` Spectrum of one or more science targets -``standard`` Spectrum of spectrophotometric standard star PypeIt includes a list of pre-defined standards -``trace`` Spectrum taken to define the slit edges. Often this is an exposure using a flat lamp, but for observations in the very blue, this may be on-sky. The slit length of a trace frame should be the same as the science slit. -``tilt`` Exposure used to trace the tilt in the wavelength solution. Often the same file(s) as the arc. -``sky`` On-sky observation of the sky used for background subtraction -``None`` File could not be automatically identified by PypeIt -================ ============================================================= +==================== ============================================================= +Frame Type Description +==================== ============================================================= +``align`` Used to align spatial positions in multiple slits. This frame is particularly useful for slit-based IFU, such as Keck KCWI. +``arc`` Spectrum of one or more calibration arc lamps +``bias`` Bias frame; typically a 0s exposure with the shutter closed +``dark`` Dark frame; typically a >0s exposure to assess dark current (shutter closed) +``illumflat`` Spectrum taken to correct illumination profile of the slit(s). This is often the same as the trace flat (below). +``lampoffflats`` Spectrum taken to remove persistence from lamp on flat exposures and/or thermal emission from the telescope and dome. Usually this is an exposure using a flat with lamps OFF +``pinhole`` Spectrum taken through a pinhole slit (i.e. a very short slit length), and is used to define the centre if a slit (currently, this frame is only used for echelle data reduction). Often this is an exposure using a flat lamp, but one can in principle use a standard star frame too (or a science frame if the spectrum is uniform). +``pixelflat`` Spectrum taken to correct for pixel-to-pixel detector variations. Often an exposure using a dome (recommended) or internal flat lamp, but for observations in the very blue, this may be on-sky +``slitless_pixflat`` Spectrum taken without a slitmask or longslit to correct for pixel-to-pixel detector variations. This is often an exposure taken on-sky +``science`` Spectrum of one or more science targets +``standard`` Spectrum of spectrophotometric standard star PypeIt includes a list of pre-defined standards +``trace`` Spectrum taken to define the slit edges. Often this is an exposure using a flat lamp, but for observations in the very blue, this may be on-sky. The slit length of a trace frame should be the same as the science slit. +``tilt`` Exposure used to trace the tilt in the wavelength solution. Often the same file(s) as the arc. +``sky`` On-sky observation of the sky used for background subtraction +``None`` File could not be automatically identified by PypeIt +==================== ============================================================= .. TODO: Need to check that "sky" frametype is correct and/or used! diff --git a/doc/releases/1.16.1dev.rst b/doc/releases/1.16.1dev.rst index 5399153230..372081dbeb 100644 --- a/doc/releases/1.16.1dev.rst +++ b/doc/releases/1.16.1dev.rst @@ -23,10 +23,19 @@ Functionality/Performance Improvements and Additions This is done by setting the parameter ``use_std_trace`` in FindObjPar. - Now PypeIt can handle the case where "Standard star trace does not match the number of orders in the echelle data" both in `run_pypeit` and in `pypeit_coadd_1dspec`. +- Added the functionality to use slitless flats to create pixelflats. Note: new frametype + `slitless_pixflat` is added to the PypeIt frametype list. +- The created pixelflats are stored in the reduction directory and in the PypeIt cache directory + `data/pixelflats`. +- Added a functionality that allows, when multiple frames are combined, to scale each + frame to have the same mean value before combining. To use this + functionality, the new parameter ``scale_mean`` should be set to ``True``. Instrument-specific Updates --------------------------- +- Improved LRIS frame typing, including the typing of slitless flats and sky flats. +- Improved HIRES frame typing and configuration setup. - Added support for the (decommissioned) AAT/UHRF instrument - Updated the requirements of a spectral flip for KCWI (blue) data. If all amplifiers are used, the data will be flipped. If all amplifiers are used, the data will not be flipped. @@ -64,6 +73,8 @@ Script Changes - Added the ``--extr`` parameter in the ``pypeit_sensfunc`` script (also as a SensFuncPar) to allow the user to specify the extraction method to use when computing the sensitivity function (before only optimal extraction was used). +- Added ``pypeit_show_pixflat`` script to inspect the (slitless) pixel flat + generated during the reduction and stored in ``data/pixelflats``. - Added ``pypeit_chk_flexure`` script to check both spatial and spectral flexure applied to the reduced data. - Treatment of file names is now more formal. Compression signatures are now diff --git a/doc/scripts.rst b/doc/scripts.rst index 0b37cc7b0c..e5c4aee3d6 100644 --- a/doc/scripts.rst +++ b/doc/scripts.rst @@ -566,6 +566,26 @@ The script usage can be displayed by calling the script with the .. include:: help/pypeit_chk_flats.rst +.. _pypeit_show_pixflat: + +pypeit_show_pixflat +------------------- + +Inspect in a Ginga window the (slitless) pixel flat produced by PypeIt and stored +in the PypeIt cache (see ref:`data_installation`). It displays each detector separately +in different channels. The script is useful for assessing the quality of the pixel-to-pixel +response of the detector. Typical call is: + +.. code-block:: console + + pypeit_show_pixflat PYPEIT_LRISb_pixflat_B600_2x2_17sep2009_specflip.fits.gz + + +The script usage can be displayed by calling the script with the +``-h`` option: + +.. include:: help/pypeit_show_pixflat.rst + pypeit_show_2dspec ------------------ diff --git a/doc/spectrographs/deimos.rst b/doc/spectrographs/deimos.rst index a6e4d8bc81..d9131773c8 100644 --- a/doc/spectrographs/deimos.rst +++ b/doc/spectrographs/deimos.rst @@ -15,9 +15,9 @@ settings that are related to the Keck/DEIMOS spectrograph. .. warning:: PypeIt currently *cannot* reduce images produced by reading - the DEIMOS CCDs with the A amplifier or those taken in imaging + the DEIMOS CCDs with the A+B amplifier or those taken in imaging mode. All image-handling assumes DEIMOS images have been read - with the B amplifier in the "Spectral" observing mode. PypeIt + with the A or B amplifier in the "Spectral" observing mode. PypeIt handles files that do not meet these criteria in two ways: - When running :ref:`pypeit_setup`, any frames not in diff --git a/doc/spectrographs/keck_hires.rst b/doc/spectrographs/keck_hires.rst index f8d3277554..34ff1dd6fd 100644 --- a/doc/spectrographs/keck_hires.rst +++ b/doc/spectrographs/keck_hires.rst @@ -6,13 +6,85 @@ Overview ======== This file summarizes several instrument specific settings that are related to the Keck/HIRES spectrograph. +Currently, PypeIt only supports the reduction of the post detector upgrade HIRES data (around August 2004). +Information on the frame typing and instrument configuration can be found in ref:`hiresframes` and :ref:`hiresconfig`. + +Default Settings +++++++++++++++++ + +See :ref:`instr_par-keck_hires` for a list of modifications to the default settings. +*You do not have to add these changes to your PypeIt reduction file!* This is just a list of +how the parameters used for HIRES differ from the defaults listed in the preceding tables on that page. +Moreover, additional modifications may have been made for specific setups, e.g, for different binning, +etc. You can see a list of all the used parameters in the ``keck_hires_XXX.par`` +file generated by PypeIt at the beginning of the reduction. + +MOSAIC +====== + +PypeIt, by default, uses a mosaic approach for the reduction. It basically constructs a mosaic +of the blue, green, and red detector data and reduces it, instead of processing the detector data individually. +The mosaic reduction is switched on by setting the parameter ``detnum`` in :ref:`reduxpar` to be a +tuple of the detector indices that are mosaiced together. For HIRES, it looks like: + +.. code-block:: ini + + [rdx] + spectrograph = keck_hires + detnum = (1,2,3) + +This is already the default for HIRES, but the user can modify it in the :ref:`pypeit_file` to +turn off the mosaic reduction. + + +Calibrations +============ + +Flat Fielding +------------- +Special type of "slitless" flat images are sometime used for correcting +pixel-to-pixel variations in the HIRES data. These flats images are taken +with the cross disperser cover on, so that the light is somewhat diffused +and the orders are not visible. If this type of frames are present in the +:ref:`pypeit_file`, during the main reduction PypeIt will be able to identify them, +assign the correct frame type (`slitless_pixflat`), and use them for creating a "slitless" +pixel flat. Since, these frames are taken with different exposure time and +cross-disperser angle for each detector, PypeIt is able to select the correct +observations for each detector. The pixelflat is generated by first scaling +each flat frame to have the same mean counts, then median combining them. +A pixelflat file will be generated and stored in the reduction folder. +In addition, the constructed pixelflat is saved to the PypeIt cache (see ref:`data_installation`). +This allows you to use the file for both current and future reductions. +To use this file in future reductions, the user can add the +slitless pixelflat file name to the :ref:`pypeit_file` as shown below: + +.. code-block:: ini + + [calibrations] + [[flatfield]] + pixelflat_file = pixelflat_keck_hires_RED_1x2_20160330.fits.gz + +See more in :ref:`generate-pixflat`. + Wavelengths -=========== +----------- See :ref:`wvcalib-echelle` for details on the wavelength calibration. -We also note that several Orders from 40-45 are -frequently flagged as bad in the wavelength solution. -This is due, in part, to very bright ThAr line contamination. \ No newline at end of file + +Additional Reading +================== + +Here are additional docs related to Keck/HIRES. Note all of them are related +to the development of PypeIt for use with HIRES data: + +.. TODO: Generally useful information in these dev docs should be moved into +.. user-level doc pages, even if that means repeating information. + +.. toctree:: + :maxdepth: 1 + + ../dev/hiresframes + ../dev/hiresconfig \ No newline at end of file diff --git a/doc/spectrographs/lris.rst b/doc/spectrographs/lris.rst index f6b4b786c9..e28b7cf3d5 100644 --- a/doc/spectrographs/lris.rst +++ b/doc/spectrographs/lris.rst @@ -212,21 +212,45 @@ Pixel Flat It is recommend to correct for pixel-to-pixel variations using a slitless flat. If you did not take such calibration frames or cannot process them, -you may wish to use an archival. -`This link `__ -has the existing ones staged by the PypeIt team. +you may wish to use an archival one available in the PypeIt cache (see ref:`data_installation`). And then set the following in your :ref:`pypeit_file`: .. code-block:: ini - [calibrations] - [[flatfield]] - frame = path_to_the_file/PYPEIT_LRISb_pixflat_B600_2x2_17sep2009.fits.gz - -.. warning:: - - Internal flats may be too bright and need to be tested. + [calibrations] + [[flatfield]] + pixelflat_file = PYPEIT_LRISb_pixflat_B600_2x2_17sep2009_specflip.fits.gz + + +If, instead, you have taken slitless pixelflat frames, PypeIt will be able to identify them +(see :ref:`lris_frames_report`) and process them during the reduction. The slitless pixelflat +is generated by first scaling each slitless flat frame to have the same mean counts, then +median combining them. A slitless pixelflat file will be generated +and stored in the reduction folder. In addition, the constructed pixelflat is saved to the +PypeIt cache. This allows you to use the file for both current and future reductions. +To use this file in future reductions, the user can add the +slitless pixelflat file name to the :ref:`pypeit_file` as shown above. Consider sharing +your slitless pixelflat file with the PypeIt Developers. See more in :ref:`generate-pixflat`. + +To generate slitless pixelflat, we recommend using twilight flats. As these are taken +on the twilight sky, it is important to dither between exposures to avoid bright stars +from landing in the same part of the detector. Related, it is best to take these as close +to sunset (or sunrise) as possible (about 10 minutes after/before sunset/sunrise) so that +one can use the shortest exposure time possible (3s for LRIS given its shutter). + +Internal flats may be too bright and need to be tested. +Our recommended recipe for taking twilight flats is: + + 1. Point away from the Galaxy (i.e. avoid stars) + 2. Select the desired setup for your spectroscopy + 3. Remove any selected slitmask/longslit + 4. Expose for 3s + 5. Check counts and make sure you have more than 40,000 + 6. Offset by ~1' and 1' in NE + 7. Repeat 4-6, increasing the exposure each time you dip below 30,000 counts or so + +About 5 exposures is enough but the more the better. Trace Flat ---------- diff --git a/pypeit/cache.py b/pypeit/cache.py index 52379cf6ad..b1d6555d1f 100644 --- a/pypeit/cache.py +++ b/pypeit/cache.py @@ -162,6 +162,7 @@ def fetch_remote_file( install_script: bool=False, force_update: bool=False, full_url: str=None, + return_none: bool=False, ) -> pathlib.Path: """ Use `astropy.utils.data`_ to fetch file from remote or cache @@ -191,6 +192,8 @@ def fetch_remote_file( full_url (:obj:`str`, optional): The full url. If None, use :func:`_build_remote_url`). Defaults to None. + return_none (:obj:`bool`, optional): + Return None if the file is not found. Defaults to False. Returns: `Path`_: The local path to the desired file in the cache @@ -248,6 +251,9 @@ def fetch_remote_file( "https://pypeit.readthedocs.io/en/latest/fluxing.html#extinction-correction" ) + elif return_none: + return None + else: err_msg = ( f"Error downloading {filename}: {error}{msgs.newline()}" diff --git a/pypeit/calibrations.py b/pypeit/calibrations.py index 13f60553c6..63c2d313b8 100644 --- a/pypeit/calibrations.py +++ b/pypeit/calibrations.py @@ -39,10 +39,13 @@ from pypeit.core import framematch from pypeit.core import parse from pypeit.core import scattlight as core_scattlight +from pypeit.core.mosaic import build_image_mosaic from pypeit.par import pypeitpar from pypeit.spectrographs.spectrograph import Spectrograph from pypeit import io from pypeit import utils +from pypeit import cache +from pypeit import dataPaths class Calibrations: @@ -732,21 +735,50 @@ def get_flats(self): # Check internals self._chk_set(['det', 'calib_ID', 'par']) - pixel_frame = {'type': 'pixelflat', 'class': flatfield.FlatImages} - raw_pixel_files, pixel_cal_file, pixel_calib_key, pixel_setup, pixel_calib_id, detname \ - = self.find_calibrations(pixel_frame['type'], pixel_frame['class']) - + # generate the slitless pixel flat (if frames available). + slitless_rows = self.fitstbl.find_frames('slitless_pixflat', calib_ID=self.calib_ID, index=True) + if len(slitless_rows) > 0: + sflat = flatfield.SlitlessFlat(self.fitstbl, slitless_rows, self.spectrograph, + self.par, qa_path=self.qa_path) + # A pixel flat will be saved to disc and self.par['flatfield']['pixelflat_file'] will be updated + self.par['flatfield']['pixelflat_file'] = \ + sflat.make_slitless_pixflat(msbias=self.msbias, msdark=self.msdark, calib_dir=self.calib_dir, + write_qa=self.write_qa, show=self.show) + + # get illumination flat frames illum_frame = {'type': 'illumflat', 'class': flatfield.FlatImages} raw_illum_files, illum_cal_file, illum_calib_key, illum_setup, illum_calib_id, detname \ = self.find_calibrations(illum_frame['type'], illum_frame['class']) + # get pixel flat frames + pixel_frame = {'type': 'pixelflat', 'class': flatfield.FlatImages} + raw_pixel_files, pixel_cal_file, pixel_calib_key, pixel_setup, pixel_calib_id, detname \ + = [], None, None, illum_setup, None, detname + # read in the raw pixelflat frames only if the user has not provided a pixelflat_file + if self.par['flatfield']['pixelflat_file'] is None: + raw_pixel_files, pixel_cal_file, pixel_calib_key, pixel_setup, pixel_calib_id, detname \ + = self.find_calibrations(pixel_frame['type'], pixel_frame['class']) + + # get lamp off flat frames raw_lampoff_files = self.fitstbl.find_frame_files('lampoffflats', calib_ID=self.calib_ID) + # Check if we have any calibration frames to work with if len(raw_pixel_files) == 0 and pixel_cal_file is None \ and len(raw_illum_files) == 0 and illum_cal_file is None: - msgs.warn(f'No raw {pixel_frame["type"]} or {illum_frame["type"]} frames found and ' - 'unable to identify a relevant processed calibration frame. Continuing...') - self.flatimages = None + # if no calibration frames are found, check if the user has provided a pixel flat file + if self.par['flatfield']['pixelflat_file'] is not None: + msgs.warn(f'No raw {pixel_frame["type"]} or {illum_frame["type"]} frames found but a ' + 'user-defined pixel flat file was provided. Using that file.') + self.flatimages = flatfield.FlatImages(PYP_SPEC=self.spectrograph.name, spat_id=self.slits.spat_id) + self.flatimages.calib_key = flatfield.FlatImages.construct_calib_key(self.fitstbl['setup'][self.frame], + self.calib_ID, detname) + self.flatimages = flatfield.load_pixflat(self.par['flatfield']['pixelflat_file'], self.spectrograph, + self.det, self.flatimages, calib_dir=self.calib_dir, + chk_version=self.chk_version) + else: + msgs.warn(f'No raw {pixel_frame["type"]} or {illum_frame["type"]} frames found and ' + 'unable to identify a relevant processed calibration frame. Continuing...') + self.flatimages = None return self.flatimages # If a processed calibration frame exists and we want to reuse it, do @@ -768,10 +800,9 @@ def get_flats(self): # Load user defined files if self.par['flatfield']['pixelflat_file'] is not None: # Load - msgs.info(f'Using user-defined file: {self.par["flatfield"]["pixelflat_file"]}') - with io.fits_open(self.par['flatfield']['pixelflat_file']) as hdu: - nrm_image = flatfield.FlatImages(pixelflat_norm=hdu[self.det].data) - self.flatimages = flatfield.merge(self.flatimages, nrm_image) + self.flatimages = flatfield.load_pixflat(self.par['flatfield']['pixelflat_file'], self.spectrograph, + self.det, self.flatimages, calib_dir=self.calib_dir, + chk_version=self.chk_version) # update slits self.slits.mask_flats(self.flatimages) return self.flatimages @@ -817,8 +848,8 @@ def get_flats(self): # Initialise the pixel flat pixelFlatField = flatfield.FlatField(pixel_flat, self.spectrograph, - self.par['flatfield'], self.slits, self.wavetilts, - self.wv_calib, qa_path=self.qa_path, + self.par['flatfield'], self.slits, wavetilts=self.wavetilts, + wv_calib=self.wv_calib, qa_path=self.qa_path, calib_key=calib_key) # Generate pixelflatImages = pixelFlatField.run(doqa=self.write_qa, show=self.show) @@ -863,8 +894,8 @@ def get_flats(self): # Initialise the illum flat illumFlatField = flatfield.FlatField(illum_flat, self.spectrograph, - self.par['flatfield'], self.slits, self.wavetilts, - self.wv_calib, spat_illum_only=True, + self.par['flatfield'], self.slits, wavetilts=self.wavetilts, + wv_calib=self.wv_calib, spat_illum_only=True, qa_path=self.qa_path, calib_key=calib_key) # Generate illumflatImages = illumFlatField.run(doqa=self.write_qa, show=self.show) @@ -899,10 +930,9 @@ def get_flats(self): # Should we allow that? if self.par['flatfield']['pixelflat_file'] is not None: # Load - msgs.info(f'Using user-defined file: {self.par["flatfield"]["pixelflat_file"]}') - with io.fits_open(self.par['flatfield']['pixelflat_file']) as hdu: - self.flatimages = flatfield.merge(self.flatimages, - flatfield.FlatImages(pixelflat_norm=hdu[self.det].data)) + self.flatimages = flatfield.load_pixflat(self.par['flatfield']['pixelflat_file'], self.spectrograph, + self.det, self.flatimages, calib_dir=self.calib_dir, + chk_version=self.chk_version) return self.flatimages @@ -1278,6 +1308,7 @@ def get_association(fitstbl, spectrograph, caldir, setup, calib_ID, det, must_ex 'pixelflat': [flatfield.FlatImages], 'illumflat': [flatfield.FlatImages], 'lampoffflats': [flatfield.FlatImages], + 'slitless_pixflat': [flatfield.FlatImages], 'trace': [edgetrace.EdgeTraceSet, slittrace.SlitTraceSet], 'tilt': [buildimage.TiltImage, wavetilts.WaveTilts] } @@ -1313,7 +1344,8 @@ def get_association(fitstbl, spectrograph, caldir, setup, calib_ID, det, must_ex indx = fitstbl.find_frames(frametype) & in_grp if not any(indx): continue - if not all(fitstbl['calib'][indx] == fitstbl['calib'][indx][0]): + if not (all(fitstbl['calib'][indx] == fitstbl['calib'][indx][0]) or + all([fitstbl['calib'][indx][0] in cc.split(',') for cc in fitstbl['calib'][indx]])): msgs.error(f'CODING ERROR: All {frametype} frames in group {calib_ID} ' 'are not all associated with the same subset of calibration ' 'groups; calib for the first file is ' @@ -1539,8 +1571,13 @@ def check_for_calibs(par, fitstbl, raise_error=True, cut_cfg=None): if ftype == 'pixelflat' \ and par['calibrations']['flatfield']['pixelflat_file'] is not None: continue + # Allow for no pixelflat but slitless_pixflat needs to exist + elif ftype == 'pixelflat' \ + and len(fitstbl.find_frame_files('slitless_pixflat', calib_ID=calib_ID)) > 0: + continue # Otherwise fail - msg = f'No frames of type={ftype} provide for the *{key}* processing ' \ + add_msg = ' or slitless_pixflat' if ftype == 'pixelflat' else '' + msg = f'No frames of type={ftype}{add_msg} provided for the *{key}* processing ' \ 'step. Add them to your PypeIt file!' pass_calib = False if raise_error: diff --git a/pypeit/core/extract.py b/pypeit/core/extract.py index a820b27be5..5e48c01567 100644 --- a/pypeit/core/extract.py +++ b/pypeit/core/extract.py @@ -365,7 +365,8 @@ def extract_boxcar(imgminsky, ivar, mask, waveimg, skyimg, spec, fwhmimg=None, b A scale factor, :math:`s`, that *has already been applied* to the provided science image. It accounts for the number of frames contributing to the provided counts, and the relative throughput factors that can be measured - from flat-field frames. For example, if the image has been flat-field + from flat-field frames plus a scaling factor applied if the counts of each frame are + scaled to the mean counts of all frames. For example, if the image has been flat-field corrected, this is the inverse of the flat-field counts. If None, set to 1. If a single float, assumed to be constant across the full image. If an array, the shape must match ``base_var``. The variance will be 0 diff --git a/pypeit/core/framematch.py b/pypeit/core/framematch.py index 4125f7011e..94200c405f 100644 --- a/pypeit/core/framematch.py +++ b/pypeit/core/framematch.py @@ -32,6 +32,7 @@ def __init__(self): ('lampoffflats', 'Flat-field exposure with lamps off used to remove ' 'persistence from lamp on flat exposures and/or thermal emission ' 'from the telescope and dome'), + ('slitless_pixflat', 'Flat-field exposure without slitmask used for pixel-to-pixel response'), ('scattlight', 'Frame (ideally with lots of counts) used to determine the scattered light model'), ('science', 'On-sky observation of a primary target'), ('standard', 'On-sky observation of a flux calibrator'), diff --git a/pypeit/core/procimg.py b/pypeit/core/procimg.py index c68be8d379..795a198446 100644 --- a/pypeit/core/procimg.py +++ b/pypeit/core/procimg.py @@ -1157,7 +1157,9 @@ def base_variance(rn_var, darkcurr=None, exptime=None, proc_var=None, count_scal - :math:`C` is the observed number of sky + object counts, - :math:`s=s\prime / N_{\rm frames}` is a scale factor derived from the (inverse of the) flat-field frames plus the number - of frames contributing to the object counts (see ``count_scale``), + of frames contributing to the object counts plus a scaling + factor applied if the counts of each frame are scaled to the + mean counts of all frames (see ``count_scale``), - :math:`D` is the dark current in electrons per **hour** (see ``darkcurr``), - :math:`t_{\rm exp}` is the effective exposure time in seconds (see @@ -1229,8 +1231,9 @@ def base_variance(rn_var, darkcurr=None, exptime=None, proc_var=None, count_scal A scale factor that *has already been applied* to the provided counts. It accounts for the number of frames contributing to the provided counts, and the relative throughput factors that - can be measured from flat-field frames. For example, if the image - has been flat-field corrected, this is the inverse of the flat-field counts. + can be measured from flat-field frames plus a scaling factor applied + if the counts of each frame are scaled to the mean counts of all frames. + For example, if the image has been flat-field corrected, this is the inverse of the flat-field counts. If None, set to 1. If a single float, assumed to be constant across the full image. If an array, the shape must match ``rn_var``. The variance will be 0 wherever :math:`s \leq 0`, modulo the provided ``noise_floor``. @@ -1287,7 +1290,9 @@ def variance_model(base, counts=None, count_scale=None, noise_floor=None): - :math:`C` is the observed number of sky + object counts, - :math:`s=s\prime / N_{\rm frames}` is a scale factor derived from the (inverse of the) flat-field frames plus the number - of frames contributing to the object counts (see ``count_scale``), + of frames contributing to the object counts plus a scaling factor + applied if the counts of each frame are scaled to the mean counts + of all frames (see ``count_scale``), - :math:`D` is the dark current in electrons per **hour**, - :math:`t_{\rm exp}` is the effective exposure time in seconds, - :math:`V_{\rm rn}` is the detector readnoise variance (i.e., @@ -1347,7 +1352,9 @@ def variance_model(base, counts=None, count_scale=None, noise_floor=None): A scale factor that *has already been applied* to the provided counts; see :math:`s` in the equations above. It accounts for the number of frames contributing to the provided counts, and - the relative throughput factors that can be measured from flat-field frames. + the relative throughput factors that can be measured from flat-field frames + plus a scaling factor applied if the counts of each frame are + scaled to the mean counts of all frames. For example, if the image has been flat-field corrected, this is the inverse of the flat-field counts. If None, no scaling is expected, meaning ``counts`` are exactly the observed detector counts. If a single diff --git a/pypeit/core/skysub.py b/pypeit/core/skysub.py index d84c4d5c22..b4d5a7abf7 100644 --- a/pypeit/core/skysub.py +++ b/pypeit/core/skysub.py @@ -702,7 +702,8 @@ def local_skysub_extract(sciimg, sciivar, tilts, waveimg, global_sky, thismask, A scale factor, :math:`s`, that *has already been applied* to the provided science image. It accounts for the number of frames contributing to the provided counts, and the relative throughput factors that can be measured - from flat-field frames. For example, if the image has been flat-field + from flat-field frames plus a scaling factor applied if the counts of each frame are + scaled to the mean counts of all frames. For example, if the image has been flat-field corrected, this is the inverse of the flat-field counts. If None, set to 1. If a single float, assumed to be constant across the full image. If an array, the shape must match ``base_var``. The variance will be 0 @@ -972,10 +973,12 @@ def local_skysub_extract(sciimg, sciivar, tilts, waveimg, global_sky, thismask, else: msgs.warn('ERROR: Bspline sky subtraction failed after 4 iterations of bkpt spacing') msgs.warn(' Moving on......') - obj_profiles = np.zeros_like(obj_profiles) + # obj_profiles = np.zeros_like(obj_profiles) isub, = np.where(localmask.flatten()) # Just replace with the global sky skyimage.flat[isub] = global_sky.flat[isub] + if iiter == niter: + msgs.warn('WARNING: LOCAL SKY SUBTRACTION NOT PERFORMED') outmask_extract = outmask if use_2dmodel_mask else inmask @@ -1226,7 +1229,8 @@ def ech_local_skysub_extract(sciimg, sciivar, fullmask, tilts, waveimg, A scale factor that *has already been applied* to the provided science image. It accounts for the number of frames contributing to the provided counts, and the relative throughput factors that can be measured - from flat-field frames. For example, if the image has been flat-field corrected, + from flat-field frames plus a scaling factor applied if the counts of each frame are + scaled to the mean counts of all frames. For example, if the image has been flat-field corrected, this is the inverse of the flat-field counts. If None, set to 1. If a single float, assumed to be constant across the full image. If an array, the shape must match ``base_var``. The variance will be 0 wherever this diff --git a/pypeit/data/pixelflats/PYPEIT_LRISb_pixflat_B400_2x2_15apr2015_specflip.fits.gz b/pypeit/data/pixelflats/PYPEIT_LRISb_pixflat_B400_2x2_15apr2015_specflip.fits.gz new file mode 100644 index 0000000000..5fd3c06e3d Binary files /dev/null and b/pypeit/data/pixelflats/PYPEIT_LRISb_pixflat_B400_2x2_15apr2015_specflip.fits.gz differ diff --git a/pypeit/data/pixelflats/PYPEIT_LRISb_pixflat_B600_2x2_17sep2009_specflip.fits.gz b/pypeit/data/pixelflats/PYPEIT_LRISb_pixflat_B600_2x2_17sep2009_specflip.fits.gz new file mode 100644 index 0000000000..f22c724280 Binary files /dev/null and b/pypeit/data/pixelflats/PYPEIT_LRISb_pixflat_B600_2x2_17sep2009_specflip.fits.gz differ diff --git a/pypeit/data/pixelflats/README b/pypeit/data/pixelflats/README new file mode 100644 index 0000000000..3617e4096c --- /dev/null +++ b/pypeit/data/pixelflats/README @@ -0,0 +1,11 @@ +Directory includes custom pixel flats to be used for Flat Fielding. + +The general naming structure is: +'pixelflat_{spec_name}{dispname}{dichroic}_{binning}_{date}.fits.gz' + +spec_name: spectrograph name (e.g. keck_lris_blue) +dispname: metadata dispname (if part of the spectrograph configuration keys) +dichroic: metadata dichroic (if part of the spectrograph configuration keys) +binning: spectral x spatial binning +date: date of the observations + diff --git a/pypeit/data/pixelflats/pixelflat_keck_hires_RED_1x2_20160330.fits.gz b/pypeit/data/pixelflats/pixelflat_keck_hires_RED_1x2_20160330.fits.gz new file mode 100644 index 0000000000..e5bb58b7b6 Binary files /dev/null and b/pypeit/data/pixelflats/pixelflat_keck_hires_RED_1x2_20160330.fits.gz differ diff --git a/pypeit/data/pixelflats/pixelflat_keck_hires_RED_1x3_20170223.fits.gz b/pypeit/data/pixelflats/pixelflat_keck_hires_RED_1x3_20170223.fits.gz new file mode 100644 index 0000000000..999703ae24 Binary files /dev/null and b/pypeit/data/pixelflats/pixelflat_keck_hires_RED_1x3_20170223.fits.gz differ diff --git a/pypeit/data/pixelflats/pixelflat_keck_hires_RED_2x2_20170614.fits.gz b/pypeit/data/pixelflats/pixelflat_keck_hires_RED_2x2_20170614.fits.gz new file mode 100644 index 0000000000..e38d2d80f2 Binary files /dev/null and b/pypeit/data/pixelflats/pixelflat_keck_hires_RED_2x2_20170614.fits.gz differ diff --git a/pypeit/data/pixelflats/pixelflat_keck_lris_blue_600_4000_d560_2x2_20210411.fits.gz b/pypeit/data/pixelflats/pixelflat_keck_lris_blue_600_4000_d560_2x2_20210411.fits.gz new file mode 100644 index 0000000000..9eba3afdf1 Binary files /dev/null and b/pypeit/data/pixelflats/pixelflat_keck_lris_blue_600_4000_d560_2x2_20210411.fits.gz differ diff --git a/pypeit/flatfield.py b/pypeit/flatfield.py index dc83d693a6..b7d5868ee8 100644 --- a/pypeit/flatfield.py +++ b/pypeit/flatfield.py @@ -5,11 +5,15 @@ .. include:: ../include/links.rst """ +from pathlib import Path +from copy import deepcopy import inspect import numpy as np from scipy import interpolate, ndimage +from astropy.io import fits + from matplotlib import pyplot as plt from matplotlib import gridspec @@ -22,13 +26,21 @@ from pypeit import datamodel from pypeit import calibframe +from pypeit import edgetrace +from pypeit import io from pypeit.display import display +from pypeit.images import buildimage from pypeit.core import qa from pypeit.core import flat from pypeit.core import tracewave from pypeit.core import basis from pypeit.core import fitting +from pypeit.core import parse +from pypeit.core.mosaic import build_image_mosaic +from pypeit.spectrographs.util import load_spectrograph from pypeit import slittrace +from pypeit import dataPaths +from pypeit import cache class FlatImages(calibframe.CalibFrame): @@ -483,10 +495,14 @@ class FlatField: corrections. If None, the default parameters are used. slits (:class:`~pypeit.slittrace.SlitTraceSet`): The current slit traces. - wavetilts (:class:`~pypeit.wavetilts.WaveTilts`): - The current wavelength tilt traces; see - wv_calib (:class:`~pypeit.wavecalib.WaveCalib`): - Wavelength calibration object + wavetilts (:class:`~pypeit.wavetilts.WaveTilts`, optional): + The current fit to the wavelength tilts. I can be None, + for example, if slitless is True. + wv_calib (:class:`~pypeit.wavecalib.WaveCalib`, optional): + Wavelength calibration object. It can be None, for example, if + slitless is True. + slitless (bool, optional): + True if the input rawflatimg is a slitless flat. Default is False. spat_illum_only (bool, optional): Only perform the spatial illumination calculation, and ignore the 2D bspline fit. This should only be set to true if you @@ -510,8 +526,8 @@ class FlatField: Image of the relative spectral illumination for a multislit spectrograph """ - def __init__(self, rawflatimg, spectrograph, flatpar, slits, wavetilts, wv_calib, - spat_illum_only=False, qa_path=None, calib_key=None): + def __init__(self, rawflatimg, spectrograph, flatpar, slits, wavetilts=None, wv_calib=None, + slitless=False, spat_illum_only=False, qa_path=None, calib_key=None): # Defaults self.spectrograph = spectrograph @@ -528,7 +544,8 @@ def __init__(self, rawflatimg, spectrograph, flatpar, slits, wavetilts, wv_calib self.wv_calib = wv_calib # Worth a check - self.wavetilts.is_synced(self.slits) + if self.wavetilts is not None and not slitless: + self.wavetilts.is_synced(self.slits) # Attributes unique to this Object self.rawflatimg = rawflatimg # Un-normalized pixel flat as a PypeItImage @@ -540,6 +557,13 @@ def __init__(self, rawflatimg, spectrograph, flatpar, slits, wavetilts, wv_calib self.spat_illum_only = spat_illum_only self.spec_illum = None # Relative spectral illumination image self.waveimg = None + self.slitless = slitless # is this a slitless flat? + + # get waveimg here if available + if self.wavetilts is None or self.wv_calib is None: + msgs.warn("Wavelength calib or tilts are not available. Wavelength image not generated.") + else: + self.build_waveimg() # this set self.waveimg # Completed steps self.steps = [] @@ -578,13 +602,20 @@ def run(self, doqa=False, debug=False, show=False): :class:`FlatImages`: Container with the results of the flat-field analysis. """ + + # check if self.wavetilts is available. It can be None if the flat is slitless, but it's needed otherwise + if self.wavetilts is None and not self.slitless: + msgs.warn("Wavelength tilts are not available. Cannot generate this flat image.") + return None + # Fit it # NOTE: Tilts do not change and self.slits is updated internally. if not self.flatpar['fit_2d_det_response']: # This spectrograph does not have a structure correction # implemented. Ignore detector structure. self.fit(spat_illum_only=self.spat_illum_only, doqa=doqa, debug=debug) - else: # Iterate on the pixelflat if required by the spectrograph + elif self.waveimg is not None: + # Iterate on the pixelflat if required by the spectrograph # User has requested a structure correction. # Note: This will only be performed if it is coded for each individual spectrograph. # Make a copy of the original flat @@ -671,11 +702,14 @@ def build_waveimg(self): Generate an image of the wavelength of each pixel. """ msgs.info("Generating wavelength image") - flex = self.wavetilts.spat_flexure - slitmask = self.slits.slit_img(initial=True, flexure=flex) - tilts = self.wavetilts.fit2tiltimg(slitmask, flexure=flex) - # Save to class attribute for inclusion in the Flat calibration frame - self.waveimg = self.wv_calib.build_waveimg(tilts, self.slits, spat_flexure=flex) + if self.wavetilts is None or self.wv_calib is None: + msgs.error("Wavelength calib or tilts are not available. Cannot generate wavelength image.") + else: + flex = self.wavetilts.spat_flexure + slitmask = self.slits.slit_img(initial=True, flexure=flex) + tilts = self.wavetilts.fit2tiltimg(slitmask, flexure=flex) + # Save to class attribute for inclusion in the Flat calibration frame + self.waveimg = self.wv_calib.build_waveimg(tilts, self.slits, spat_flexure=flex) def show(self, wcs_match=True): """ @@ -816,9 +850,6 @@ def fit(self, spat_illum_only=False, doqa=True, debug=False): npoly = self.flatpar['twod_fit_npoly'] saturated_slits = self.flatpar['saturated_slits'] - # Build wavelength image -- not always used, but for convenience done here - if self.waveimg is None: self.build_waveimg() - # Setup images nspec, nspat = self.rawflatimg.image.shape rawflat = self.rawflatimg.image @@ -833,14 +864,11 @@ def fit(self, spat_illum_only=False, doqa=True, debug=False): ivar_log = gpm_log.astype(float)/0.5**2 # Get the non-linear count level - # TODO: This is currently hacked to deal with Mosaics - try: + if self.rawflatimg.is_mosaic: + # if this is a mosaic we take the maximum value among all the detectors + nonlinear_counts = np.max([rawdets.nonlinear_counts() for rawdets in self.rawflatimg.detector.detectors]) + else: nonlinear_counts = self.rawflatimg.detector.nonlinear_counts() - except: - nonlinear_counts = 1e10 - # Other setup -# nonlinear_counts = self.spectrograph.nonlinear_counts(self.rawflatimg.detector) -# nonlinear_counts = self.rawflatimg.detector.nonlinear_counts() # TODO -- JFH -- CONFIRM THIS SHOULD BE ON INIT # It does need to be *all* of the slits @@ -962,10 +990,13 @@ def fit(self, spat_illum_only=False, doqa=True, debug=False): # TODO: Put this stuff in a self.spectral_fit method? # Create the tilts image for this slit - # TODO -- JFH Confirm the sign of this shift is correct! - _flexure = 0. if self.wavetilts.spat_flexure is None else self.wavetilts.spat_flexure - tilts = tracewave.fit2tilts(rawflat.shape, self.wavetilts['coeffs'][:,:,slit_idx], - self.wavetilts['func2d'], spat_shift=-1*_flexure) + if self.slitless: + tilts = np.tile(np.arange(rawflat.shape[0]) / rawflat.shape[0], (rawflat.shape[1], 1)).T + else: + # TODO -- JFH Confirm the sign of this shift is correct! + _flexure = 0. if self.wavetilts.spat_flexure is None else self.wavetilts.spat_flexure + tilts = tracewave.fit2tilts(rawflat.shape, self.wavetilts['coeffs'][:,:,slit_idx], + self.wavetilts['func2d'], spat_shift=-1*_flexure) # Convert the tilt image to an image with the spectral pixel index spec_coo = tilts * (nspec-1) @@ -1291,11 +1322,11 @@ def fit(self, spat_illum_only=False, doqa=True, debug=False): # go bad? # Minimum wavelength? - if self.flatpar['pixelflat_min_wave'] is not None: + if self.flatpar['pixelflat_min_wave'] is not None and self.waveimg is not None: bad_wv = self.waveimg[onslit_tweak] < self.flatpar['pixelflat_min_wave'] self.mspixelflat[np.where(onslit_tweak)[0][bad_wv]] = 1. # Maximum wavelength? - if self.flatpar['pixelflat_max_wave'] is not None: + if self.flatpar['pixelflat_max_wave'] is not None and self.waveimg is not None: bad_wv = self.waveimg[onslit_tweak] > self.flatpar['pixelflat_max_wave'] self.mspixelflat[np.where(onslit_tweak)[0][bad_wv]] = 1. @@ -1410,6 +1441,10 @@ def spatial_fit_finecorr(self, normed, onslit_tweak, slit_idx, slit_spat, gpm, s illumflat_finecorr: `numpy.ndarray`_ An image (same shape as normed) containing the fine correction to the spatial illumination profile """ + # check id self.waveimg is available + if self.waveimg is None: + msgs.warn("Cannot perform the fine correction to the spatial illumination without the wavelength image.") + return # TODO :: Include fit_order in the parset?? fit_order = np.array([3, 6]) slit_txt = self.slits.slitord_txt @@ -1421,7 +1456,8 @@ def spatial_fit_finecorr(self, normed, onslit_tweak, slit_idx, slit_spat, gpm, s onslit_tweak_trim = self.slits.slit_img(pad=-slit_trim, slitidx=slit_idx, initial=False) == slit_spat # Setup slitimg = (slit_spat + 1) * onslit_tweak.astype(int) - 1 # Need to +1 and -1 so that slitimg=-1 when off the slit - left, right, msk = self.slits.select_edges(flexure=self.wavetilts.spat_flexure) + + left, right, msk = self.slits.select_edges(flexure=self.wavetilts.spat_flexure if self.wavetilts is not None else 0.0) this_left = left[:, slit_idx] this_right = right[:, slit_idx] slitlen = int(np.median(this_right - this_left)) @@ -1431,7 +1467,7 @@ def spatial_fit_finecorr(self, normed, onslit_tweak, slit_idx, slit_spat, gpm, s this_wave = self.waveimg[this_slit] xpos_img = self.slits.spatial_coordinate_image(slitidx=slit_idx, slitid_img=slitimg, - flexure_shift=self.wavetilts.spat_flexure) + flexure_shift=self.wavetilts.spat_flexure if self.wavetilts is not None else 0.0) # Generate the trimmed versions for fitting this_slit_trim = np.where(onslit_tweak_trim & self.rawflatimg.select_flag(invert=True)) this_wave_trim = self.waveimg[this_slit_trim] @@ -1496,6 +1532,11 @@ def extract_structure(self, rawflat_orig, slit_trim=3): divided by the spectral and spatial illumination profile fits). """ msgs.info("Extracting flatfield structure") + + # check if the waveimg is available + if self.waveimg is None: + msgs.error("Cannot perform the extraction of the flatfield structure without the wavelength image.") + # Build the mask and make a temporary instance of FlatImages bpmflats = self.build_mask() # Initialise bad splines (for when the fit goes wrong) @@ -1519,7 +1560,7 @@ def extract_structure(self, rawflat_orig, slit_trim=3): scale_model = illum_profile_spectral(rawflat, self.waveimg, self.slits, slit_illum_ref_idx=self.flatpar['slit_illum_ref_idx'], model=None, gpmask=gpm, skymask=None, trim=self.flatpar['slit_trim'], - flexure=self.wavetilts.spat_flexure, + flexure=self.wavetilts.spat_flexure if self.wavetilts is not None else 0.0, smooth_npix=self.flatpar['slit_illum_smooth_npix']) # Trim the edges by a few pixels to avoid edge effects onslits_trim = gpm & (self.slits.slit_img(pad=-slit_trim, initial=False) != -1) @@ -1565,8 +1606,10 @@ def spectral_illumination(self, gpm=None, debug=False): An image containing the appropriate scaling """ msgs.info("Deriving spectral illumination profile") - # Generate a wavelength image - if self.waveimg is None: self.build_waveimg() + # check if the waveimg is available + if self.waveimg is None: + msgs.warn("Cannot perform the spectral illumination without the wavelength image.") + return None msgs.info('Performing a joint fit to the flat-field response') # Grab some parameters trim = self.flatpar['slit_trim'] @@ -1580,7 +1623,7 @@ def spectral_illumination(self, gpm=None, debug=False): return illum_profile_spectral(rawflat, self.waveimg, self.slits, slit_illum_ref_idx=self.flatpar['slit_illum_ref_idx'], model=None, gpmask=gpm, skymask=None, trim=trim, - flexure=self.wavetilts.spat_flexure, + flexure=self.wavetilts.spat_flexure if self.wavetilts is not None else 0.0, smooth_npix=self.flatpar['slit_illum_smooth_npix'], debug=debug) @@ -1654,6 +1697,208 @@ def tweak_slit_edges(self, left, right, spat_coo, norm_flat, method='threshold', msgs.error("Method for tweaking slit edges not recognized: {0}".format(method)) +class SlitlessFlat: + """ + Class to generate a slitless pixel flat-field calibration image. + + Args: + fitstbl (:class:`~pypeit.metadata.PypeItMetaData`): + The class holding the metadata for all the frames. + slitless_rows (`numpy.ndarray`_): + Boolean array selecting the rows in the fitstbl that + correspond to the slitless frames. + spectrograph (:class:`~pypeit.spectrographs.spectrograph.Spectrograph`): + The spectrograph object. + par (:class:`~pypeit.par.pypeitpar.CalibrationsPar`): + Parameter set defining optional parameters of PypeIt's algorithms + for Calibrations + qa_path (`Path`_): + Path for the QA diagnostics. + + """ + + def __init__(self, fitstbl, slitless_rows, spectrograph, par, qa_path=None): + + self.fitstbl = fitstbl + # Boolean array selecting the rows in the fitstbl that correspond to the slitless frames. + self.slitless_rows = slitless_rows + self.spectrograph = spectrograph + self.par = par + self.qa_path = qa_path + + def slitless_pixflat_fname(self): + """ + Generate the name of the slitless pixel flat file. + + Returns: + :obj:`str`: The name of the slitless pixel flat + + """ + if len(self.slitless_rows) == 0: + msgs.error('No slitless_pixflat frames found. Cannot generate the slitless pixel flat file name.') + + # generate the slitless pixel flat file name + spec_name = self.fitstbl.spectrograph.name + date = self.fitstbl.construct_obstime(self.slitless_rows[0]).iso.split(' ')[0].replace('-', '') if \ + self.fitstbl[self.slitless_rows][0]['mjd'] is not None else '00000000' + # setup info to add to the filename + dispname = '' if 'dispname' not in self.spectrograph.configuration_keys() else \ + f"_{self.fitstbl[self.slitless_rows[0]]['dispname'].replace('/', '_').replace(' ', '_').replace('(', '').replace(')', '').replace(':', '_').replace('+', '_')}" + dichroic = '' if 'dichroic' not in self.spectrograph.configuration_keys() else \ + f"_d{self.fitstbl[self.slitless_rows[0]]['dichroic']}" + binning = self.fitstbl[self.slitless_rows[0]]['binning'].replace(',', 'x') + # file name + return f'pixelflat_{spec_name}{dispname}{dichroic}_{binning}_{date}.fits' + + def make_slitless_pixflat(self, msbias=None, msdark=None, calib_dir=None, write_qa=False, show=False): + """ + Generate and save to disc a slitless pixel flat-field calibration images. + The pixel flat file will have one extension per detector, even in the case of a mosaic. + Contrary to the regular calibration flow, the slitless pixel flat is created for all detectors + of the current spectrograph at once, and not only the one for the current detector. + Since the slitless pixel flat images are saved to disc, this approach helps with the I/O + This is a method is used in `~pypeit.calibrations.get_flats()`. + + Note: par['flatfield']['pixelflat_file'] is updated in this method. + + Args: + msbias (:class:`~pypeit.images.buildimage.BiasImage`, optional): + Bias image for bias subtraction; passed to + :func:`~pypeit.images.buildimage.buildimage_fromlist()` + msdark (:class:`~pypeit.images.buildimage.DarkImage`, optional): + Dark-current image; passed to + :func:`~pypeit.images.buildimage.buildimage_fromlist()` + calib_dir (`Path`_): + Path for the processed calibration files. + write_qa (:obj:`bool`, optional): + Write QA plots to disk? + show (:obj:`bool`, optional): + Show the diagnostic plots? + + Returns: + :obj:`str`: The name of the slitless pixel flat file that was generated. + + """ + + # First thing first, check if the user has provided slitless_pixflat frames + if len(self.slitless_rows) == 0: + # return unchanged self.par['flatfield']['pixelflat_file'] + return self.par['flatfield']['pixelflat_file'] + + # all detectors of this spectrograph + _detectors = np.array(self.spectrograph.select_detectors()) + + # Check if a user-provided slitless pixelflat already exists for the current detectors + if self.par['flatfield']['pixelflat_file'] is not None: + _pixel_flat_file = dataPaths.pixelflat.get_file_path(self.par['flatfield']['pixelflat_file'], + return_none=True) + + if _pixel_flat_file is not None: + # get detector names + detnames = np.array([self.spectrograph.get_det_name(_det) for _det in _detectors]) + # open the file + with io.fits_open(_pixel_flat_file) as hdu: + # list of available detectors in the pixel flat file + file_detnames = [h.name.split('-')[0] for h in hdu] + # check if the current detnames are in the list + in_file = np.array([d in file_detnames for d in detnames]) + # if all detectors are in the file, return + if np.all(in_file): + msgs.info(f"Both slitless_pixflat frames and user-defined file found. " + f"The user-defined file will be used: {self.par['flatfield']['pixelflat_file']}") + # return unchanged self.par['flatfield']['pixelflat_file'] + return self.par['flatfield']['pixelflat_file'] + else: + # get the detectors that are not in the file + _detectors = _detectors[np.logical_not(in_file)] + detnames = detnames[np.logical_not(in_file)] + msgs.info(f'Both slitless_pixflat frames and user-defined file found, but the ' + f'following detectors are not in the file: {detnames}. Using the ' + f'slitless_pixflat frames to generate the missing detectors.') + + # make the slitless pixel flat + pixflat_norm_list = [] + detname_list = [] + for _det in _detectors: + # Parse the raw slitless pixelflat frames. Note that this is spectrograph dependent. + # If the method does not exist in the specific spectrograph class, nothing will happen + this_raw_idx = self.spectrograph.parse_raw_files(self.fitstbl[self.slitless_rows], det=_det, + ftype='slitless_pixflat') + if len(this_raw_idx) == 0: + msgs.warn(f'No raw slitless_pixflat frames found for {self.spectrograph.get_det_name(_det)}. ' + f'Continuing...') + continue + this_raw_files = self.fitstbl.frame_paths(self.slitless_rows[this_raw_idx]) + msgs.info(f'Creating slitless pixel-flat calibration frame ' + f'for {self.spectrograph.get_det_name(_det)} using files: ') + for f in this_raw_files: + msgs.prindent(f'{Path(f).name}') + + # Reset the BPM + msbpm = self.spectrograph.bpm(this_raw_files[0], _det, msbias=msbias if self.par['bpm_usebias'] else None) + + # trace image + traceimg = buildimage.buildimage_fromlist(self.spectrograph, _det, self.par['traceframe'], + [this_raw_files[0]], dark=msdark, bias=msbias, bpm=msbpm) + # slit edges + # we need to change some parameters for the slit edge tracing + edges_par = deepcopy(self.par['slitedges']) + # lower the threshold for edge detection + edges_par['edge_thresh'] = 50. + # this is used for longslit (i.e., no pca) + edges_par['sync_predict'] = 'nearest' + # remove spurious edges by setting a large minimum slit gap (20% of the detector size + platescale = parse.parse_binning(traceimg.detector.binning)[1] * traceimg.detector['platescale'] + edges_par['minimum_slit_gap'] = 0.2 * traceimg.image.shape[1] * platescale + # if no slits are found the bound_detector parameter add 2 traces at the detector edges + edges_par['bound_detector'] = True + # set the buffer to 0 + edges_par['det_buffer'] = 0 + _spectrograph = deepcopy(self.spectrograph) + # need to treat this as a MultiSlit spectrograph (no echelle parameters used) + _spectrograph.pypeline = 'MultiSlit' + edges = edgetrace.EdgeTraceSet(traceimg, _spectrograph, edges_par, auto=True) + slits = edges.get_slits() + if show: + edges.show(title='Slitless flat edge tracing') + # + # flat image + slitless_pixel_flat = buildimage.buildimage_fromlist(self.spectrograph, _det, self.par['slitless_pixflatframe'], + this_raw_files, dark=msdark, bias=msbias, bpm=msbpm) + + # increase saturation threshold (some hires slitless flats are very bright) + slitless_pixel_flat.detector.saturation *= 1.5 + # Initialise the pixel flat + flatpar = deepcopy(self.par['flatfield']) + # do not tweak the slits + flatpar['tweak_slits'] = False + pixelFlatField = FlatField(slitless_pixel_flat, self.spectrograph, flatpar, slits, wavetilts=None, + wv_calib=None, slitless=True, qa_path=self.qa_path) + + # Generate + pixelflatImages = pixelFlatField.run(doqa=write_qa, show=show) + pixflat_norm_list.append(pixelflatImages.pixelflat_norm) + detname_list.append(self.spectrograph.get_det_name(_det)) + + if len(detname_list) > 0: + # get the pixel flat file name + if self.par['flatfield']['pixelflat_file'] is not None and _pixel_flat_file is not None: + fname = self.par['flatfield']['pixelflat_file'] + else: + fname = self.slitless_pixflat_fname() + # file will be saved in the reduction directory, but also cached in the data/pixelflats folder + # therefore we update self.par['flatfield']['pixelflat_file'] to the new file, + # so that it can be used for the rest of the reduction and for the other files in the same run + self.par['flatfield']['pixelflat_file'] = fname + + # Save the result + write_pixflat_to_fits(pixflat_norm_list, detname_list, self.spectrograph.name, + calib_dir.parent if calib_dir is not None else Path('.').absolute(), + fname, to_cache=True) + + return self.par['flatfield']['pixelflat_file'] + + def spatillum_finecorr_qa(normed, finecorr, left, right, ypos, cut, outfile=None, title=None, half_slen=50): """ Plot the QA for the fine correction fits to the spatial illumination profile @@ -2100,4 +2345,203 @@ def merge(init_cls, merge_cls): return FlatImages(**dd) +def write_pixflat_to_fits(pixflat_norm_list, detname_list, spec_name, outdir, pixelflat_name, to_cache=True): + """ + Write the pixel-to-pixel flat-field images to a FITS file. + The FITS file will have an extension for each detector (never a mosaic). + The `load_pixflat()` method read this file and transform it into a mosaic if needed. + This image is generally used as a user-provided pixel flat-field image and ingested + in the reduction using the `pixelflat_file` parameter in the PypeIt file. + + Args: + pixflat_norm_list (:obj:`list`): + List of 2D `numpy.ndarray`_ arrays containing the pixel-to-pixel flat-field images. + detname_list (:obj:`list`): + List of detector names. + spec_name (:obj:`str`): + Name of the spectrograph. + outdir (:obj:`pathlib.Path`): + Path to the output directory. + pixelflat_name (:obj:`str`): + Name of the output file to be written. + to_cache (:obj:`bool`, optional): + If True, the file will be written to the cache directory pypeit/data/pixflats. + + """ + + msgs.info("Writing the pixel-to-pixel flat-field images to a FITS file.") + + # Check that the number of detectors matches the number of pixelflat_norm arrays + if len(pixflat_norm_list) != len(detname_list): + msgs.error("The number of detectors does not match the number of pixelflat_norm arrays. " + "The pixelflat file cannot be written.") + + # local output (reduction directory) + pixelflat_file = outdir / pixelflat_name + + # Check if the file already exists + old_hdus = [] + old_detnames = [] + old_hdr = None + if pixelflat_file.exists(): + msgs.warn("The pixelflat file already exists. It will be overwritten/updated.") + old_hdus = fits.open(pixelflat_file) + old_detnames = [h.name.split('-')[0] for h in old_hdus] # this has also 'PRIMARY' + old_hdr = old_hdus[0].header + + # load spectrograph + spec = load_spectrograph(spec_name) + + # Create the new HDUList + _hdr = io.initialize_header(hdr=old_hdr) + prihdu = fits.PrimaryHDU(header=_hdr) + prihdu.header['CALIBTYP'] = (FlatImages.calib_type, 'PypeIt: Calibration frame type') + new_hdus = [prihdu] + + extnum = 1 + for d in spec.select_detectors(): + detname = spec.get_det_name(d) + extname = f'{detname}-PIXELFLAT_NORM' + # update or add the detectors that we want to save + if detname in detname_list: + det_idx = detname_list.index(detname) + pixflat_norm = pixflat_norm_list[det_idx] + hdu = fits.ImageHDU(data=pixflat_norm, name=extname) + prihdu.header[f'EXT{extnum:04d}'] = hdu.name + new_hdus.append(hdu) + # keep the old detectors that were not updated + elif detname in old_detnames: + old_det_idx = old_detnames.index(detname) + hdu = old_hdus[old_det_idx] + prihdu.header[f'EXT{extnum:04d}'] = hdu.name + new_hdus.append(hdu) + extnum += 1 + + # Write the new HDUList + new_hdulist = fits.HDUList(new_hdus) + # Check if the directory exists + if not pixelflat_file.parent.is_dir(): + pixelflat_file.parent.mkdir(parents=True) + new_hdulist.writeto(pixelflat_file, overwrite=True) + msgs.info(f'A slitless Pixel Flat file for detectors {detname_list} has been saved to {msgs.newline()}' + f'{pixelflat_file}') + + # common msg + add_msgs = f"add the following to your PypeIt Reduction File:{msgs.newline()}" \ + f" [calibrations]{msgs.newline()}" \ + f" [[flatfield]]{msgs.newline()}" \ + f" pixelflat_file = {pixelflat_name}{msgs.newline()}{msgs.newline()}{msgs.newline()}" \ + f"Please consider sharing your Pixel Flat file with the PypeIt Developers.{msgs.newline()}" \ + + + if to_cache: + # NOTE that the file saved in the cache is gzipped, while the one saved in the outdir is not + # This prevents `dataPaths.pixelflat.get_file_path()` from returning the file saved in the outdir + cache.write_file_to_cache(pixelflat_file, pixelflat_name+'.gz', f"pixelflats") + msgs.info(f"The slitless Pixel Flat file has also been saved to the PypeIt cache directory {msgs.newline()}" + f"{str(dataPaths.pixelflat)} {msgs.newline()}" + f"It will be automatically used in this run. " + f"If you want to use this file in future runs, {add_msgs}") + else: + msgs.info(f"To use this file, move it to the PypeIt data directory {msgs.newline()}" + f"{str(dataPaths.pixelflat)} {msgs.newline()} and {add_msgs}") + + +def load_pixflat(pixel_flat_file, spectrograph, det, flatimages, calib_dir=None, chk_version=False): + """ + Load a pixel flat from a file and add it to the flatimages object. + The pixel flat file has one detector per extension, even in the case of a mosaic. + Therefore, if this is a mosaic reduction, this script will construct a pixel flat + mosaic. The Edges file needs to exist in the Calibration Folder, since the mosaic + parameters are pulled from it. + This is used in `~pypeit.calibrations.get_flats()`. + + Args: + pixel_flat_file (:obj:`str`): + Name of the pixel flat file. + spectrograph (:class:`~pypeit.spectrographs.spectrograph.Spectrograph`): + The spectrograph object. + det (:obj:`int`, :obj:`tuple`): + The single detector or set of detectors in a mosaic to process. + flatimages (:class:`~pypeit.flatfield.FlatImages`): + The flat field images object. + calib_dir (:obj:`str`, optional): + The path to the calibration directory. + chk_version (:obj:`bool`, optional): + Check the version of the file. + + Returns: + :class:`~pypeit.flatfield.FlatImages`: The flat images object with the pixel flat added. + + """ + # Check if the pixel flat file exists + if pixel_flat_file is None: + msgs.error('No pixel flat file defined. Cannot load the pixel flat!') + + # get the path + _pixel_flat_file = dataPaths.pixelflat.get_file_path(pixel_flat_file, return_none=True) + if _pixel_flat_file is None: + msgs.error(f'Cannot load the pixel flat file, {pixel_flat_file}. It is not a direct path, ' + f'a cached file, or a file that can be downloaded from a PypeIt repository.') + + # If this is a mosaic, we need to construct the pixel flat mosaic + if isinstance(det, tuple): + # We need to grab mosaic info from another existing calibration frame. + # We use EdgeTraceSet image to get `tform` and `msc_ord`. Check if EdgeTraceSet file exists. + edges_file = Path(edgetrace.EdgeTraceSet.construct_file_name(flatimages.calib_key, + calib_dir=calib_dir)).absolute() + if not edges_file.exists(): + msgs.error('Edges file not found in the Calibrations folder. ' + 'It is needed to grab the mosaic parameters to load and mosaic the input pixel flat!') + + # Load detector info from EdgeTraceSet file + traceimg = edgetrace.EdgeTraceSet.from_file(edges_file, chk_version=chk_version).traceimg + det_info = traceimg.detector + # check that the mosaic parameters are defined + if not np.all(np.in1d(['tform', 'msc_ord'], list(det_info.keys()))) or \ + det_info.tform is None or det_info.msc_ord is None: + msgs.error('Mosaic parameters are not defined in the Edges frame. Cannot load the pixel flat!') + + # read the file + with io.fits_open(_pixel_flat_file) as hdu: + # list of available detectors in the pixel flat file + file_dets = [int(h.name.split('-')[0].split('DET')[1]) for h in hdu[1:]] + # check if all detectors required for the mosaic are in the list + if not np.all(np.in1d(list(det), file_dets)): + msgs.error(f'Not all detectors in the mosaic are in the pixel flat file: ' + f'{pixel_flat_file}. Cannot load the pixel flat!') + + # get the pixel flat images of only the detectors in the mosaic + pixflat_images = np.concatenate([hdu[f'DET{d:02d}-PIXELFLAT_NORM'].data[None,:,:] for d in det]) + # construct the pixel flat mosaic + pixflat_msc, _,_,_ = build_image_mosaic(pixflat_images, det_info.tform, order=det_info.msc_ord) + # check that the mosaic has the correct shape + if pixflat_msc.shape != traceimg.image.shape: + msgs.error('The constructed pixel flat mosaic does not have the correct shape. ' + 'Cannot load this pixel flat as a mosaic!') + msgs.info(f'Using pixelflat file: {pixel_flat_file} ' + f'for {spectrograph.get_det_name(det)}.') + nrm_image = FlatImages(pixelflat_norm=pixflat_msc) + + # If this is not a mosaic, we can simply read the pixel flat for the current detector + else: + # current detector name + detname = spectrograph.get_det_name(det) + # read the file + with io.fits_open(_pixel_flat_file) as hdu: + # list of available detectors in the pixel flat file + file_detnames = [h.name.split('-')[0] for h in hdu] # this list has also the 'PRIMARY' extension + # check if the current detector is in the list + if detname in file_detnames: + # get the index of the current detector + idx = file_detnames.index(detname) + # get the pixel flat image + msgs.info(f'Using pixelflat file: {pixel_flat_file} for {detname}.') + nrm_image = FlatImages(pixelflat_norm=hdu[idx].data) + else: + msgs.error(f'{detname} not found in the pixel flat file: ' + f'{pixel_flat_file}. Cannot load the pixel flat!') + nrm_image = None + + return merge(flatimages, nrm_image) diff --git a/pypeit/images/combineimage.py b/pypeit/images/combineimage.py index 1e20c37710..99fb4f1629 100644 --- a/pypeit/images/combineimage.py +++ b/pypeit/images/combineimage.py @@ -195,6 +195,33 @@ def run(self, ignore_saturation=False, maxiters=5): else: comb_texp = exptime[0] + # scale the images to their mean, if requested, before combining + if self.par['scale_to_mean']: + msgs.info("Scaling images to have the same mean before combining") + # calculate the mean of the images + [mean_img], _, mean_gpm, _ = combine.weighted_combine(np.ones(self.nimgs, dtype=float)/self.nimgs, + [img_stack], + [rn2img_stack], + # var_list is added because it is + # required by the function but not used + gpm_stack, sigma_clip=self.par['clip'], + sigma_clip_stack=img_stack, + sigrej=self.par['comb_sigrej'], maxiters=maxiters) + + # scale factor + # TODO: Chose the median over the whole frame to avoid outliers. Is this the right choice? + _mscale = np.nanmedian(mean_img[None, mean_gpm]/img_stack[:, mean_gpm], axis=1) + # reshape the scale factor + mscale = _mscale[:, None, None] + # scale the images + img_stack *= mscale + # scale the scales + scl_stack *= mscale + + # scale the variances + rn2img_stack *= mscale**2 + basev_stack *= mscale**2 + # Coadd them if self.par['combine'] == 'mean': weights = np.ones(self.nimgs, dtype=float)/self.nimgs diff --git a/pypeit/images/mosaic.py b/pypeit/images/mosaic.py index 9bf4d95355..1ab8dd5fdc 100644 --- a/pypeit/images/mosaic.py +++ b/pypeit/images/mosaic.py @@ -30,7 +30,7 @@ class Mosaic(datamodel.DataContainer): """ # Set the version of this class - version = '1.0.0' + version = '1.0.1' # WARNING: `binning` and `platescale` have the same names as datamodel # components in pypeit.images.detector_container.DetectorContainer. This is @@ -53,14 +53,14 @@ class Mosaic(datamodel.DataContainer): 'tform': dict(otype=np.ndarray, atype=float, descr='The full transformation matrix for each detector used to ' 'construct the mosaic.'), - 'msc_order': dict(otype=int, descr='Order of the interpolation used to construct the mosaic.')} + 'msc_ord': dict(otype=int, descr='Order of the interpolation used to construct the mosaic.')} name_prefix = 'MSC' """ Prefix for the name of the mosaic. """ - def __init__(self, id, detectors, shape, shift, rot, tform, msc_order): + def __init__(self, id, detectors, shape, shift, rot, tform, msc_ord): args, _, _, values = inspect.getargvalues(inspect.currentframe()) d = dict([(k,values[k]) for k in args[1:]]) @@ -107,8 +107,8 @@ def _bundle(self): tbl['rot'] = self.rot if self.tform is not None: tbl['tform'] = self.tform - if self.msc_order is not None: - tbl.meta['msc_order'] = self.msc_order + if self.msc_ord is not None: + tbl.meta['msc_ord'] = self.msc_ord if self.id is not None: tbl.meta['id'] = self.id if self.shape is not None: @@ -213,5 +213,5 @@ def copy(self): """ Return a (deep) copy of the object. """ - return Mosaic(id=self.id, detectors=np.array([det.copy() for det in self.detectors]), shape=self.shape, shift=self.shift.copy(), rot=self.rot.copy(), tform=self.tform.copy(), msc_order=self.msc_order) + return Mosaic(id=self.id, detectors=np.array([det.copy() for det in self.detectors]), shape=self.shape, shift=self.shift.copy(), rot=self.rot.copy(), tform=self.tform.copy(), msc_ord=self.msc_ord) diff --git a/pypeit/images/rawimage.py b/pypeit/images/rawimage.py index 0e8fb26758..8389cc0ac9 100644 --- a/pypeit/images/rawimage.py +++ b/pypeit/images/rawimage.py @@ -1358,7 +1358,7 @@ def build_mosaic(self): # Transform the image data to the mosaic frame. This call determines # the shape of the mosaic image and adjusts the relative transforms to # the absolute mosaic frame. - self.image, _, _img_npix, _tforms = build_image_mosaic(self.image, self.mosaic.tform, order=self.mosaic.msc_order) + self.image, _, _img_npix, _tforms = build_image_mosaic(self.image, self.mosaic.tform, order=self.mosaic.msc_ord) shape = self.image.shape # Maintain dimensionality self.image = np.expand_dims(self.image, 0) @@ -1369,7 +1369,7 @@ def build_mosaic(self): # Transform the BPM and maintain its type bpm_type = self.bpm.dtype - self._bpm = build_image_mosaic(self.bpm.astype(float), _tforms, mosaic_shape=shape, order=self.mosaic.msc_order)[0] + self._bpm = build_image_mosaic(self.bpm.astype(float), _tforms, mosaic_shape=shape, order=self.mosaic.msc_ord)[0] # Include pixels that have no contribution from the original image in # the bad pixel mask of the mosaic. self._bpm[_img_npix < 1] = 1 @@ -1384,29 +1384,29 @@ def build_mosaic(self): # Get the pixels associated with each amplifier self.datasec_img = build_image_mosaic(self.datasec_img.astype(float), _tforms, - mosaic_shape=shape, order=self.mosaic.msc_order)[0] + mosaic_shape=shape, order=self.mosaic.msc_ord)[0] self.datasec_img = np.expand_dims(np.round(self.datasec_img).astype(int), 0) # Get the pixels associated with each detector self.det_img = build_image_mosaic(self.det_img.astype(float), _tforms, - mosaic_shape=shape, order=self.mosaic.msc_order)[0] + mosaic_shape=shape, order=self.mosaic.msc_ord)[0] self.det_img = np.expand_dims(np.round(self.det_img).astype(int), 0) # Transform all the variance arrays, as necessary if self.rn2img is not None: - self.rn2img = build_image_mosaic(self.rn2img, _tforms, mosaic_shape=shape, order=self.mosaic.msc_order)[0] + self.rn2img = build_image_mosaic(self.rn2img, _tforms, mosaic_shape=shape, order=self.mosaic.msc_ord)[0] self.rn2img = np.expand_dims(self.rn2img, 0) if self.dark is not None: - self.dark = build_image_mosaic(self.dark, _tforms, mosaic_shape=shape, order=self.mosaic.msc_order)[0] + self.dark = build_image_mosaic(self.dark, _tforms, mosaic_shape=shape, order=self.mosaic.msc_ord)[0] self.dark = np.expand_dims(self.dark, 0) if self.dark_var is not None: - self.dark_var = build_image_mosaic(self.dark_var, _tforms, mosaic_shape=shape, order=self.mosaic.msc_order)[0] + self.dark_var = build_image_mosaic(self.dark_var, _tforms, mosaic_shape=shape, order=self.mosaic.msc_ord)[0] self.dark_var = np.expand_dims(self.dark_var, 0) if self.proc_var is not None: - self.proc_var = build_image_mosaic(self.proc_var, _tforms, mosaic_shape=shape, order=self.mosaic.msc_order)[0] + self.proc_var = build_image_mosaic(self.proc_var, _tforms, mosaic_shape=shape, order=self.mosaic.msc_ord)[0] self.proc_var = np.expand_dims(self.proc_var, 0) if self.base_var is not None: - self.base_var = build_image_mosaic(self.base_var, _tforms, mosaic_shape=shape, order=self.mosaic.msc_order)[0] + self.base_var = build_image_mosaic(self.base_var, _tforms, mosaic_shape=shape, order=self.mosaic.msc_ord)[0] self.base_var = np.expand_dims(self.base_var, 0) # TODO: Mosaicing means that many of the internals are no longer diff --git a/pypeit/metadata.py b/pypeit/metadata.py index faabcea586..f72a7c20c9 100644 --- a/pypeit/metadata.py +++ b/pypeit/metadata.py @@ -20,7 +20,6 @@ from pypeit import msgs from pypeit import inputfiles from pypeit.core import framematch -from pypeit.core import flux_calib from pypeit.core import parse from pypeit.core import meta from pypeit.io import dict_to_lines @@ -1449,28 +1448,10 @@ def get_frame_types(self, flag_unknown=False, user=None, merge=True): indx = self.spectrograph.check_frame_type(ftype, self.table, exprng=exprng) # Turn on the relevant bits type_bits[indx] = self.type_bitmask.turn_on(type_bits[indx], flag=ftype) - - # Find the nearest standard star to each science frame - # TODO: Should this be 'standard' or 'science' or both? - if 'ra' not in self.keys() or 'dec' not in self.keys(): - msgs.warn('Cannot associate standard with science frames without sky coordinates.') - else: - # TODO: Do we want to do this here? - indx = self.type_bitmask.flagged(type_bits, flag='standard') - for b, f, ra, dec in zip(type_bits[indx], self['filename'][indx], self['ra'][indx], - self['dec'][indx]): - if ra == 'None' or dec == 'None': - msgs.warn('RA and DEC must not be None for file:' + msgs.newline() + f) - msgs.warn('The above file could be a twilight flat frame that was' - + msgs.newline() + 'missed by the automatic identification.') - b = self.type_bitmask.turn_off(b, flag='standard') - continue - # If an object exists within 20 arcmins of a listed standard, - # then it is probably a standard star - foundstd = flux_calib.find_standard_file(ra, dec, check=True) - b = self.type_bitmask.turn_off(b, flag='science' if foundstd else 'standard') - + # Vet assigned frame types (this can be spectrograph dependent) + self.spectrograph.vet_assigned_ftypes(type_bits, self) + # Find the files without any types indx = np.logical_not(self.type_bitmask.flagged(type_bits)) if np.any(indx): diff --git a/pypeit/par/pypeitpar.py b/pypeit/par/pypeitpar.py index 0d4fd0e200..721c8ceac1 100644 --- a/pypeit/par/pypeitpar.py +++ b/pypeit/par/pypeitpar.py @@ -73,6 +73,7 @@ def __init__(self, existing_par=None, foo=None): from pypeit.par import util from pypeit.core.framematch import FrameTypeBitMask from pypeit import msgs +from pypeit import dataPaths def tuple_force(par): @@ -209,6 +210,7 @@ def __init__(self, trim=None, apply_gain=None, orient=None, overscan_method=None, overscan_par=None, combine=None, satpix=None, mask_cr=None, clip=None, + scale_to_mean=None, #cr_sigrej=None, n_lohi=None, #replace=None, lamaxiter=None, grow=None, @@ -372,6 +374,10 @@ def __init__(self, trim=None, apply_gain=None, orient=None, dtypes['clip'] = bool descr['clip'] = 'Perform sigma clipping when combining. Only used with combine=mean' + defaults['scale_to_mean'] = False + dtypes['scale_to_mean'] = bool + descr['scale_to_mean'] = 'If True, scale the input images to have the same mean before combining.' + defaults['comb_sigrej'] = None dtypes['comb_sigrej'] = float descr['comb_sigrej'] = 'Sigma-clipping level for when clip=True; ' \ @@ -457,7 +463,7 @@ def from_dict(cls, cfg): 'overscan_method', 'overscan_par', 'use_darkimage', 'dark_expscale', 'spat_flexure_correct', 'spat_flexure_maxlag', 'use_illumflat', 'use_specillum', 'empirical_rn', 'shot_noise', 'noise_floor', 'use_pixelflat', 'combine', - 'correct_nonlinear', 'satpix', #'calib_setup_and_bit', + 'scale_to_mean', 'correct_nonlinear', 'satpix', #'calib_setup_and_bit', 'n_lohi', 'mask_cr', 'lamaxiter', 'grow', 'clip', 'comb_sigrej', 'rmcompact', 'sigclip', 'sigfrac', 'objlim'] @@ -833,9 +839,13 @@ def validate(self): return # Check the frame exists - if not os.path.isfile(self.data['pixelflat_file']): - raise ValueError('Provided frame file name does not exist: {0}'.format( - self.data['pixelflat_file'])) + # only the file name is provided, so we need to check if the file exists + # in the right place (data/pixelflats) + file_path = dataPaths.pixelflat.get_file_path(self.data['pixelflat_file'], return_none=True) + if file_path is None: + msgs.error( + f'Provided pixelflat file, {self.data["pixelflat_file"]} not found. It is not a direct path, ' + 'a cached file, or a file that can be downloaded from a PypeIt repository.') # Check that if tweak slits is true that illumflatten is alwo true # TODO -- We don't need this set, do we?? See the desc of tweak_slits above @@ -4495,7 +4505,7 @@ class CalibrationsPar(ParSet): def __init__(self, calib_dir=None, bpm_usebias=None, biasframe=None, darkframe=None, arcframe=None, tiltframe=None, pixelflatframe=None, pinholeframe=None, alignframe=None, alignment=None, traceframe=None, illumflatframe=None, - lampoffflatsframe=None, scattlightframe=None, skyframe=None, standardframe=None, + lampoffflatsframe=None, slitless_pixflatframe=None, scattlightframe=None, skyframe=None, standardframe=None, scattlight_pad=None, flatfield=None, wavelengths=None, slitedges=None, tilts=None, raise_chk_error=None): @@ -4581,6 +4591,15 @@ def __init__(self, calib_dir=None, bpm_usebias=None, biasframe=None, darkframe=N dtypes['lampoffflatsframe'] = [ ParSet, dict ] descr['lampoffflatsframe'] = 'The frames and combination rules for the lamp off flats' + defaults['slitless_pixflatframe'] = FrameGroupPar(frametype='slitless_pixflat', + process=ProcessImagesPar(satpix='nothing', + use_pixelflat=False, + use_illumflat=False, + use_specillum=False, + combine='median')) + dtypes['slitless_pixflatframe'] = [ ParSet, dict ] + descr['slitless_pixflatframe'] = 'The frames and combination rules for the slitless pixel flat' + defaults['pinholeframe'] = FrameGroupPar(frametype='pinhole') dtypes['pinholeframe'] = [ ParSet, dict ] descr['pinholeframe'] = 'The frames and combination rules for the pinholes' @@ -4672,7 +4691,7 @@ def from_dict(cls, cfg): parkeys = [ 'calib_dir', 'bpm_usebias', 'raise_chk_error'] allkeys = parkeys + ['biasframe', 'darkframe', 'arcframe', 'tiltframe', 'pixelflatframe', - 'illumflatframe', 'lampoffflatsframe', 'scattlightframe', + 'illumflatframe', 'lampoffflatsframe', 'slitless_pixflatframe', 'scattlightframe', 'pinholeframe', 'alignframe', 'alignment', 'traceframe', 'standardframe', 'skyframe', 'scattlight_pad', 'flatfield', 'wavelengths', 'slitedges', 'tilts'] badkeys = np.array([pk not in allkeys for pk in k]) @@ -4698,6 +4717,8 @@ def from_dict(cls, cfg): kwargs[pk] = FrameGroupPar.from_dict('illumflat', cfg[pk]) if pk in k else None pk = 'lampoffflatsframe' kwargs[pk] = FrameGroupPar.from_dict('lampoffflats', cfg[pk]) if pk in k else None + pk = 'slitless_pixflatframe' + kwargs[pk] = FrameGroupPar.from_dict('slitless_pixflat', cfg[pk]) if pk in k else None pk = 'pinholeframe' kwargs[pk] = FrameGroupPar.from_dict('pinhole', cfg[pk]) if pk in k else None pk = 'scattlightframe' diff --git a/pypeit/pypeitdata.py b/pypeit/pypeitdata.py index 5cfad9b8cc..779aeb4241 100644 --- a/pypeit/pypeitdata.py +++ b/pypeit/pypeitdata.py @@ -224,7 +224,7 @@ def _parse_format(f): return _f.suffix.replace('.','').lower() def get_file_path(self, data_file, force_update=False, to_pkg=None, return_format=False, - quiet=False): + return_none=False, quiet=False): """ Return the path to a file. @@ -266,6 +266,9 @@ def get_file_path(self, data_file, force_update=False, to_pkg=None, return_forma If True, the returned object is a :obj:`tuple` that includes the file path and its format (e.g., ``'fits'``). If False, only the file path is returned. + return_none (:obj:`bool`, optional): + If True, return None if the file does not exist. If False, an + error is raised if the file does not exist. quiet (:obj:`bool`, optional): Suppress messages @@ -300,7 +303,10 @@ def get_file_path(self, data_file, force_update=False, to_pkg=None, return_forma # if the file exists in the cache and force_update is False. subdir = str(self.path.relative_to(self.data)) _cached_file = cache.fetch_remote_file(data_file, subdir, remote_host=self.host, - force_update=force_update) + force_update=force_update, return_none=return_none) + if _cached_file is None: + msgs.warn(f'File {data_file} not found in the cache.') + return None # If we've made it this far, the file is being pulled from the cache. if to_pkg is None: @@ -359,6 +365,8 @@ class PypeItDataPaths: 'skisim': {'path': 'skisim', 'host': 'github'}, 'filters': {'path': 'filters', 'host': None}, 'sensfunc': {'path': 'sensfuncs', 'host': 'github'}, + # Pixel Flats + 'pixelflat': {'path': 'pixelflats', 'host': 'github'}, # Other 'sky_spec': {'path': 'sky_spec', 'host': None}, 'static_calibs': {'path': 'static_calibs', 'host': None}, diff --git a/pypeit/scripts/__init__.py b/pypeit/scripts/__init__.py index b16da7c36e..9f4bef02c6 100644 --- a/pypeit/scripts/__init__.py +++ b/pypeit/scripts/__init__.py @@ -52,6 +52,7 @@ from pypeit.scripts import trace_edges from pypeit.scripts import view_fits from pypeit.scripts import compile_wvarxiv +from pypeit.scripts import show_pixflat # Build the list of script classes diff --git a/pypeit/scripts/identify.py b/pypeit/scripts/identify.py index 64a688242d..f5d123ed62 100644 --- a/pypeit/scripts/identify.py +++ b/pypeit/scripts/identify.py @@ -180,12 +180,14 @@ def main(args): arccen, arc_maskslit = wavecal.extract_arcs(slitIDs=[slit_val]) - # Launch the identify window - # TODO -- REMOVE THIS HACK - try: + # Get the non-linear count level + if msarc.is_mosaic: + # if this is a mosaic we take the maximum value among all the detectors + nonlinear_counts = np.max([rawdets.nonlinear_counts() for rawdets in msarc.detector.detectors]) + else: nonlinear_counts = msarc.detector.nonlinear_counts() - except AttributeError: - nonlinear_counts = None + + # Launch the identify window arcfitter = Identify.initialise(arccen, lamps, slits, slit=int(slit_val), par=par, wv_calib_all=wv_calib_slit, wavelim=[args.wmin, args.wmax], nonlinear_counts=nonlinear_counts, @@ -272,12 +274,15 @@ def main(args): # If we just want the normal one-trace output else: arccen, arc_maskslit = wavecal.extract_arcs(slitIDs=[int(args.slits)]) - # Launch the identify window - # TODO -- REMOVE THIS HACK - try: + + # Get the non-linear count level + if msarc.is_mosaic: + # if this is a mosaic we take the maximum value among all the detectors + nonlinear_counts = np.max([rawdets.nonlinear_counts() for rawdets in msarc.detector.detectors]) + else: nonlinear_counts = msarc.detector.nonlinear_counts() - except AttributeError: - nonlinear_counts = None + + # Launch the identify window arcfitter = Identify.initialise(arccen, lamps, slits, slit=int(args.slits), par=par, wv_calib_all=wv_calib, wavelim=[args.wmin, args.wmax], nonlinear_counts=nonlinear_counts, diff --git a/pypeit/scripts/show_2dspec.py b/pypeit/scripts/show_2dspec.py index 07d9b6ae32..eb701fad1e 100644 --- a/pypeit/scripts/show_2dspec.py +++ b/pypeit/scripts/show_2dspec.py @@ -64,8 +64,11 @@ def show_trace(sobjs, det, viewer, ch): maskdef_extr_list.append(maskdef_extr_flag is True) manual_extr_list.append(manual_extr_flag is True) - display.show_trace(viewer, ch, np.swapaxes(trace_list, 1,0), np.array(trc_name_list), - maskdef_extr=np.array(maskdef_extr_list), manual_extr=np.array(manual_extr_list)) + if len(trace_list) > 0: + display.show_trace(viewer, ch, np.swapaxes(trace_list, 1,0), np.array(trc_name_list), + maskdef_extr=np.array(maskdef_extr_list), manual_extr=np.array(manual_extr_list)) + else: + msgs.warn('spec1d file found, but no objects were extracted for this detector.') class Show2DSpec(scriptbase.ScriptBase): diff --git a/pypeit/scripts/show_pixflat.py b/pypeit/scripts/show_pixflat.py new file mode 100644 index 0000000000..0240837691 --- /dev/null +++ b/pypeit/scripts/show_pixflat.py @@ -0,0 +1,62 @@ +""" +Show on a ginga window the archived pixel flat field image + +.. include common links, assuming primary doc root is up one directory +.. include:: ../include/links.rst +""" + +from pypeit.scripts import scriptbase +from IPython import embed + + +class ShowPixFlat(scriptbase.ScriptBase): + + @classmethod + def get_parser(cls, width=None): + parser = super().get_parser(description='Show an archived Pixel Flat image in a ginga window.', + width=width) + parser.add_argument("file", type=str, help="Pixel Flat filename, e.g. pixelflat_keck_lris_blue.fits.gz") + parser.add_argument('--det', default=None, type=int, nargs='+', + help='Detector(s) to show. If more than one, list the detectors as, e.g. --det 1 2 ' + 'to show detectors 1 and 2. If not provided, all detectors will be shown.') + return parser + + @staticmethod + def main(args): + import numpy as np + from pypeit import msgs + from pypeit import io + from pypeit.display import display + from pypeit import dataPaths + + # check if the file exists + file_path = dataPaths.pixelflat.get_file_path(args.file, return_none=True) + if file_path is None: + msgs.error(f'Provided pixelflat file, {args.file} not found. It is not a direct path, ' + f'a cached file, or a file that can be downloaded from a PypeIt repository.') + + # Load the image + with io.fits_open(file_path) as hdu: + # get all the available detectors in the file + file_dets = [int(h.name.split('-')[0].split('DET')[1]) for h in hdu[1:]] + # if detectors are provided, check if they are in the file + if args.det is not None: + in_file = np.isin(args.det, file_dets) + # if none of the provided detectors are in the file, raise an error + if not np.any(in_file): + msgs.error(f"Provided detector(s) not found in the file. Available detectors are {file_dets}") + # if some of the provided detectors are not in the file, warn the user + elif np.any(np.logical_not(in_file)): + det_not_in_file = np.array(args.det)[np.logical_not(in_file)] + msgs.warn(f"Detector(s) {det_not_in_file} not found in the file. Available detectors are {file_dets}") + + # show the image + display.connect_to_ginga(raise_err=True, allow_new=True) + for h in hdu[1:]: + det = int(h.name.split('-')[0].split('DET')[1]) + if args.det is not None and det not in args.det: + continue + display.show_image(h.data, chname=h.name, cuts=(0.9, 1.1), clear=False, wcs_match=True) + + + diff --git a/pypeit/spectrographs/gemini_gmos.py b/pypeit/spectrographs/gemini_gmos.py index 37616553c7..3c861056aa 100644 --- a/pypeit/spectrographs/gemini_gmos.py +++ b/pypeit/spectrographs/gemini_gmos.py @@ -461,7 +461,7 @@ def get_rawimage(self, raw_file, det): return detectors[0], array[0], hdu, exptime, rawdatasec_img[0], oscansec_img[0] return mosaic, array, hdu, exptime, rawdatasec_img, oscansec_img - def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): + def get_mosaic_par(self, mosaic, hdu=None, msc_ord=0): """ Return the hard-coded parameters needed to construct detector mosaics from unbinned images. @@ -482,7 +482,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): default. BEWARE: If ``hdu`` is not provided, the binning is assumed to be `1,1`, which will cause faults if applied to binned images! - msc_order (:obj:`int`, optional): + msc_ord (:obj:`int`, optional): Order of the interpolation used to construct the mosaic. Returns: @@ -536,7 +536,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): msc_tfm[i] = build_image_mosaic_transform(shape, msc_sft[i], msc_rot[i], tuple(reversed(binning))) return Mosaic(mosaic_id, detectors, shape, np.array(msc_sft), np.array(msc_rot), - np.array(msc_tfm), msc_order) + np.array(msc_tfm), msc_ord) @property def allowed_mosaics(self): @@ -865,7 +865,7 @@ def get_detector_par(self, det, hdu=None): # Return return detector_container.DetectorContainer(**detectors[det-1]) - def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): + def get_mosaic_par(self, mosaic, hdu=None, msc_ord=0): """ Return the hard-coded parameters needed to construct detector mosaics from unbinned images. @@ -886,7 +886,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): default. BEWARE: If ``hdu`` is not provided, the binning is assumed to be `1,1`, which will cause faults if applied to binned images! - msc_order (:obj:`int`, optional): + msc_ord (:obj:`int`, optional): Order of the interpolation used to construct the mosaic. Returns: @@ -908,7 +908,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): else: self.detid = 'BI5-36-4k-2,BI11-33-4k-1,BI12-34-4k-1' - return super().get_mosaic_par(mosaic, hdu=hdu, msc_order=msc_order) + return super().get_mosaic_par(mosaic, hdu=hdu, msc_ord=msc_ord) @classmethod def default_pypeit_par(cls): @@ -1144,7 +1144,7 @@ def get_detector_par(self, det, hdu=None): # Return return detector_container.DetectorContainer(**detectors[det-1]) - def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): + def get_mosaic_par(self, mosaic, hdu=None, msc_ord=0): """ Return the hard-coded parameters needed to construct detector mosaics from unbinned images. @@ -1165,7 +1165,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): default. BEWARE: If ``hdu`` is not provided, the binning is assumed to be `1,1`, which will cause faults if applied to binned images! - msc_order (:obj:`int`, optional): + msc_ord (:obj:`int`, optional): Order of the interpolation used to construct the mosaic. Returns: @@ -1176,7 +1176,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): # Detector ID (it is used to identify the correct mosaic geometry) self.detid = 'BI13-20-4k-1,BI12-09-4k-2,BI13-18-4k-2' - return super().get_mosaic_par(mosaic, hdu=hdu, msc_order=msc_order) + return super().get_mosaic_par(mosaic, hdu=hdu, msc_ord=msc_ord) def config_specific_par(self, scifile, inp_par=None): """ @@ -1412,7 +1412,7 @@ def get_detector_par(self, det, hdu=None): # Return return detector_container.DetectorContainer(**detectors[det-1]) - def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): + def get_mosaic_par(self, mosaic, hdu=None, msc_ord=0): """ Return the hard-coded parameters needed to construct detector mosaics from unbinned images. @@ -1433,7 +1433,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): default. BEWARE: If ``hdu`` is not provided, the binning is assumed to be `1,1`, which will cause faults if applied to binned images! - msc_order (:obj:`int`, optional): + msc_ord (:obj:`int`, optional): Order of the interpolation used to construct the mosaic. Returns: @@ -1445,7 +1445,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): # TODO: Check this is correct self.detid = 'e2v 10031-23-05,10031-01-03,10031-18-04' - return super().get_mosaic_par(mosaic, hdu=hdu, msc_order=msc_order) + return super().get_mosaic_par(mosaic, hdu=hdu, msc_ord=msc_ord) def config_specific_par(self, scifile, inp_par=None): """ diff --git a/pypeit/spectrographs/keck_deimos.py b/pypeit/spectrographs/keck_deimos.py index 250f09af3a..c126b118eb 100644 --- a/pypeit/spectrographs/keck_deimos.py +++ b/pypeit/spectrographs/keck_deimos.py @@ -824,7 +824,7 @@ def get_rawimage(self, raw_file, det): return detectors[0], image[0], hdu, exptime, rawdatasec_img[0], oscansec_img[0] return mosaic, image, hdu, exptime, rawdatasec_img, oscansec_img - def get_mosaic_par(self, mosaic, hdu=None, msc_order=5): + def get_mosaic_par(self, mosaic, hdu=None, msc_ord=5): """ Return the hard-coded parameters needed to construct detector mosaics from unbinned images. @@ -845,7 +845,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=5): default. BEWARE: If ``hdu`` is not provided, the binning is assumed to be `1,1`, which will cause faults if applied to binned images! - msc_order (:obj:`int`, optional): + msc_ord (:obj:`int`, optional): Order of the interpolation used to construct the mosaic. Returns: @@ -892,7 +892,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=5): msc_tfm[i] = build_image_mosaic_transform(shape, msc_sft[i], msc_rot[i], binning) return Mosaic(mosaic_id, detectors, shape, np.array(msc_sft), np.array(msc_rot), - np.array(msc_tfm), msc_order) + np.array(msc_tfm), msc_ord) @property def allowed_mosaics(self): diff --git a/pypeit/spectrographs/keck_hires.py b/pypeit/spectrographs/keck_hires.py index 22c92e639c..c8799462cc 100644 --- a/pypeit/spectrographs/keck_hires.py +++ b/pypeit/spectrographs/keck_hires.py @@ -62,9 +62,11 @@ class KECKHIRESSpectrograph(spectrograph.Spectrograph): ech_fixed_format = False supported = False # TODO before support = True - # 1. Implement flat fielding - # 2. Test on several different setups - # 3. Implement PCA extrapolation into the blue + # 1. Implement flat fielding - DONE + # 2. Test on several different setups - DONE + # 3. Implement PCA extrapolation into the blue + + comment = 'Post detector upgrade (~ August 2004). See :doc:`keck_hires`' # TODO: Place holder parameter set taken from X-shooter VIS for now. @@ -99,6 +101,9 @@ def default_pypeit_par(cls): par['calibrations']['standardframe']['exprng'] = [1, 600] par['scienceframe']['exprng'] = [601, None] + # Set default processing for slitless_pixflat + par['calibrations']['slitless_pixflatframe']['process']['scale_to_mean'] = True + # Slit tracing par['calibrations']['slitedges']['edge_thresh'] = 8.0 par['calibrations']['slitedges']['fit_order'] = 8 @@ -221,22 +226,21 @@ def init_meta(self): # Required (core) self.meta['ra'] = dict(ext=0, card='RA', required_ftypes=['science', 'standard']) self.meta['dec'] = dict(ext=0, card='DEC', required_ftypes=['science', 'standard']) - self.meta['target'] = dict(ext=0, card='OBJECT') + self.meta['target'] = dict(ext=0, card='TARGNAME') self.meta['decker'] = dict(ext=0, card='DECKNAME') self.meta['binning'] = dict(card=None, compound=True) self.meta['mjd'] = dict(card=None, compound=True) # This may depend on the old/new detector self.meta['exptime'] = dict(ext=0, card='ELAPTIME') self.meta['airmass'] = dict(ext=0, card='AIRMASS') - #self.meta['dispname'] = dict(ext=0, card='ECHNAME') + # Extras for config and frametyping self.meta['hatch'] = dict(ext=0, card='HATOPEN') self.meta['dispname'] = dict(ext=0, card='XDISPERS') self.meta['filter1'] = dict(ext=0, card='FIL1NAME') self.meta['echangle'] = dict(ext=0, card='ECHANGL', rtol=1e-3, atol=1e-2) self.meta['xdangle'] = dict(ext=0, card='XDANGL', rtol=1e-2) -# self.meta['idname'] = dict(ext=0, card='IMAGETYP') - # NOTE: This is the native keyword. IMAGETYP is from KOA. + self.meta['object'] = dict(ext=0, card='OBJECT') self.meta['idname'] = dict(card=None, compound=True) self.meta['frameno'] = dict(ext=0, card='FRAMENO') self.meta['instrument'] = dict(ext=0, card='INSTRUME') @@ -277,22 +281,28 @@ def compound_meta(self, headarr, meta_key): return 'off' elif meta_key == 'idname': - if not headarr[0].get('LAMPCAT1') and not headarr[0].get('LAMPCAT2') and \ + xcovopen = headarr[0].get('XCOVOPEN') + collcoveropen = (headarr[0].get('XDISPERS') == 'RED' and headarr[0].get('RCCVOPEN')) or \ + (headarr[0].get('XDISPERS') == 'UV' and headarr[0].get('BCCVOPEN')) + + if xcovopen and collcoveropen and \ + not headarr[0].get('LAMPCAT1') and not headarr[0].get('LAMPCAT2') and \ not headarr[0].get('LAMPQTZ2') and not (headarr[0].get('LAMPNAME') == 'quartz1'): if headarr[0].get('HATOPEN') and headarr[0].get('AUTOSHUT'): return 'Object' elif not headarr[0].get('HATOPEN'): return 'Bias' if not headarr[0].get('AUTOSHUT') else 'Dark' - elif headarr[0].get('AUTOSHUT') and (headarr[0].get('LAMPCAT1') or headarr[0].get('LAMPCAT2')): - if (headarr[0].get('XDISPERS') == 'RED' and not headarr[0].get('RCCVOPEN')) or \ - (headarr[0].get('XDISPERS') == 'UV' and not headarr[0].get('BCCVOPEN')): + elif xcovopen and collcoveropen and \ + headarr[0].get('AUTOSHUT') and (headarr[0].get('LAMPCAT1') or headarr[0].get('LAMPCAT2')): + return 'Line' + elif collcoveropen and \ + headarr[0].get('AUTOSHUT') and \ + (headarr[0].get('LAMPQTZ2') or (headarr[0].get('LAMPNAME') == 'quartz1')) and \ + not headarr[0].get('HATOPEN'): + if not xcovopen: return 'slitlessFlat' else: - return 'Line' - elif headarr[0].get('AUTOSHUT') and \ - (headarr[0].get('LAMPQTZ2') or (headarr[0].get('LAMPNAME') == 'quartz1')) \ - and not headarr[0].get('HATOPEN'): - return 'IntFlat' + return 'IntFlat' else: msgs.error("Not ready for this compound meta") @@ -311,7 +321,26 @@ def configuration_keys(self): and used to constuct the :class:`~pypeit.metadata.PypeItMetaData` object. """ - return ['decker', 'dispname', 'filter1', 'echangle', 'xdangle', 'binning'] + return ['dispname', 'decker', 'filter1', 'echangle', 'xdangle', 'binning'] + + def config_independent_frames(self): + """ + Define frame types that are independent of the fully defined + instrument configuration. + + Bias and dark frames are considered independent of a configuration, + but the DATE-OBS keyword is used to assign each to the most-relevant + configuration frame group. See + :func:`~pypeit.metadata.PypeItMetaData.set_configurations`. + + Returns: + :obj:`dict`: Dictionary where the keys are the frame types that + are configuration independent and the values are the metadata + keywords that can be used to assign the frames to a configuration + group. + """ + return {'bias': ['dispname', 'binning'], 'dark': ['dispname', 'binning'], + 'slitless_pixflat': ['dispname', 'binning']} def raw_header_cards(self): """ @@ -344,9 +373,6 @@ def pypeit_file_keys(self): """ return super().pypeit_file_keys() + ['hatch', 'lampstat01', 'frameno'] - - - def check_frame_type(self, ftype, fitstbl, exprng=None): """ Check for frames of the provided type. @@ -369,14 +395,14 @@ def check_frame_type(self, ftype, fitstbl, exprng=None): good_exp = framematch.check_frame_exptime(fitstbl['exptime'], exprng) # TODO: Allow for 'sky' frame type, for now include sky in # 'science' category - if ftype == 'science': - return good_exp & (fitstbl['idname'] == 'Object') - if ftype == 'standard': + if ftype in ['science', 'standard']: return good_exp & (fitstbl['idname'] == 'Object') if ftype == 'bias': return good_exp & (fitstbl['idname'] == 'Bias') if ftype == 'dark': return good_exp & (fitstbl['idname'] == 'Dark') + if ftype == 'slitless_pixflat': + return good_exp & (fitstbl['idname'] == 'slitlessFlat') if ftype in ['illumflat', 'pixelflat', 'trace']: # Flats and trace frames are typed together return good_exp & (fitstbl['idname'] == 'IntFlat') @@ -387,6 +413,117 @@ def check_frame_type(self, ftype, fitstbl, exprng=None): msgs.warn('Cannot determine if frames are of type {0}.'.format(ftype)) return np.zeros(len(fitstbl), dtype=bool) + def vet_assigned_ftypes(self, type_bits, fitstbl): + """ + + NOTE: this function should only be called when running pypeit_setup, + in order to not overwrite any user-provided frame types. + + This method checks the assigned frame types for consistency. + For frames that are assigned both the science and standard types, + this method chooses the one that is most likely, by checking if the + frames are within 10 arcmin of a listed standard star. + + In addition, for this instrument, if a frame is assigned both a + pixelflat and slitless_pixflat type, the pixelflat type is removed. + NOTE: if the same frame is assigned to multiple configurations, this + method will remove the pixelflat type for all configurations, i.e., + it is not possible to use slitless_pixflat type for one calibration group + and pixelflat for another. + + Args: + type_bits (`numpy.ndarray`_): + Array with the frame types assigned to each frame. + fitstbl (:class:`~pypeit.metadata.PypeItMetaData`): + The class holding the metadata for all the frames. + + Returns: + `numpy.ndarray`_: The updated frame types. + + """ + type_bits = super().vet_assigned_ftypes(type_bits, fitstbl) + + # If both pixelflat and slitless_pixflat are assigned to the same frame, remove pixelflat + + # where slitless_pixflat is assigned + slitless_idx = fitstbl.type_bitmask.flagged(type_bits, flag='slitless_pixflat') + # where pixelflat is assigned + pixelflat_idx = fitstbl.type_bitmask.flagged(type_bits, flag='pixelflat') + + # find configurations where both pixelflat and slitless_pixflat are assigned + pixflat_match = np.zeros(len(fitstbl), dtype=bool) + + for f, frame in enumerate(fitstbl): + if pixelflat_idx[f]: + match_config_values = [] + for slitless in fitstbl[slitless_idx]: + match_config_values.append(np.all([frame[c] == slitless[c] + for c in self.config_independent_frames()['slitless_pixflat']])) + pixflat_match[f] = np.any(match_config_values) + + # remove pixelflat from the type_bits + type_bits[pixflat_match] = fitstbl.type_bitmask.turn_off(type_bits[pixflat_match], 'pixelflat') + + return type_bits + + def parse_raw_files(self, fitstbl, det=1, ftype=None): + """ + Parse the list of raw files with given frame type and detector. + This is spectrograph-specific, and it is not defined for all + spectrographs. + Since different slitless_pixflat frames are usually taken for + each of the three detectors, this method parses the slitless_pixflat + frames and returns the correct one for the requested detector. + + Args: + fitstbl (`astropy.table.Table`_): + Table with metadata of the raw files to parse. + det (:obj:`int`, optional): + 1-indexed detector number to parse. + ftype (:obj:`str`, optional): + Frame type to parse. If None, no frames are parsed + and the indices of all frames are returned. + + Returns: + `numpy.ndarray`_: The indices of the raw files in the fitstbl that are parsed. + + """ + + if ftype == 'slitless_pixflat': + # Check for the required info + if len(fitstbl) == 0: + msgs.warn('Fitstbl provided is emtpy. No parsing done.') + # return empty array + return np.array([], dtype=int) + elif det is None: + msgs.warn('Detector number must be provided to parse slitless_pixflat frames. No parsing done.') + # return index array of length of fitstbl + return np.arange(len(fitstbl)) + + # how many unique xdangle values are there? + # If they are 3, then we have a different slitless flat file per detector + xdangles = np.unique(np.int32(fitstbl['xdangle'].value)) + if len(xdangles) == 3: + sort_xdagles = np.argsort(xdangles) + # xdagles: -5 for red(det=3), -4 for green (det=2), -3 for blue (det=1) dets + # select the corresponding files for the requested detector + if det == 1: + # blue detector + return np.where(np.int32(fitstbl['xdangle'].value) == -3)[0] + elif det == 2: + # green detector + return np.where(np.int32(fitstbl['xdangle'].value) == -4)[0] + elif det == 3: + # red detector + return np.where(np.int32(fitstbl['xdangle'].value) == -5)[0] + else: + msgs.warn('The provided list of slitless_pixflat frames does not have exactly 3 unique XDANGLE values. ' + 'Pypeit cannot determine which slitless_pixflat frame corresponds to the requested detector. ' + 'All frames will be used.') + return np.arange(len(fitstbl)) + + else: + return super().parse_raw_files(fitstbl, det=det, ftype=ftype) def get_rawimage(self, raw_file, det, spectrim=20): """ @@ -502,7 +639,7 @@ def get_rawimage(self, raw_file, det, spectrim=20): return mosaic, image, hdu, exptime, rawdatasec_img, oscansec_img - def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): + def get_mosaic_par(self, mosaic, hdu=None, msc_ord=0): """ Return the hard-coded parameters needed to construct detector mosaics from unbinned images. @@ -523,7 +660,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): default. BEWARE: If ``hdu`` is not provided, the binning is assumed to be `1,1`, which will cause faults if applied to binned images! - msc_order (:obj:`int`, optional): + msc_ord (:obj:`int`, optional): Order of the interpolation used to construct the mosaic. Returns: @@ -574,7 +711,7 @@ def get_mosaic_par(self, mosaic, hdu=None, msc_order=0): msc_tfm[i] = build_image_mosaic_transform(shape, msc_sft[i], msc_rot[i], tuple(reversed(binning))) return Mosaic(mosaic_id, detectors, shape, np.array(msc_sft), np.array(msc_rot), - np.array(msc_tfm), msc_order) + np.array(msc_tfm), msc_ord) @property diff --git a/pypeit/spectrographs/keck_lris.py b/pypeit/spectrographs/keck_lris.py index ffc42afb87..8c807d052a 100644 --- a/pypeit/spectrographs/keck_lris.py +++ b/pypeit/spectrographs/keck_lris.py @@ -22,6 +22,7 @@ from pypeit import io from pypeit.core import parse from pypeit.core import framematch +from pypeit.core import flux_calib from pypeit.spectrographs import spectrograph from pypeit.spectrographs import slitmask from pypeit.images import detector_container @@ -80,14 +81,21 @@ def default_pypeit_par(cls): par['calibrations']['wavelengths']['n_first'] = 3 par['calibrations']['wavelengths']['n_final'] = 5 # Set the default exposure time ranges for the frame typing - par['calibrations']['biasframe']['exprng'] = [None, 0.001] + par['calibrations']['biasframe']['exprng'] = [None, 1] par['calibrations']['darkframe']['exprng'] = [999999, None] # No dark frames par['calibrations']['pinholeframe']['exprng'] = [999999, None] # No pinhole frames - par['calibrations']['pixelflatframe']['exprng'] = [None, 60] - par['calibrations']['traceframe']['exprng'] = [None, 60] - par['calibrations']['illumflatframe']['exprng'] = [None, 60] + par['calibrations']['pixelflatframe']['exprng'] = [0, 60] + par['calibrations']['traceframe']['exprng'] = [0, 60] + par['calibrations']['illumflatframe']['exprng'] = [0, 60] + par['calibrations']['slitless_pixflatframe']['exprng'] = [0, 60] par['calibrations']['standardframe']['exprng'] = [1, 61] + # Set default processing for slitless_pixflat + par['calibrations']['slitless_pixflatframe']['process']['scale_to_mean'] = True + + # Turn off flat illumination fine correction (it's more often bad than good) + par['calibrations']['flatfield']['slit_illum_finecorr'] = False + # Flexure # Always correct for spectral flexure, starting with default parameters par['flexure']['spec_method'] = 'boxcar' @@ -153,8 +161,8 @@ def init_meta(self): """ self.meta = {} # Required (core) - self.meta['ra'] = dict(ext=0, card='RA') - self.meta['dec'] = dict(ext=0, card='DEC') + self.meta['ra'] = dict(card=None, compound=True) + self.meta['dec'] = dict(card=None, compound=True) self.meta['target'] = dict(ext=0, card='TARGNAME') self.meta['decker'] = dict(ext=0, card='SLITNAME') self.meta['binning'] = dict(card=None, compound=True) @@ -174,6 +182,7 @@ def init_meta(self): # Extras for pypeit file self.meta['dateobs'] = dict(card=None, compound=True) self.meta['amp'] = dict(ext=0, card='NUMAMPS') + self.meta['object'] = dict(ext=0, card='OBJECT') # Lamps # similar approach to DEIMOS @@ -193,7 +202,20 @@ def compound_meta(self, headarr, meta_key): Returns: object: Metadata value read from the header(s). """ - if meta_key == 'binning': + # LRIS sometime misses RA and/or Dec in the header. When this happens, set them to 0 + if meta_key == 'ra': + if headarr[0].get('RA') is None: + msgs.warn('Keyword RA not found in header. Setting to 0') + return '00:00:00.00' + else: + return headarr[0]['RA'] + elif meta_key == 'dec': + if headarr[0].get('DEC') is None: + msgs.warn('Keyword DEC not found in header. Setting to 0') + return '+00:00:00.0' + else: + return headarr[0]['DEC'] + elif meta_key == 'binning': binspatial, binspec = parse.parse_binning(headarr[0]['BINNING']) binning = parse.binning2string(binspec, binspatial) return binning @@ -292,7 +314,8 @@ def config_independent_frames(self): keywords that can be used to assign the frames to a configuration group. """ - return {'bias': ['amp', 'binning', 'dateobs'], 'dark': ['amp', 'binning', 'dateobs']} + return {'bias': ['amp', 'binning', 'dateobs'], 'dark': ['amp', 'binning', 'dateobs'], + 'slitless_pixflat': ['amp', 'binning', 'dateobs', 'dispname', 'dichroic']} def pypeit_file_keys(self): """ @@ -324,20 +347,49 @@ def check_frame_type(self, ftype, fitstbl, exprng=None): `numpy.ndarray`_: Boolean array with the flags selecting the exposures in ``fitstbl`` that are ``ftype`` type frames. """ - good_exp = framematch.check_frame_exptime(fitstbl['exptime'], exprng) + # good exposures + good_exp = framematch.check_frame_exptime(fitstbl['exptime'], exprng) & (fitstbl['decker'] != 'GOH_LRIS') + # no images no_img = np.array([d not in ['Mirror', 'mirror', 'clear'] for d in fitstbl['dispname']]) + + # Check frame type if ftype == 'science': return good_exp & self.lamps(fitstbl, 'off') & (fitstbl['hatch'] == 'open') & no_img if ftype == 'standard': - return good_exp & self.lamps(fitstbl, 'off') & (fitstbl['hatch'] == 'open') & no_img + std = np.zeros(len(fitstbl), dtype=bool) + if 'ra' in fitstbl.keys() and 'dec' in fitstbl.keys(): + std = np.array([flux_calib.find_standard_file(ra, dec, toler=10.*units.arcmin, check=True) + for ra, dec in zip(fitstbl['ra'], fitstbl['dec'])]) + return good_exp & self.lamps(fitstbl, 'off') & (fitstbl['hatch'] == 'open') & no_img & std if ftype == 'bias': return good_exp & self.lamps(fitstbl, 'off') & (fitstbl['hatch'] == 'closed') + if ftype == 'slitless_pixflat': + # these are sky flats, like science but without the slitmask + return (good_exp & self.lamps(fitstbl, 'off') & + (fitstbl['hatch'] == 'open') & no_img & (fitstbl['decker'] == 'direct')) if ftype in ['pixelflat', 'trace', 'illumflat']: # Allow for dome or internal good_dome = self.lamps(fitstbl, 'dome') & (fitstbl['hatch'] == 'open') good_internal = self.lamps(fitstbl, 'internal') & (fitstbl['hatch'] == 'closed') + # attempt at identifying sky flats (not robust, but better than nothing) + # they are basically science frames, so we look for "sky" words in the header + is_sky = self.lamps(fitstbl, 'off') & (fitstbl['hatch'] == 'open') + # look for specific words in the target or object header keywords + words_to_search = ['sky', 'blank', 'twilight', 'twiflat', 'twi flat'] + for i, row in enumerate(fitstbl): + in_target = False + if row['target'] is not None: + if np.any([w in row['target'].lower() for w in words_to_search]): + in_target = True + in_object = False + if row['object'] is not None: + if np.any([w in row['object'].lower() for w in words_to_search]): + in_object = True + is_sky[i] = in_target or in_object + # put together the sky flats requirement + sky_flat = is_sky & (fitstbl['decker'] != 'direct') # Flats and trace frames are typed together - return good_exp & (good_dome + good_internal) & no_img + return good_exp & (good_dome + good_internal + sky_flat) & no_img if ftype in ['pinhole', 'dark']: # Don't type pinhole or dark frames return np.zeros(len(fitstbl), dtype=bool) @@ -346,6 +398,59 @@ def check_frame_type(self, ftype, fitstbl, exprng=None): msgs.warn('Cannot determine if frames are of type {0}.'.format(ftype)) return np.zeros(len(fitstbl), dtype=bool) + + def vet_assigned_ftypes(self, type_bits, fitstbl): + """ + + NOTE: this function should only be called when running pypeit_setup, + in order to not overwrite any user-provided frame types. + + This method checks the assigned frame types for consistency. + For frames that are assigned both the science and standard types, + this method chooses the one that is most likely, by checking if the + frames are within 10 arcmin of a listed standard star. + + In addition, for this instrument, if a frame is assigned both a + pixelflat and slitless_pixflat type, the pixelflat type is removed. + NOTE: if the same frame is assigned to multiple configurations, this + method will remove the pixelflat type for all configurations, i.e., + it is not possible to use slitless_pixflat type for one calibration group + and pixelflat for another. + + Args: + type_bits (`numpy.ndarray`_): + Array with the frame types assigned to each frame. + fitstbl (:class:`~pypeit.metadata.PypeItMetaData`): + The class holding the metadata for all the frames. + + Returns: + `numpy.ndarray`_: The updated frame types. + + """ + type_bits = super().vet_assigned_ftypes(type_bits, fitstbl) + + # If both pixelflat and slitless_pixflat are assigned to the same frame, remove pixelflat + + # where slitless_pixflat is assigned + slitless_idx = fitstbl.type_bitmask.flagged(type_bits, flag='slitless_pixflat') + # where pixelflat is assigned + pixelflat_idx = fitstbl.type_bitmask.flagged(type_bits, flag='pixelflat') + + # find configurations where both pixelflat and slitless_pixflat are assigned + pixflat_match = np.zeros(len(fitstbl), dtype=bool) + + for f, frame in enumerate(fitstbl): + if pixelflat_idx[f]: + match_config_values = [] + for slitless in fitstbl[slitless_idx]: + match_config_values.append(np.all([frame[c] == slitless[c] + for c in self.config_independent_frames()['slitless_pixflat']])) + pixflat_match[f] = np.any(match_config_values) + + # remove pixelflat from the type_bits + type_bits[pixflat_match] = fitstbl.type_bitmask.turn_off(type_bits[pixflat_match], 'pixelflat') + + return type_bits def lamps(self, fitstbl, status): """ @@ -853,6 +958,8 @@ def default_pypeit_par(cls): par['calibrations']['traceframe']['exprng'] = [None, 300] par['calibrations']['illumflatframe']['exprng'] = [None, 300] + par['calibrations']['standardframe']['exprng'] = [1, 901] + return par def config_specific_par(self, scifile, inp_par=None): @@ -1927,38 +2034,48 @@ def lris_read_amp(inp, ext): return data, predata, postdata, x1, y1 -def convert_lowredux_pixelflat(infil, outfil): +def convert_lowredux_pixelflat(infil, outfil, specflip=False, separate_extensions=False): """ Convert LowRedux pixelflat to PYPIT format Returns ------- """ # Read - hdu = io.fits_open(infil) - data = hdu[0].data + hdu0 = io.fits_open(infil) + data = hdu0[0].data # prihdu = fits.PrimaryHDU() hdus = [prihdu] - prihdu.header['FRAMETYP'] = 'pixelflat' + prihdu.header['CALIBTYP'] = ('Flat', 'PypeIt: Calibration frame type') # Detector 1 - img1 = data[:,:data.shape[1]//2] + if separate_extensions: + img1 = hdu0['DET1'].data + else: + img1 = data[:, :data.shape[1] // 2] + if specflip: + img1 = np.flip(img1, axis=0) hdu = fits.ImageHDU(img1) - hdu.name = 'DET1' - prihdu.header['EXT0001'] = 'DET1-pixelflat' + hdu.name = 'DET01-PIXELFLAT_NORM' + prihdu.header['EXT0001'] = hdu.name hdus.append(hdu) # Detector 2 - img2 = data[:,data.shape[1]//2:] + if separate_extensions: + img2 = hdu0['DET2'].data + else: + img2 = data[:, data.shape[1] // 2:] + if specflip: + img2 = np.flip(img2, axis=0) hdu = fits.ImageHDU(img2) - hdu.name = 'DET2' - prihdu.header['EXT0002'] = 'DET2-pixelflat' + hdu.name = 'DET02-PIXELFLAT_NORM' + prihdu.header['EXT0002'] = hdu.name hdus.append(hdu) # Finish hdulist = fits.HDUList(hdus) - hdulist.writeto(outfil, clobber=True) + hdulist.writeto(outfil, overwrite=True) print('Wrote {:s}'.format(outfil)) diff --git a/pypeit/spectrographs/keck_mosfire.py b/pypeit/spectrographs/keck_mosfire.py index e874baea83..44ffc91f1d 100644 --- a/pypeit/spectrographs/keck_mosfire.py +++ b/pypeit/spectrographs/keck_mosfire.py @@ -335,7 +335,7 @@ def compound_meta(self, headarr, meta_key): FLATSPEC = headarr[0].get('FLATSPEC') PWSTATA7 = headarr[0].get('PWSTATA7') PWSTATA8 = headarr[0].get('PWSTATA8') - if FLATSPEC == 0 and PWSTATA7 == 0 and PWSTATA8 == 0: + if FLATSPEC == 0 and PWSTATA7 == 0 and PWSTATA8 == 0 and headarr[0].get('FILTER') != 'Dark': if 'Flat' in headarr[0].get('OBJECT'): return 'flatlampoff' else: diff --git a/pypeit/spectrographs/spectrograph.py b/pypeit/spectrographs/spectrograph.py index 24d38b22c9..46779d8f8d 100644 --- a/pypeit/spectrographs/spectrograph.py +++ b/pypeit/spectrographs/spectrograph.py @@ -33,12 +33,14 @@ import numpy as np from astropy.io import fits +from astropy import units from pypeit import msgs from pypeit import io from pypeit.core import parse from pypeit.core import procimg from pypeit.core import meta +from pypeit.core import flux_calib from pypeit.par import pypeitpar from pypeit.images.detector_container import DetectorContainer from pypeit.images.mosaic import Mosaic @@ -665,6 +667,28 @@ def list_detectors(self, mosaic=False): dets = self.allowed_mosaics if mosaic else range(1,self.ndet+1) return np.array([self.get_det_name(det) for det in dets]) + def parse_raw_files(self, fitstbl, det=1, ftype=None): + """ + Parse the list of raw files with given frame type and detector. + This is spectrograph-specific, and it is not defined for all + spectrographs. Therefore, this generic method + returns the indices of all the files in the input table. + + Args: + fitstbl (`astropy.table.Table`_): + Table with metadata of the raw files to parse. + det (:obj:`int`, optional): + 1-indexed detector number to parse. + ftype (:obj:`str`, optional): + Frame type to parse. If None, no frames are parsed + and the indices of all frames are returned. + + Returns: + `numpy.ndarray`_: The indices of the raw files in the fitstbl that are parsed. + + """ + + return np.arange(len(fitstbl)) def get_lamps(self, fitstbl): """ @@ -1634,6 +1658,73 @@ def check_frame_type(self, ftype, fitstbl, exprng=None): """ raise NotImplementedError('Frame typing not defined for {0}.'.format(self.name)) + def vet_assigned_ftypes(self, type_bits, fitstbl): + """ + NOTE: this function should only be called when running pypeit_setup, + in order to not overwrite any user-provided frame types. + + This method checks the assigned frame types for consistency. + For frames that are assigned both the science and standard types, + this method chooses the one that is most likely, by checking if the + frames are within 10 arcmin of a listed standard star. + + In addition, this method can perform other checks on the assigned frame types + that are spectrograph-specific. + + Args: + type_bits (`numpy.ndarray`_): + Array with the frame types assigned to each frame. + fitstbl (:class:`~pypeit.metadata.PypeItMetaData`): + The class holding the metadata for all the frames. + + Returns: + `numpy.ndarray`_: The updated frame types. + + """ + # For frames that are assigned both science and standard types, choose the one that is most likely + # find frames that are assigned both science and standard star types + indx = fitstbl.type_bitmask.flagged(type_bits, flag='standard') & \ + fitstbl.type_bitmask.flagged(type_bits, flag='science') + if np.any(indx): + msgs.warn('Some frames are assigned both science and standard types. Choosing the most likely type.') + if 'ra' not in fitstbl.keys() or 'dec' not in fitstbl.keys(): + msgs.warn('Sky coordinates are not available. Standard stars cannot be identified.') + # turn off the standard flag for all frames + type_bits[indx] = fitstbl.type_bitmask.turn_off(type_bits[indx], flag='standard') + return type_bits + # check if any coordinates are None + none_coords = indx & ((fitstbl['ra'] == 'None') | (fitstbl['dec'] == 'None') | + np.isnan(fitstbl['ra']) | np.isnan(fitstbl['dec'])) + if np.any(none_coords): + msgs.warn('The following frames have None coordinates. ' + 'They could be a twilight flat frame that was missed by the automatic identification') + [msgs.prindent(f) for f in fitstbl['filename'][none_coords]] + # turn off the standard star flag for these frames + type_bits[none_coords] = fitstbl.type_bitmask.turn_off(type_bits[none_coords], flag='standard') + + # If the frame is within 10 arcmin of a listed standard star, then it is probably a standard star + # Find the nearest standard star to each frame that is assigned both science and standard types + # deal with possible None coordinates + is_std = np.array([], dtype=bool) + for ra, dec in zip(fitstbl['ra'], fitstbl['dec']): + if ra == 'None' or dec == 'None' or np.isnan(ra) or np.isnan(dec): + is_std = np.append(is_std, False) + else: + is_std = np.append(is_std, flux_calib.find_standard_file(ra, dec, toler=10.*units.arcmin, check=True)) + + foundstd = indx & is_std + # turn off the science flag for frames that are found to be standard stars and + # turn off the standard flag for frames that are not + if np.any(foundstd): + type_bits[foundstd] = fitstbl.type_bitmask.turn_off(type_bits[foundstd], flag='science') + type_bits[np.logical_not(foundstd)] = \ + fitstbl.type_bitmask.turn_off(type_bits[np.logical_not(foundstd)], flag='standard') + else: + # if no standard stars are found, turn off the standard flag for all frames + type_bits[indx] = fitstbl.type_bitmask.turn_off(type_bits[indx], flag='standard') + + return type_bits + def idname(self, ftype): """ Return the ``idname`` for the selected frame type for this diff --git a/pypeit/wavecalib.py b/pypeit/wavecalib.py index 4e576ee53f..ca223a39e2 100644 --- a/pypeit/wavecalib.py +++ b/pypeit/wavecalib.py @@ -526,11 +526,11 @@ def __init__(self, msarc, slits, spectrograph, par, lamps, self.arccen = None # central arc spectrum # Get the non-linear count level - # TODO: This is currently hacked to deal with Mosaics - try: + if self.msarc.is_mosaic: + # if this is a mosaic we take the maximum value among all the detectors + self.nonlinear_counts = np.max([rawdets.nonlinear_counts() for rawdets in self.msarc.detector.detectors]) + else: self.nonlinear_counts = self.msarc.detector.nonlinear_counts() - except: - self.nonlinear_counts = 1e10 # -------------------------------------------------------------- # TODO: Build another base class that does these things for both diff --git a/pypeit/wavetilts.py b/pypeit/wavetilts.py index cf9977744a..2b24d0005f 100644 --- a/pypeit/wavetilts.py +++ b/pypeit/wavetilts.py @@ -318,11 +318,11 @@ def __init__(self, mstilt, slits, spectrograph, par, wavepar, det=1, qa_path=Non self.spat_flexure = spat_flexure # Get the non-linear count level - # TODO: This is currently hacked to deal with Mosaics - try: + if self.mstilt.is_mosaic: + # if this is a mosaic we take the maximum value among all the detectors + self.nonlinear_counts = np.max([rawdets.nonlinear_counts() for rawdets in self.mstilt.detector.detectors]) + else: self.nonlinear_counts = self.mstilt.detector.nonlinear_counts() - except: - self.nonlinear_counts = 1e10 # Set the slitmask and slit boundary related attributes that the # code needs for execution. This also deals with arcimages that diff --git a/setup.cfg b/setup.cfg index e707ae37ac..c910a2c648 100644 --- a/setup.cfg +++ b/setup.cfg @@ -170,6 +170,7 @@ console_scripts = pypeit_skysub_regions = pypeit.scripts.skysub_regions:SkySubRegions.entry_point pypeit_view_fits = pypeit.scripts.view_fits:ViewFits.entry_point pypeit_setup_gui = pypeit.scripts.setup_gui:SetupGUI.entry_point + pypeit_show_pixflat = pypeit.scripts.show_pixflat:ShowPixFlat.entry_point ginga.rv.plugins = SlitWavelength = pypeit.display:setup_SlitWavelength