sdss · kbwestfall · Feb 24, 2022 · Feb 24, 2022 · Feb 24, 2022 · Feb 24, 2022
diff --git a/.github/workflows/ci_tests.yml b/.github/workflows/ci_tests.yml
@@ -37,10 +37,10 @@ jobs:
     - name: Upload coverage to codecov
       if: "contains(matrix.toxenv, '-cov')"
       uses: codecov/codecov-action@v1
-      with:
-        token: ${{ secrets.CODECOV }}
-        file: ./coverage.xml
-        fail_ci_if_error: true
+#      with:
+#        token: ${{ secrets.CODECOV }}
+#        file: ./coverage.xml
+#        fail_ci_if_error: false
 
   os-tests:
     name: Python ${{ matrix.python }} on ${{ matrix.os }}

diff --git a/CHANGES.md b/CHANGES.md
@@ -1,6 +1,27 @@
 
-3.1.3dev
---------
+4.0.3 (29 Apr 2022)
+-------------------
+
+ - Minor change to `manga_dap_inspector` to change initial spectral plotting
+   limits to the minimum and maximum of the whole cube instead of just the
+   center spaxel.
+ - Deprecated most of the configuration files and directories to the `outofdate/`
+   directory.
+ - Fixed survey-level execution script `rundap` and the post-processing scripts
+   used to create the DAPall file and QA plots.
+
+4.0.2 (1 Apr 2022)
+------------------
+
+ - tomli dependency hotfix for pip distribution
+
+4.0.1 (1 Apr 2022)
+------------------
+
+ - Package name hot fix for pip distribution
+
+4.0.0 (1 Apr 2022)
+------------------
 
  - Update to custom datacube class implementation.  Previous version
    required a file within the mangadap/ directory structure; new code
@@ -14,14 +35,33 @@
     - Change scripting protocol
     - tox testing CI    
     - Updated MANIFEST
-
+- Major overhaul of how paths and output files are defined, needed to complete
+  abstraction and move away from MaNGA-specific code.
+    - Input paths and file names now defined as part of the DataCube subclass.
+      For MaNGA, this is done by subclassing the `MaNGADataCube` from the
+      `mangadap.config.manga.MaNGAConfig` class.  For user-defined DataCube
+      subclasses, these paths can be defined directly.
+    - Output paths for primary output files now defined by the `AnalysisPlanSet`
+      object or a relevant subclass.  For MaNGA, a subclass is necessary to put
+      all the files in a subdirectory using the plate and ifu, but likely most
+      other implementations can use `AnalysisPlanSet` directly.  Reference file
+      names adjusted and defined by each main analysis class.
+ - Remove all instances of `numpy.float`, `numpy.int`, `numpy.bool`, given
+   impending numpy deprecation of these types.
+ - Added documentation for how to fit one spectrum and to fit a (non-MaNGA)
+   datacube.
+ - Remove "filtered" fitting from pPXF module
+ - Major refactor of how module parameters are input and changed.
+    - Input analysis plan is now provided by a toml file.
+    - Module parameter definition classes all now have defaults.  Altering any
+      of those defaults is done via the plan toml file.
 
 3.1.2 (29 Jul 2021)
 -------------------
 
  - Additional updates to documentation for DR17 release.
- - Includes includes of MaNGA IDs used in the construction of the MaStar
-   HC version 2 library.
+ - Includes MaNGA IDs used in the construction of the MaStar HC version 2
+   library.
 
 
 3.1.1 (1 Mar 2021)

diff --git a/README.md b/README.md
@@ -1,101 +1,57 @@
 # The MaNGA Data Analysis Pipeline
 
-[![Build Status](https://travis-ci.org/sdss/mangadap.svg?branch=master)](https://travis-ci.org/sdss/mangadap)
-[![Coverage Status](https://coveralls.io/repos/github/sdss/mangadap/badge.svg?branch=master)](https://coveralls.io/github/sdss/mangadap?branch=master)
+[![Build Status](https://github.com/sdss/mangadap/actions/workflows/ci_tests.yml/badge.svg)](https://github.com/sdss/mangadap/actions)
+[![Coverage Status](https://codecov.io/gh/sdss/mangadap/branch/master/graph/badge.svg?token=S4veEPJwS1)](https://codecov.io/gh/sdss/mangadap)
 [![Doc Status](https://readthedocs.org/projects/sdss-mangadap/badge/?version=latest)](https://sdss-mangadap.readthedocs.io/en/latest/)
-[![astropy](http://img.shields.io/badge/powered%20by-AstroPy-orange.svg?style=flat)](http://www.astropy.org/)
 [![License](https://img.shields.io/github/license/sdss/mangadap)](https://github.com/sdss/mangadap/blob/master/LICENSE.md)
+[![astropy](http://img.shields.io/badge/powered%20by-AstroPy-orange.svg?style=flat)](http://www.astropy.org/)
 
-The MaNGA data-analysis pipeline (MaNGA DAP) is the survey-led software
-package that analyzes the data produced by the MaNGA data-reduction
-pipeline (MaNGA DRP) to produced physical properties derived from the
-MaNGA spectroscopy.
-
-For full documentation, see: https://sdss-mangadap.readthedocs.io/en/latest/
-
-**Note that the version of the DAP used for DR15 is [2.2.1](https://github.com/sdss/mangadap/releases/tag/2.2.1).**
-
-## Citation
-
-If you use the DAP software and/or its output products, please cite the following two papers:
-
- - *Overview*: [Westfall et al. (2019, AJ, 158, 231)](https://ui.adsabs.harvard.edu/abs/2019AJ....158..231W/abstract)
- - *Emission-line Modeling*: [Belfiore et al. (2019, AJ, 158, 160)](https://ui.adsabs.harvard.edu/abs/2019AJ....158..160B/abstract)
-
-## Installation
-
-To install, first clone the repository:
-
-`git clone https://github.com/sdss/mangadap.git`
-
-We recommend using the most recent tag:
+The MaNGA data-analysis pipeline (MaNGA DAP) is the survey-led software package
+that has analyzed all galaxy data produced by the MaNGA data-reduction pipeline
+(MaNGA DRP).  Its goal is to produce high-level, science-ready data products
+derived from MaNGA spectra.  The products currently provided are:
 
-```
-cd mangadap
-./checkout_current_tag
-```
+ - Spatially stacked spectra
+ - Stellar kinematics
+ - Nebular emission-line properties: fluxes, equivalent widths, and
+   kinematics
+ - Spectral indices: absorption-line (e.g., H-delta) and bandhead/color
+   (e.g., TiO, D4000) measurements
 
-----
+That is, the DAP currently focuses on "model-independent" properties.
+Higher-level, model-dependent properties, such as stellar-population parameters,
+are outside of the scope of the current pipeline.
 
-To install, run:
+**See our full documentation at [sdss-mangadap.readthedocs.io](https://sdss-mangadap.readthedocs.io/en/latest/).**
 
-`python3 setup.py install`
 
-On MacOSX, you may need to add `CC=clang`, i.e.:
-
-`CC=clang python3 setup.py install`
+## SDSS Data Release Versions
 
-----
+ - [SDSS-IV/MaNGA DR15](https://www.sdss.org/dr15/manga/) is based on version 2.2.1
 
-The DAP uses environmental variable to define the paths to specific data
-and other repositories.  If these are not defined, warnings will be
-issued everytime the DAP is installed or imported.  The relevant
-environmental variables, their default, and their usage are provided
-below.
+ - [SDSS-IV/MaNGA DR17](https://www.sdss.org/dr17/manga/) is based on version
+   3.1.2.  Note that the version of the code that produced the DR17 was version
+   3.1.0; version 3.1.1 and 3.1.2 only include documentation updates.
 
-|                 Variable |                           Default |                                       Comments |
-|:------------------------ |:--------------------------------- |:---------------------------------------------- |
-| `MANGADRP_VER`           | `v2_4_3`                          | Version of the DRP, used for path construction |
-| `MANGA_SPECTRO_REDUX`    | `$HOME/MaNGA/redux`               | Root path for the reduced data                 |
-| `MANGADAP_VER`           | `mangadap.__version__`            | Version of the DAP, used for path construction |
-| `MANGA_SPECTRO_ANALYSIS` | `$HOME/MaNGA/analysis`            | Root path for the analysis data                |
-| `MANGACORE_VER`          | `v1_6_2`                          | Version of MaNGA core (survey-level meta data) |
-| `MANGACORE_DIR`          | `$HOME/MaNGA/core/$MANGACORE_VER` | Root path with the MaNGA core repository       |
 
-**Notes:**
- - `$MANGACORE_VER` and `$MANGACORE_DIR` are only needed to perform the
-   survey-level execution of the pipeline.
- - The DAP expects to find the DRP `LOGCUBE` files in
-   `$MANGA_SPECTRO_REDUX/$MANGADRP_VER/[plate]/stack`, where `[plate]`
-   is the desired plate number.
- - The DAP expects to find/write data to
-   `$MANGA_SPECTRO_ANALYSIS/$MANGADRP_VER/$MANGADAP_VER`.
- - `$MANGADAP_VER` is only used to set the path names, not to select the
-   specific version of the pipeline to use
+## Citation
 
-These environmental variables can be added to your `.bash_profile` file
-in your home directory or be included in a script that is sourced when
-you want to run the DAP.  The added lines to your `.bash_profile` file
-could look something like this:
+If you use the DAP software and/or its output products, please cite the
+following two papers:
 
-```
-export MANGA_SPECTRO_REDUX=/Volumes/MaNGA/redux
-export MANGA_SPECTRO_ANALYSIS=/Volumes/MaNGA/analysis
+ - *Overview*: [Westfall et al. (2019, AJ, 158, 231)](https://ui.adsabs.harvard.edu/abs/2019AJ....158..231W/abstract)
+ - *Emission-line Modeling*: [Belfiore et al. (2019, AJ, 158, 160)](https://ui.adsabs.harvard.edu/abs/2019AJ....158..160B/abstract)
 
-export MANGADRP_VER=v2_4_3
+Additionally, if you use SDSS-IV/MaNGA data, please see:
 
-export MANGADAP_VER=2.2.1
+ - [How to Cite SDSS](https://www.sdss.org/collaboration/citing-sdss/)
+ - [SDSS Technical Publications](https://www.sdss.org/science/technical_publications/)
 
-export MANGACORE_VER=v1_6_2
-export MANGACORE_DIR=$HOME/MaNGA/core/$MANGACORE_VER
-```
 
-**Note**: Some of these variables are also defined by Marvin; see
-[here](https://sdss-marvin.readthedocs.io/en/stable/installation.html).
-It's possible to have both Marvin and the DAP point to the same
-directory, but beware that this may mean that some of the files get
-overwritten.
+## Installation
 
+`pip install sdss-mangadap`
 
+See https://sdss-mangadap.readthedocs.io/en/latest/installation.html
 
 
diff --git a/bin/rundap b/bin/rundap
diff --git a/codecov.yml b/codecov.yml
@@ -0,0 +1,12 @@
+comment:
+  after_n_builds: 6
+
+coverage:
+  status:
+    project:
+      default: true
+         # disable the default status that measures entire project
+    patch:
+      default:
+        target: 30%   # new contributions should have a coverage at least equal to target
+
diff --git a/docs/Makefile b/docs/Makefile
@@ -22,7 +22,6 @@ apirst:
 	python3 ./scripts/write_bitmask_tables.py
 	python3 ./scripts/write_parset_tables.py
 	python3 ./scripts/write_datatables.py
-	python3 ./scripts/write_defaultplan.py
 	python3 ./scripts/write_script_help.py
 
 html:
@@ -31,3 +30,8 @@ html:
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
 
+htmlonly:
+	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
+	@echo
+	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."
+
diff --git a/docs/api/mangadap.config.analysisplan.rst b/docs/api/mangadap.config.analysisplan.rst
@@ -0,0 +1,8 @@
+mangadap.config.analysisplan module
+===================================
+
+.. automodule:: mangadap.config.analysisplan
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/api/mangadap.util.drpfits.rst → docs/api/mangadap.config.manga.rst b/docs/api/mangadap.util.drpfits.rst → docs/api/mangadap.config.manga.rst
@@ -1,7 +1,7 @@
-mangadap.util.drpfits module
+mangadap.config.manga module
 ============================
 
-.. automodule:: mangadap.util.drpfits
+.. automodule:: mangadap.config.manga
    :members:
    :private-members:
    :undoc-members:

diff --git a/docs/api/mangadap.config.rst b/docs/api/mangadap.config.rst
@@ -7,7 +7,9 @@ Submodules
 .. toctree::
    :maxdepth: 4
 
+   mangadap.config.analysisplan
    mangadap.config.defaults
+   mangadap.config.manga
 
 Module contents
 ---------------

diff --git a/docs/api/mangadap.par.analysisplan.rst b/docs/api/mangadap.par.analysisplan.rst
diff --git a/docs/api/mangadap.par.rst b/docs/api/mangadap.par.rst
@@ -8,7 +8,6 @@ Submodules
    :maxdepth: 4
 
    mangadap.par.absorptionindexdb
-   mangadap.par.analysisplan
    mangadap.par.artifactdb
    mangadap.par.bandheadindexdb
    mangadap.par.emissionlinedb

diff --git a/docs/api/mangadap.rst b/docs/api/mangadap.rst
@@ -23,6 +23,7 @@ Submodules
    :maxdepth: 4
 
    mangadap.dapfits
+   mangadap.version
 
 Module contents
 ---------------

diff --git a/docs/api/mangadap.survey.rst b/docs/api/mangadap.survey.rst
@@ -10,8 +10,7 @@ Submodules
    mangadap.survey.dapall
    mangadap.survey.drpcomplete
    mangadap.survey.manga_dap
-   mangadap.survey.mangampl
-   mangadap.survey.util
+   mangadap.survey.rundap
 
 Module contents
 ---------------

diff --git a/docs/api/mangadap.survey.rundap.rst b/docs/api/mangadap.survey.rundap.rst
@@ -0,0 +1,8 @@
+mangadap.survey.rundap module
+=============================
+
+.. automodule:: mangadap.survey.rundap
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/api/mangadap.survey.util.rst b/docs/api/mangadap.survey.util.rst
diff --git a/docs/api/mangadap.survey.mangampl.rst → docs/api/mangadap.util.drpbitmask.rst b/docs/api/mangadap.survey.mangampl.rst → docs/api/mangadap.util.drpbitmask.rst
@@ -1,7 +1,7 @@
-mangadap.survey.mangampl module
+mangadap.util.drpbitmask module
 ===============================
 
-.. automodule:: mangadap.survey.mangampl
+.. automodule:: mangadap.util.drpbitmask
    :members:
    :private-members:
    :undoc-members:

diff --git a/docs/api/mangadap.util.exception_tools.rst b/docs/api/mangadap.util.exception_tools.rst
diff --git a/docs/api/mangadap.util.pkg.rst b/docs/api/mangadap.util.pkg.rst
@@ -0,0 +1,8 @@
+mangadap.util.pkg module
+========================
+
+.. automodule:: mangadap.util.pkg
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/api/mangadap.util.rst b/docs/api/mangadap.util.rst
@@ -12,8 +12,7 @@ Submodules
    mangadap.util.covariance
    mangadap.util.dapbitmask
    mangadap.util.datatable
-   mangadap.util.drpfits
-   mangadap.util.exception_tools
+   mangadap.util.drpbitmask
    mangadap.util.extinction
    mangadap.util.fileio
    mangadap.util.filter
@@ -26,6 +25,7 @@ Submodules
    mangadap.util.modeling
    mangadap.util.parser
    mangadap.util.pixelmask
+   mangadap.util.pkg
    mangadap.util.resolution
    mangadap.util.sampling
    mangadap.util.trace