Parameter handling

.gitignore

@@ -69,7 +69,8 @@ instance/
 .scrapy
 
 # Sphinx documentation
-docs/_build/
+doc/sphinx/_build
+doc/sphinx/reference
 
 # PyBuilder
 target/
@@ -127,3 +128,10 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# VisualCode IDE
+.vscode
+
+# Scalene profiling files
+profile.json
+profile.html

CHANGELOG.txt

@@ -1,8 +1,52 @@
+This file contains a log-book for major changes between releases.
+
+v23.2.0
+=======
+- Complete overhaul of SkyLLH for more generic handling of parameters
+  (fixed & fit parameters) and their mapping to sources.
+
+    - Single class core.parameters.Parameter for fixed and fit parameters.
+    - New class core.parameters.ParameterModelMapper to map global parameters
+      to local model (e.g. source) parameters.
+    - Class method core.pdf.PDF.get_pd takes structured numpy.ndarray as
+      argument, holding local parameter names and values for each source.
+
+- Intrinsic support for multiple sources.
+
+    - New class property core.trialdata.TrialDataManager.src_evt_idxs with
+      source and event indices mapping selected events to the sources.
+
+- Simplification of the class hierarchy & source code structure.
+- New generic flux model function concept.
+- New generic PDF class for time PDFs.
+- Support for Parquet data files via optional pyarrow Python package.
+- Introduce factory concept for detector signal yield construction.
+
+    DetSigYieldBuilder classes can provide a factory function to construct
+    DetSigYield instances efficiently for multiple flux models.
+
+- Speed improvements for analysis creation and trial generation.
+
+    - i3.detsigyield.SingleParamFluxPointLikeSourceI3DetSigYieldBuilder:
+
+        MC events are sorted in sin(true_dec) before histograms are made for
+        each parameter value. Thus, numpy.histogram works faster.
+
+    - core.event_selection.SpatialBoxEventSelectionMethod:
+
+        Improved calculation of right-ascention difference. Thus, speed-up in
+        trial generation when using this event selection method.
+
+- Usage of the tqdm Python package for progress bars.
+- More unit tests.
+- Improved documentation.
+- Following PEP8 coding style standard.
+
 v23.1.1
 =======
 - bugfix of core.pdf.MultiDimGridPDFSet
 - bugfix of core.pdf.MappedMultiDimGridPDFSet
 - removal of depricated proptery core.pdf.PDFSet.pdf_axes
 
 v23.1.0
-=======
+=======

README.md

@@ -19,12 +19,12 @@ pip install skyllh
 
 The current development version can be installed using pip:
 ```bash
-pip install git+https://github.com/icecube/skyllh.git#egg=skyllh 
+pip install git+https://github.com/icecube/skyllh.git#egg=skyllh
 ```
 
 Optionally, the editable package version with a specified reference can be installed by:
 ```bash
-pip install -e git+https://github.com/icecube/skyllh.git@[ref]#egg=skyllh 
+pip install -e git+https://github.com/icecube/skyllh.git@[ref]#egg=skyllh
 ```
 where
 - `-e` is an editable flag
@@ -41,6 +41,15 @@ sys.path.insert(0, '/path/to/skyllh')
 sys.path.insert(0, '/path/to/i3skyllh')  # optional
 ```
 
+# Publications
+
+Several publications about the SkyLLH software are available:
+
+- IceCube Collaboration, T. Kontrimas, M. Wolf, et al. PoS ICRC2021 (2022) 1073
+  [DOI](http://doi.org/10.22323/1.395.1073)
+- IceCube Collaboration, M. Wolf, et al. PoS ICRC2019 (2020) 1035
+  [DOI](https://doi.org/10.22323/1.358.1035)
+
 # i3skyllh
 
 The [`i3skyllh`](https://github.com/icecube/i3skyllh) package provides complementary pre-defined common analyses and datasets for the [IceCube Neutrino Observatory](https://icecube.wisc.edu) detector in a private [repository](https://github.com/icecube/i3skyllh).

doc/sphinx/Makefile

@@ -9,24 +9,24 @@ BUILDDIR      = _build
 
 # Put it first so that "make" without argument is like "make help".
 help:
-	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)
 
 .PHONY: help Makefile html clean
 
 # Build html documentation.
 html:
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)
 
 # Build html documentation for each branch separately.
 html-multiversion:
 	sphinx-multiversion "$(SOURCEDIR)" "$(BUILDDIR)"
 
 # Prepend clean target to remove skyllh API reference.
 clean:
-	rm -rf reference
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	rm -rf $(SOURCEDIR)/reference
+	@$(SPHINXBUILD) -M clean "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)
 
 # Catch-all target: route all unknown targets to Sphinx using the new
-# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+# "make mode" option.
 %: Makefile
-	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS)

doc/sphinx/concepts/detsigyield.ipynb

@@ -0,0 +1,94 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "12151349",
+   "metadata": {},
+   "source": [
+    "# Detector Signal Yield"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "69d138fc",
+   "metadata": {
+    "raw_mimetype": "text/restructuredtext"
+   },
+   "source": [
+    "Several parts of an analysis will require the calculation of the detector signal yield, which is the mean number of expected signal events in the detector for a given source hypothesis, i.e. source flux function $\\Phi(\\alpha,\\delta,E,t)$."
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "a1560531",
+   "metadata": {
+    "raw_mimetype": "text/restructuredtext"
+   },
+   "source": [
+    "SkyLLH provides two abstract base classes for creating a detector signal yield instance, :py:class:`~skyllh.core.detsigyield.DetSigYieldBuilder` and :py:class:`~skyllh.core.detsigyield.DetSigYield`. The first is the builder class, which will build a :py:class:`~skyllh.core.detsigyield.DetSigYield` class instance."
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "3b4bd127",
+   "metadata": {
+    "raw_mimetype": "text/restructuredtext"
+   },
+   "source": [
+    "The builder class has the abstract method :py:meth:`~skyllh.core.detsigyield.DetSigYieldBuilder.construct_detsigyield`, which will take a :py:class:`~skyllh.core.dataset.Dataset`, :py:class:`~skyllh.core.dataset.DatasetData`, :py:class:`~skyllh.physics.flux_model.FluxModel` instance, and live-time to construct a :py:class:`~skyllh.core.detsigyield.DetSigYield` class instance, which will provide an evaluation method to calculate the detector signal yield for a given source, for the given dataset. Hence, the detector signal yield is dataset and source model dependent."
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "653d0c06",
+   "metadata": {
+    "raw_mimetype": "text/restructuredtext"
+   },
+   "source": [
+    "The :py:class:`~skyllh.core.detsigyield.DetSigYield` class has two abstract methods, :py:meth:`~skyllh.core.detsigyield.DetSigYield.source_to_array` and :py:meth:`~skyllh.core.detsigyield.DetSigYield.__call__`."
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "60c16a6d",
+   "metadata": {
+    "raw_mimetype": "text/restructuredtext"
+   },
+   "source": [
+    "The :py:meth:`~skyllh.core.detsigyield.DetSigYield.source_to_array` method takes a sequence of source models and converts it into a numpy record array suitable for the :py:meth:`~skyllh.core.detsigyield.DetSigYield.__call__` method to evaluate the detector signal yield efficiently for a list of sources. The :py:meth:`~skyllh.core.detsigyield.DetSigYield.__call__` method evaluates the :py:class:`~skyllh.core.detsigyield.DetSigYield` instance. As arguments it takes the source record array created by the :py:meth:`~skyllh.core.detsigyield.DetSigYield.source_to_array` method, and the numpy record array holding the (local) source parameter values."
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "id": "d75df35f",
+   "metadata": {
+    "raw_mimetype": "text/restructuredtext"
+   },
+   "source": [
+    "The record array holding the local source parameter values can be generated through the :py:class:`~skyllh.core.parameters.ParameterModelMapper.create_src_params_recarray` of the :py:class:`~skyllh.core.parameters.ParameterModelMapper` instance of the analysis. See also the  :ref:`Parameter to Model mapping <sec:ParameterToModelMapping>` section."
+   ]
+  }
+ ],
+ "metadata": {
+  "celltoolbar": "Raw Cell Format",
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.8.10"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}