Update documentation

docs/developer_guides/benchmark.rst

@@ -0,0 +1,78 @@
+.. _int_benchmark:
+
+===============================
+Contributing to Orion Benchmark
+===============================
+
+Below we provide a guide on how to integrate new pipelines, datasets, and metrics.
+
+Adding a Pipeline to Orion Benchmark
+------------------------------------
+
+To add a new pipeline to the benchmark, we need to make it part of the Orion pipelines available for all users. Before starting, please review some of the following concepts to help you understand pipelines. 
+
+- **Standardized Orion I/O Format:** please visit :ref:`data` page.
+- **Primitives and Pipelines:**
+    - **Creating Primitives:** please visit :ref:`primitives` page.
+    - **Creating Pipelines:** please visit :ref:`pipelines` page.
+
+Once you have a pipeline that you have been using that you would like to add to Orion, please follow these steps in order to make an official Orion pipeline!
+
+1. **Open an issue:** first task is to `open an issue`_ on Orion github detailing your pipeline and its primitives. This will help maintainers understand what the pipeline is doing and suggest potential primitives that can be used rather than creating one from scratch.
+2. **Open a pull request:** after describing your pipeline clearly in an issue, you can open a pull request on Orion github to make your contributions. This pull request should have the following files added or modified:
+    1. **primitive files:** this includes any python code and primitive JSON files needed by the pipeline. These files should be stored in ``orion/primitives`` and ``orion/primitives/jsons`` respectively. Moreover, for any newly added classes and functions, please include unit tests to make sure they are performing as intended in ``tests/primitives``.
+    2. **pipeline files:** this includes the main pipeline JSON file. Create a new directory under ``orion/pipelines/sandbox`` with the given pipeline name and store the pipeline JSON file inside it. Imagine the model name is ``new_model`` then the pipeline should have the following path ``orion/pipelines/sandbox/new_model/new_model.json``.
+    3. **hyperparameter files:** in addition to the pipeline JSON file, create hyperparameter JSON files for each dataset used in the benchmark and store it in the same location as the pipeline. As an example, for dataset SMAP, create ``orion/pipelines/sandbox/new_model/new_model_smap.json``. The hyperparameter file should only modify the data specific hyperparameters such as the interval level of the signals in the dataset.
+    4. **documentation:** lastly there is documentation files. Create a new ``docs/user_guides/primitives_pipelines/primitives/new_primitive.rst`` that describes the primitive for any newly added primitive. Moreover, include modify ``docs/user_guides/primitives_pipelines/pipelines.rst`` and ``docs/user_guides/primitives_pipelines/primitives.rst`` as needed.
+3. **Reviewing:** once the pull request is made, the Orion team will take a look at your contributions and make any necessary suggestions. When the pull request passes unit and integration tests and the code is approved by the reviewers, it will be merged. The pipeline will remain in sandbox until it passes the verification testing phase.
+4. **Verification Testing:** to ensure that pipelines are robust, reproducible, and can be maintained in the long run, several tests and validations are made. This includes testing the pipeline in our benchmarks.
+5. **Transferring from Sandbox to Verified:** once the pipeline passes verification testing, it becomes an Orion verified pipeline and will be included in all future releases of the benchmark.
+
+
+Adding a Dataset to Orion Benchmark
+-----------------------------------
+
+To add a new dataset to the benchmark, it needs to follow the Orion format. Visit the :ref:`data` to familiarize yourself with it.
+
+Once the data follows the expected input format, please follow these steps to introduce the dataset to the benchmark:
+
+1. **Open an issue:** first task is to `open an issue`_ on Orion github pointing to the source of the data. If the data needs formatting, please attach a link to a notebook (either in your fork or a colab notebook) to make the integration faster.
+2. **Adding the dataset to S3:** if the data is publicly available, we would like to make it available to all Orion users by adding it to our ``sintel-orion`` public s3 bucket. This way, users can load any signal directly from Orion using ``load_signal`` functionality. This task will be performed by the Orion team.
+3. **Adding dataset/pipeline hyperparameters:** open a new pull request that will feature adding the respective hyperparameters of the dataset for each pipeline. For example, for a new dataset named ``new_data`` we will have ``aer_new_data.json``, ``tadgan_new_data.json``, etc. While this might feel redundant, it is crucial to maintain transparency of hyperparameter settings in our benchmarks.
+4. **Adding the dataset to leaderboard:** in the same pull request, kindly add the name of the dataset to the dictionaries included in this `file <https://github.com/sintel-dev/Orion/blob/master/orion/results.py>`__.
+
+Once the pull request is merged, the dataset will then be featured in subsequent releases of the benchmark!
+
+
+Adding an Evaluation Metric to Orion Benchmark
+----------------------------------------------
+
+Orion has an evaluation sub-package for evaluating anomaly detection performance. Before adding a new evaluation metric, please visit :ref:`evaluation_doc`.
+
+To make an evaluation function it needs to accept at least two arguments:
+
+- ``expected``: which is a list of known ground truth anomalies.
+- ``observed``: which is a list of detected anomalies.
+
+If you are working with *point* anomalies, you can add your metric to ``point.py`` in the evaluation sub-package, and if you are working with *contextual* anomalies, please include it in ``contextual.py``.
+
+Once you have created your metric, you can start the process of integrating it to the benchmark:
+
+1. **Open an issue:** first task is to `open an issue`_ on Orion github detailing the specifications of the new evaluation metric and how it will be useful to users.
+2. **Open a pull request:** after describing your new metric clearly in an issue, you can open a pull request on Orion github to make your contributions. Below we describe what the PR should include:
+    1. **metric files:** this includes the python functions that implement the evaluation metric. Note that these files should be stored in the evaluation sub-package.
+    2. **benchmark file:** to add the new metric to the benchmark, simply add it to the dictionary ``METRICS``.  Once included, it will be added to the benchmark detailed sheet results.
+    3. **documentation:** lastly there is documentation. Add a description of the new metric in ``docs/user_guides/evaluation_doc.rst``. It is also valuable to add an example usage of the metric with an expected result.
+3. **Reviewing:** once the pull request is made, the Orion team will make any necessary suggestions. Please include docstrings in your code to help the team in the reviewing process.
+
+Once the PR is merged, the new evaluation metric will be available to all users. Moreover, subsequent benchmark released will contain the new metric in their benchmark results.
+
+
+Resources
+---------
+
+- **Data Format:** :ref:`data` page.
+- **Primitives:** :ref:`primitives` page.
+- **Pipelines:** :ref:`pipelines` page.
+
+.. _open an issue: https://github.com/sintel-dev/Orion/issues/new

docs/developer_guides/index.rst

@@ -6,4 +6,5 @@ In the Developer Guides we discuss in depth the architecture of the Orion projec
 .. toctree::
     :maxdepth: 2
 
-    contributing
+    contributing
+    int_benchmark

docs/getting_started/install.rst

@@ -10,7 +10,7 @@ Requirements
 
 Python
 ~~~~~~
-**Orion** has been tested on **GNU/Linux**, and **macOS** systems running `Python 3.6, 3.7, or 3.8`_ installed.
+**Orion** has been tested on **GNU/Linux**, **macOS**, and **windows** systems running `Python 3.6, 3.7, or 3.8`_ installed.
 
 Also, although it is not strictly required, the usage of a `virtualenv`_ is highly recommended in
 order to avoid having conflicts with other software installed in the system where you are trying to run **Orion**.
@@ -55,4 +55,4 @@ can be found in the :ref:`contributing` guide.
 .. _virtualenv: https://virtualenv.pypa.io/en/latest/
 .. _pip: https://pip.pypa.io
 .. _PyPI: https://pypi.org/
-.. _Github repository: https://github.com/sintel-dev/Orion
+.. _Github repository: https://github.com/sintel-dev/Orion

docs/user_guides/benchmarking.rst

@@ -17,7 +17,7 @@ Results obtained during benchmarking as well as previous releases can be found w
 Leaderboard
 ~~~~~~~~~~~
 
-We run the benchmark on **11** datasets with their known grounth truth. We record the score of the pipelines on each datasets. To compute the leaderboard table, we showcase the number of wins each pipeline has over the ARIMA pipeline. 
+We run the benchmark on **12** datasets with their known grounth truth. We record the score of the pipelines on each datasets. To compute the leaderboard table, we showcase the number of wins each pipeline has over the ARIMA pipeline. 
 
 .. mdinclude:: ../../benchmark/leaderboard.md
 
@@ -58,6 +58,7 @@ The complete evaluation process is directly available using the
 For further details about all the arguments and possibilities that the ``benchmark`` function offers please refer to the `Orion benchmark
 documentation <https://github.com/sintel-dev/Orion/blob/master/BENCHMARK.md>`__
 
+
 .. _benchmark/results: https://github.com/sintel-dev/Orion/tree/master/benchmark/results
 .. _summary Google Sheets document: https://docs.google.com/spreadsheets/d/1ZPUwYH8LhDovVeuJhKYGXYny7472HXVCzhX6D6PObmg/edit?usp=sharing
-.. _details Google Sheets document: https://docs.google.com/spreadsheets/d/1HaYDjY-BEXEObbi65fwG0om5d8kbRarhpK4mvOZVmqU/edit?usp=sharing
+.. _details Google Sheets document: https://docs.google.com/spreadsheets/d/1HaYDjY-BEXEObbi65fwG0om5d8kbRarhpK4mvOZVmqU/edit?usp=sharing

docs/user_guides/data.rst

@@ -9,65 +9,77 @@ Orion takes a time series signal and produces an interval of expected anomalies.
 Data Format
 -----------
 
+All Orion pipelines follow the same format in terms of what it is expecting as input (a signal represented in a ``pandas`` dataframe) and output (a ``list`` of detected anomalous intervals that later is rendered by Orion into a dataframe). 
+
 Input
 ~~~~~
 
-Orion Pipelines work on time Series that are provided as a single table of telemetry
-observations with two columns:
+Orion Pipelines work on univariate or multivariate time series that are provided as a single table of telemetry observations with at least two columns:
+
+* ``timestamp``: an ``int`` column with the time of the observation in Unix Time format.
+* ``value``: an ``int`` or ``float`` column with the observed value at the indicated timestamp. Each value is represented as a different column.
 
-* ``timestamp``: an INTEGER or FLOAT column with the time of the observation in Unix Time Format
-* ``value``: an INTEGER or FLOAT column with the observed value at the indicated timestamp
+Here is an example of signal:
 
-This is an example of such table:
++------------+-----------+------------+-----------+
+|  timestamp |   value 0 |    value 1 |   value 2 |
++------------+-----------+------------+-----------+
+| 1222819200 | -0.366358 |          0 |         1 |
++------------+-----------+------------+-----------+
+| 1222840800 | -0.394107 |          0 |         0 |
++------------+-----------+------------+-----------+
+| 1222862400 |  0.403624 |          1 |         0 |
++------------+-----------+------------+-----------+
+| 1222884000 | -0.362759 |          0 |         0 |
++------------+-----------+------------+-----------+
+| 1222905600 | -0.370746 |          0 |         0 |
++------------+-----------+------------+-----------+
 
-+------------+-----------+
-|  timestamp |     value |
-+------------+-----------+
-| 1222819200 | -0.366358 |
-+------------+-----------+
-| 1222840800 | -0.394107 |
-+------------+-----------+
-| 1222862400 |  0.403624 |
-+------------+-----------+
-| 1222884000 | -0.362759 |
-+------------+-----------+
-| 1222905600 | -0.370746 |
-+------------+-----------+
+The table above contains an observation column value 0, and two status values 1 and 2.
 
 Output
 ~~~~~~
 
-The output of the Orion Pipelines is another table that contains the detected anomalous
-intervals and that has at least two columns:
+There can be many steps (primitives) in a pipeline, however the end result must be a list of intervals. Each interval contains at least two entries: 
 
 * ``start``: timestamp where the anomalous interval starts
 * ``end``: timestamp where the anomalous interval ends
 
-Optionally, a third column called ``severity`` can be included with a value that represents the
-severity of the detected anomaly.
+Optionally, there can be a third entry that contains a proxy to the likelihood that the interval is anomalous.
+
+.. code-block:: python
+
+    [
+        (1222840800, 1222884000, 0.12572),
+        (1402012800, 1403870400, 0.64339),
+    ]
 
-An example of such a table is:
+
+For example, the list above shows that the pipeline detected two intervals as anomalous, the second one being more likely than the first.
+
+Then Orion will present this result as a ``pandas`` dataframe:
 
 +------------+------------+----------+
 |      start |        end | severity |
 +------------+------------+----------+
-| 1222970400 | 1222992000 | 0.572643 |
+| 1222840800 | 1222884000 |  0.12572 |
 +------------+------------+----------+
-| 1223013600 | 1223035200 | 0.572643 |
+| 1402012800 | 1403870400 |  0.64339 |
 +------------+------------+----------+
 
+
 Dataset we use in this library
 ------------------------------
 
 For development, evaluation of pipelines, we include a dataset which includes several signals already formatted as expected by the Orion Pipelines.
 
-This formatted dataset can be browsed and downloaded directly from the `d3-ai-orion AWS S3 Bucket`_.
+This formatted dataset can be browsed and downloaded directly from the `sintel-orion AWS S3 Bucket`_.
 
 This dataset is adapted from the one used for the experiments in the `Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding paper`_. Original source data is available for download `here`_.
 
 We thank NASA for making this data available for public use.
 
-.. _d3-ai-orion AWS S3 Bucket: https://d3-ai-orion.s3.amazonaws.com/index.html
+.. _sintel-orion AWS S3 Bucket: https://sintel-orion.s3.amazonaws.com/index.html
 .. _Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding paper: https://arxiv.org/abs/1802.04431
 .. _here: https://s3-us-west-2.amazonaws.com/telemanom/data.zip