Improve documentation for evaluate package (#543)

* Changed documentation theme to sphinx-book-theme

* Added details about Pandoc requirment

* Added initial document for accuracy.py

* Fix pre-commit hook errors

* Added confusion matrix doc

* Add auroc doc and fix titles

* Add average precision docs

* Added more documents for metrics

* Add index for user guide in side bar

* Minor patches (#542)

* Fix capitalization of Python

* Change ValueError to TypeError and update docstrings

* Add Precision-Recall Curve to experimental metrics (#544)

* Add ROC curve (#545)

* Add ROC metrics to experimental module

* fix typo

* Fix docstrings

* Add AUROC (#546)

* Add AUROC metric to experimental module

* Refactor binary and multiclass ROC functions

* Refactor tests to use a common thresholds list

* Fix mypy error

* Add regression metrics (#547)

* Add regression support

* update docstrings

* Ignore `no-any-return` errors

* Integrate experimental metrics with other modules (#549)

* integrate experimental metrics with other modules

* add average precision metric to experimental metrics package

* fix tutorials

* Add type hints and keyword arguments to metrics classes

* Update nbsphinx version to 0.9.3

* Update nbconvert version to 7.14.2

* Fix type annotations and formatting issues

* Update kernel display name in mortality_prediction.ipynb

* Add guard clause to prevent module execution on import

* Update `torch_distributed.py` with type hints

* Add multiclass and multilabel average precision metrics

* Change jupyter kernel

* Fix type annotations for metric values in ClassificationPlotter

---------

Co-authored-by: Amrit K <amritk@vectorinstitute.ai>

* Changed documentation theme to sphinx-book-theme

* Don't update deps

* Pre-commit fixes

* Add doc for ROC Curve

* Fix footer in docs

* Bump to 0.2.3

Signed-off-by: Amrit Krishnan <amrit110@gmail.com>

* Add logo and css fixes

* Fix formatting of conf.py file

* Small fix

* Move evaluation metrics docs under API reference

* Fix poetry build

* Improve documentation structure

* Formatting fixes

---------

Signed-off-by: Amrit Krishnan <amrit110@gmail.com>
Co-authored-by: Franklin <41602287+fcogidi@users.noreply.github.com>
Co-authored-by: Amrit K <amritk@vectorinstitute.ai>
Co-authored-by: Amrit Krishnan <amrit110@gmail.com>

README.md

@@ -89,6 +89,8 @@ guidelines.
 
 ## 📚 [Documentation](https://vectorinstitute.github.io/cyclops/)
 
+If you need to build the documentations locally, make sure to install ``Pandoc`` in addition to ``docs`` poetry group.
+
 
 ## 📓 Notebooks
 

docs/source/_static/css/cyclops.css

@@ -0,0 +1,50 @@
+footer span.commit code,
+.rst-content pre.literal-block,
+.rst-content div[class^='highlight'] pre,
+.rst-content .linenodiv pre,
+.rst-content tt,
+.rst-content code,
+.rst-content pre,
+.rst-content kbd,
+.rst-content samp {
+  font-family: 'IBM Plex Mono', monospace;
+  font-size: 0.8rem;
+}
+
+footer {
+  color: var(--text-color);
+}
+footer .footer-small-text {
+  font-weight: 300;
+  font-size: 0.9rem;
+}
+footer .copyright {
+  font-weight: 300;
+  font-size: 0.8rem;
+}
+footer div.logo {
+  display: flex;
+  flex-wrap: wrap;
+  justify-content: space-around;
+  margin: 0px;
+  padding: 10px 0px 0px 0px;
+}
+footer a.logo {
+  /* Using flex here (to vertically align the child img) causes aspect-ratio issues */
+  flex-basis: 120px;
+  margin: 10px auto 10px auto;
+  text-align: center;
+}
+footer a.logo:hover {
+  text-decoration: none;
+}
+footer span.logo {
+  display: inline-block;
+  height: 100%;
+  vertical-align: middle;
+}
+footer img.logo {
+  display: inline-block;
+  vertical-align: middle;
+  height: auto;
+}

docs/source/_templates/sections/footer.html

@@ -0,0 +1,26 @@
+{% block extrafooter %}
+
+<div class="footer-container" style="text-align: center;">
+  <div class="footer-small-text">
+      <br> All <a href="https://github.com/VectorInstitute/cyclops">source code for cyclops</a> is freely available under the terms of an <a href="https://opensource.org/licenses/Apache-2.0">Apache-2.0</a> license.
+  </div>
+
+  <div class="footer-small-text">
+    This work is made possible due to the data obtained from the <a href="https://www.geminimedicine.ca/">GEMINI study</a>.
+  </div>
+
+  <hr/>
+
+  <div class="logo">
+    <a class="logo" href="https://vectorinstitute.ai/">
+      <span class="logo"></span>
+      <img alt="Logo of the Vector Institute" style="width:90px; max-width:90px;" src="{{ pathto('_static/logos/vector_logo.png', 1) }}" />
+    </a>
+    <a class="logo" href="https://www.geminimedicine.ca/">
+      <span class="logo"></span>
+      <img alt="Logo of GEMINI" style="width:90px; max-width:90px;" src="{{ pathto('_static/logos/gemini_logo.png', 1) }}" />
+    </a>
+  </div>
+</div>
+
+{% endblock %}

docs/source/api.rst

@@ -2,9 +2,13 @@ API Reference
 =============
 
 .. toctree::
+   :maxdepth: 1
+   :glob:
 
    reference/api/cyclops.data.rst
    reference/api/cyclops.tasks.rst
-   reference/api/cyclops.evaluate.rst
+   reference/api/evaluator.rst
+   reference/api/fairness_evaluator.rst
+   reference/api/metrics.rst
    reference/api/cyclops.monitor.rst
    reference/api/cyclops.report.rst

docs/source/conf.py

@@ -59,6 +59,14 @@
 copybutton_prompt_text = r">>> |\.\.\. "
 copybutton_prompt_is_regexp = True
 
+html_theme_options = {
+    "collapse_navigation": False,
+    "sticky_navigation": True,
+    "navigation_depth": 4,
+    "includehidden": True,
+    "titles_only": False,
+}
+
 intersphinx_mapping = {
     "python": ("https://docs.python.org/3.9/", None),
     "numpy": ("http://docs.scipy.org/doc/numpy/", None),
@@ -84,10 +92,19 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "theme"
-html_theme_path = ["."]
+html_theme = "sphinx_book_theme"
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ["_static"]
+html_css_files = ["css/cyclops.css"]
+html_logo = "_static/cyclops_logo-dark.png"
+html_favicon = "_static/favicon.ico"
+html_theme_options = {
+    "logo": {  # type: ignore
+        "text": "cyclops documentation",
+        "image_light": "_static/cyclops_logo-dark.png",
+        "image_dark": "_static/cyclops_logo-dark.png",
+    },
+}

docs/source/developing.rst

@@ -0,0 +1,34 @@
+🧑🏿‍💻 Developing
+=======================
+
+Using poetry
+------------
+
+The development environment can be set up using `poetry <https://python-poetry.org/docs/#installation>`__. Hence, make sure it is installed and then run:
+
+.. code:: bash
+
+   python3 -m poetry install
+   source $(poetry env info --path)/bin/activate
+
+In order to install dependencies for testing (codestyle, unit tests, integration tests), run:
+
+.. code:: bash
+
+   python3 -m poetry install --with test
+
+API documentation is built using `Sphinx <https://www.sphinx-doc.org/en/master/>`__ and can be locally built by:
+
+.. code:: bash
+
+   python3 -m poetry install --with docs
+   cd docs
+   make html SPHINXOPTS="-D nbsphinx_allow_errors=True"
+
+
+If you need to build the documentations locally, make sure to install ``Pandoc`` in addition to ``docs`` poetry group.
+
+Contributing
+------------
+
+Contributing to cyclops is welcomed. See `Contributing <https://vectorinstitute.github.io/cyclops/api/contributing.html>`__ for guidelines.

docs/source/evaluation.rst

@@ -0,0 +1,2 @@
+Evaluation
+==========

docs/source/index.rst

@@ -6,7 +6,8 @@ Welcome to cyclops's documentation!
    :maxdepth: 2
    :caption: Contents:
 
-   intro
-   contributing
+   user_guide
    tutorials
+   developing
+   contributing
    api

docs/source/installation.rst

@@ -0,0 +1,29 @@
+Installation
+============
+
+Using pip
+---------
+
+.. code:: bash
+
+   python3 -m pip install pycyclops
+
+``cyclops`` has many optional dependencies that are used for specific functionality. For example, the `monai <https://github.com/Project-MONAI/MONAI>`__ library is used for loading DICOM images to create datasets. All optional dependencies can be installed with ``pycyclops[all]``, and specific sets of dependencies are listed in the sections below.
+
++-----------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------+
+| Dependency                  | pip extra                | Notes                                                                                                         |
++=============================+==========================+===============================================================================================================+
+| xgboost                     | xgboost                  | Allows use of `XGBoost <https://xgboost.readthedocs.io/en/stable/>`__ model                                   |
++-----------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------+
+| torch                       | torch                    | Allows use of `PyTorch <https://pytorch.org/>`__ models                                                       |
++-----------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------+
+| torchvision                 | torchvision              | Allows use of `Torchvision <https://pytorch.org/vision/stable/index.html>`__ library                          |
++-----------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------+
+| torchxrayvision             | torchxrayvision          | Uses `TorchXRayVision <https://mlmed.org/torchxrayvision/>`__ library                                         |
++-----------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------+
+| monai                       | monai                    | Uses `MONAI <https://github.com/Project-MONAI/MONAI>`__ to load and transform images                          |
++-----------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------+
+| alibi                       | alibi                    | Uses `Alibi <https://docs.seldon.io/projects/alibi/en/stable/>`__ for additional explainability functionality |
++-----------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------+
+| alibi-detect                | alibi-detect             | Uses `Alibi Detect <https://docs.seldon.io/projects/alibi-detect/en/stable/>`__ for dataset shift detection   |
++-----------------------------+--------------------------+---------------------------------------------------------------------------------------------------------------+

docs/source/model_report.rst

@@ -0,0 +1,2 @@
+Model Report
+============

docs/source/monitoring.rst

@@ -0,0 +1,2 @@
+Monitoring
+==========