doc: include Substra API Reference (#17)

* add substra api doc

* lint

* add myst parser to requirements

* same modifs in build.yml

* suppress warnings

* add publish doc

* uppdate

* update changelog

.github/workflows/build.yml

@@ -16,16 +16,18 @@ jobs:
         with:
           python-version: 3.8
 
-      - name: Checkout pyconfig from a private repos
+      - name: Clone substra
         uses: actions/checkout@v2
         with:
           repository: owkin/substra
           token: ${{ secrets.ACCESS_TOKEN }}
           path: substra
 
-      - name: Install package
-        run: |
-             pip install -e substra
+      - name: Install substra
+        run: pip install -e substra
+
+      - name: Copy substra api doc in the doc
+        run: cp -r substra/references docs/source/documentation/references
 
       - name: Install requirements
         run: pip install -r requirements.txt

.github/workflows/publish.yml

@@ -21,16 +21,18 @@ jobs:
         with:
           python-version: 3.8
 
-      - name: Checkout pyconfig from a private repos
+      - name: Clone substra
         uses: actions/checkout@v2
         with:
           repository: owkin/substra
           token: ${{ secrets.ACCESS_TOKEN }}
           path: substra
 
-      - name: Install package
-        run: |
-             pip install -e substra
+      - name: Install substra
+        run: pip install -e substra
+
+      - name: Copy substra api doc in the doc
+        run: cp -r substra/references docs/source/documentation/references
 
       - name: Install requirements
         run: pip install -r requirements.txt

CHANGELOG.md

@@ -5,6 +5,9 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+### Added
+- doc:  include Substra API Reference (#17)
 ## [0.1.1] - 2021-12-07
 ### Added
 - Skeleton for Connect Documentation

README.md

@@ -8,14 +8,39 @@ Welcome to Connect documentation. [Here](https://owkin-connect-documentation.rea
 
 If you would like to contribute to this documentation please clone it locally and make a new branch with the suggested changes.
 
-To deploy the documentation locally you will need to install all the necessary requirements which you can find in the 'requirements.txt' file of the root of this repository. You can use pip in your terminal to install it: pip install -r requirements.txt
+To deploy the documentation locally you will need to install all the necessary requirements which you can find in the 'requirements.txt' file of the root of this repository. You can use pip in your terminal to install it: `pip install -r requirements.txt`
 
-Next, to build the documentation move to the docs directory: cd docs
+Install substra: `pip install substra`
+Next, to build the documentation move to the docs directory: `cd docs`
 
-And then: make html
+And then: `make html`
 
-The first time you run it or if you updated the examples libary it may take a little longer to build the whole documentation.
+The first time you run it or if you updated the examples library it may take a little longer to build the whole documentation.
+
+To see the doc on your browser : `make live-html`
+And then go to http://127.0.0.1:8000
 
 Once you are happy with your changes push your branch and make a pull request.
 
 Thank you for helping us improving!
+
+## To publish on RTD before merging your PR:
+
+```sh
+    # See the tags
+    git tag
+    # Add a new tag
+    git tag -a X.X.X-your-branch
+    git push origin X.X.X-your-branch
+```
+
+Check the publish action is running: https://github.com/owkin/connect-documentation/actions
+
+Activate your version on RTD (need admin rights): https://readthedocs.com/projects/owkin-connect-documentation/versions/
+
+Follow the build here : https://readthedocs.com/projects/owkin-connect-documentation/builds/
+
+See the doc on https://owkin-connect-documentation.readthedocs-hosted.com/en/X.X.X-your-branch
+
+If everything is OK, you can delete your version on RTD (wipe button): https://readthedocs.com/projects/owkin-connect-documentation/versions/
+and delete your tag : `git push --delete origin X.X.X-your-branch`

docs/source/conf.py

@@ -69,6 +69,26 @@ def zip_dir(source_dir):
 assets_dir = Path(__file__).parents[2] / "examples" / "assets"
 zip_dir(assets_dir)
 
+# reformat links to a section in a markdown files (not supported by myst_parser)
+def reformat_md_section_links(file_path: Path):
+    # Read in the file
+    with open(file_path, 'r') as file :
+        filedata = file.read()
+
+    # replace ".md#" by ".html#"
+    filedata = filedata.replace(".md#", ".html#")
+    filedata = re.sub(r"#(.*)\)", lambda m: m.group().lower(), filedata)
+
+    # Write the file out again
+    with open(file_path, 'w') as file:
+        file.write(filedata)
+
+for file_path in Path('.').rglob('*.md'):
+    reformat_md_section_links(file_path)
+
+# suppress warnings due to sections with the same name in substra api reference
+# TODO: change those sections names in substra ? Or ignore warnings just for those files
+suppress_warnings = ['autosectionlabel.*']
 
 # -- Project information -----------------------------------------------------
 
@@ -102,12 +122,12 @@ def zip_dir(source_dir):
     "sphinx.ext.napoleon",
     "sphinx.ext.ifconfig",
     "sphinx_click",
-    "recommonmark",
     "sphinx.ext.autosectionlabel",
     "sphinx.ext.todo",
-    "sphinx_fontawesome"
+    "sphinx_fontawesome",
+    "myst_parser",  # we need it for links between md files. Recommanded by sphinx : https://www.sphinx-doc.org/en/master/usage/markdown.html
 ])
-todo_include_todos=True
+todo_include_todos = True
 
 if on_rtd:
     # The name of your GitHub repository
@@ -155,9 +175,6 @@ def zip_dir(source_dir):
 # generate autosummary even if no references
 autosummary_generate = True
 
-# The suffix of source filenames.
-source_suffix = '.rst'
-
 # Generate the plot for the gallery
 # plot_gallery = True
 

docs/source/documentation/api_reference.rst

@@ -1,2 +1,30 @@
 API reference
 =============
+
+CLI Reference
+-------------
+.. toctree::
+    :maxdepth: 2
+
+    references/cli.md
+
+SDK Reference
+-------------
+.. toctree::
+    :maxdepth: 1
+
+    references/sdk.md
+
+Models
+^^^^^^^^^^^^^
+.. toctree::
+    :maxdepth: 2
+
+    references/sdk_models.md
+
+Schemas
+^^^^^^^^^^^^^
+.. toctree::
+    :maxdepth: 2
+
+    references/sdk_schemas.md

docs/source/documentation/concepts.rst

@@ -91,20 +91,20 @@ Permissons for a node to process an asset
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 A node can execute a task (train task, composite train task, aggregate task, test task) if it has the permissions on the input assets of the task. For example, if a node wants to execute a train task, the node needs to have process permissions on the algorithm, the dataset and the input models used in the task.
-The permission on an asset is defined either at creation or by inheritance. Permissions can be defined individually for every node. Permissions cannot be modified once the asset is created. 
+The permission on an asset is defined either at creation or by inheritance. Permissions can be defined individually for every node. Permissions cannot be modified once the asset is created.
 
 
 Datasets, algorithms, metrics
 """""""""""""""""""""""""""""
-Permissions are defined at creation by their owner for datasets, algorithms and metrics. 
+Permissions are defined at creation by their owner for datasets, algorithms and metrics.
 
 
 Models
 """"""
 For train tasks and aggregate tasks, permissions on the model outputted by the task are defined by inheritance (intersection) of the permissions of the input assets. If a node can execute a train task or an aggregate task, it will necessarily have permissions on the model outputted by this task.
 
 
-For composite train tasks, the out model is split in a trunk model and a head model: 
+For composite train tasks, the out model is split in a trunk model and a head model:
 
 * The trunk model permissions are specified by the user when registering the composite train task.
 * The head model permissions are set to be non-public, meaning that the head model can only be processed by the node where the task is executed.

docs/source/index.rst

@@ -24,7 +24,7 @@ Connect is a framework offering distributed orchestration of machine learning ta
 
   Connect changelog and compatibility table.
 
-   
+
 
 .. toctree::
    :glob:
@@ -77,4 +77,3 @@ Connect is a framework offering distributed orchestration of machine learning ta
 
    additional/release.rst
    additional/contact.rst
-

requirements.txt

@@ -6,13 +6,12 @@ sphinx-autobuild==0.7.1
 sphinx-click==2.5.0
 click==7.1.2
 texttable==1.6.3
-recommonmark==0.6.0
-
+myst-parser==0.16.1
 # Sphinx 3.3.1 does not require a specific version of docutils
 # but docutils 0.17 changed the output html markup, breaking the RTD theme
 # original issue: https://github.com/sphinx-doc/sphinx/issues/9051
 docutils==0.16
 sphinx-gallery==0.7.0
 sphinx_fontawesome
 
-matplotlib
+matplotlib