From 7c07ab701487cdc4837667fce52b1bca708f6d22 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guilhem=20Barth=C3=A9s?= Date: Tue, 25 Jun 2024 17:19:47 +0200 Subject: [PATCH 1/7] doc: add volume tables MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Guilhem Barthés --- docs/source/reference/volumes.rst | 41 +++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 docs/source/reference/volumes.rst diff --git a/docs/source/reference/volumes.rst b/docs/source/reference/volumes.rst new file mode 100644 index 00000000..ba011df6 --- /dev/null +++ b/docs/source/reference/volumes.rst @@ -0,0 +1,41 @@ +Volumes +------- + +Orchestrator +************ + ++--------------+--------------+----------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+---------+ +| PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | Comment | ++==============+==============+==========+=================================+==========================+===============+================+===============+==============================================+=========+ +| Postgres PVC | orchestrator | postgres | ReadWriteOnce (can be modified) | 8 | | default | yes | postgresql.primary.persistence.existingClaim | | ++--------------+--------------+----------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+---------+ + + +Backend +******* + ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ +| PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | Comment | ++=====================+===========+============+=================================+==========================+===============+================+===============+==============================================+===============================================+ +| Postgres PVC | backend | postgres | ReadWriteOnce (can be modified) | 8 | | default | yes | postgresql.primary.persistence.existingClaim | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ +| localstack PVC | backend | localstack | ReadWriteOnce (can be modified) | 5 | | default | yes | localstack.persistence.existingClaim | Only created when `localstack.enabled = true` | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ +| minio PVC | backend | minio | ReadWriteOnce (can be modified) | 8 | | default | yes | minio.persistence.existingClaim | Only created when `minio.enabled = true` | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ +| Redis PVC | backend | redis | ReadWriteOnce (can be modified) | 8 | | default | yes | redis.master.persistence.existingClaim | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ +| Kaniko cache warmer | backend | builder | ReadWriteOnce | 10 | | default | No | | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ +| builder PVC | backend | builder | ReadWriteOnce | 10 | | default | No | | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ +| worker subtuple | backend | worker | ReadWriteOnce (can be modified) | 10 | | default | No | | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ +| Docker registry | backend | registry | ReadWriteOnce (can be modified) | 10 | | default | yes | containerRegistry.persistence.existingClaim | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ + + +Frontend +******** + +no volume used \ No newline at end of file From 4cafd99019881b706244031afb24bf66ea81fa75 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guilhem=20Barth=C3=A9s?= Date: Tue, 25 Jun 2024 17:20:13 +0200 Subject: [PATCH 2/7] doc: include volume table in reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Guilhem Barthés --- docs/source/reference/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/source/reference/index.rst b/docs/source/reference/index.rst index 7a7e449b..4cac0616 100644 --- a/docs/source/reference/index.rst +++ b/docs/source/reference/index.rst @@ -9,3 +9,4 @@ Substra is distributed as Helm charts, and most values can be customized to acco .. include:: pss.rst .. include:: netpol.rst +.. include:: volumes.rst From 3e8fae84f50ee35b80bfed38ebeaee239b93dfd8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guilhem=20Barth=C3=A9s?= Date: Tue, 25 Jun 2024 17:25:02 +0200 Subject: [PATCH 3/7] doc: improve information about `docker-registry` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Guilhem Barthés --- docs/source/reference/volumes.rst | 38 +++++++++++++++---------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/source/reference/volumes.rst b/docs/source/reference/volumes.rst index ba011df6..b31278fb 100644 --- a/docs/source/reference/volumes.rst +++ b/docs/source/reference/volumes.rst @@ -14,25 +14,25 @@ Orchestrator Backend ******* -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ -| PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | Comment | -+=====================+===========+============+=================================+==========================+===============+================+===============+==============================================+===============================================+ -| Postgres PVC | backend | postgres | ReadWriteOnce (can be modified) | 8 | | default | yes | postgresql.primary.persistence.existingClaim | | -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ -| localstack PVC | backend | localstack | ReadWriteOnce (can be modified) | 5 | | default | yes | localstack.persistence.existingClaim | Only created when `localstack.enabled = true` | -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ -| minio PVC | backend | minio | ReadWriteOnce (can be modified) | 8 | | default | yes | minio.persistence.existingClaim | Only created when `minio.enabled = true` | -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ -| Redis PVC | backend | redis | ReadWriteOnce (can be modified) | 8 | | default | yes | redis.master.persistence.existingClaim | | -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ -| Kaniko cache warmer | backend | builder | ReadWriteOnce | 10 | | default | No | | | -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ -| builder PVC | backend | builder | ReadWriteOnce | 10 | | default | No | | | -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ -| worker subtuple | backend | worker | ReadWriteOnce (can be modified) | 10 | | default | No | | | -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ -| Docker registry | backend | registry | ReadWriteOnce (can be modified) | 10 | | default | yes | containerRegistry.persistence.existingClaim | | -+---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+-----------------------------------------------+ ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ +| PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | Comment | ++=====================+===========+============+=================================+==========================+===============+================+===============+==============================================+====================================================+ +| Postgres PVC | backend | postgres | ReadWriteOnce (can be modified) | 8 | | default | yes | postgresql.primary.persistence.existingClaim | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ +| localstack PVC | backend | localstack | ReadWriteOnce (can be modified) | 5 | | default | yes | localstack.persistence.existingClaim | Only created when `localstack.enabled = true` | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ +| minio PVC | backend | minio | ReadWriteOnce (can be modified) | 8 | | default | yes | minio.persistence.existingClaim | Only created when `minio.enabled = true` | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ +| Redis PVC | backend | redis | ReadWriteOnce (can be modified) | 8 | | default | yes | redis.master.persistence.existingClaim | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ +| Kaniko cache warmer | backend | builder | ReadWriteOnce | 10 | | default | No | | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ +| builder PVC | backend | builder | ReadWriteOnce | 10 | | default | No | | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ +| worker subtuple | backend | worker | ReadWriteOnce (can be modified) | 10 | | default | No | | | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ +| Docker registry | backend | registry | ReadWriteOnce (can be modified) | 10 | | default | yes | docker-registry.persistence.existingClaim | Only created when `docker-registry.enabled = true` | ++---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ Frontend From 4e48156f117a3de0c28c3bdaf60ecde981d5f5ad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guilhem=20Barth=C3=A9s?= Date: Tue, 25 Jun 2024 17:28:57 +0200 Subject: [PATCH 4/7] doc: add some sentences about the volumes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Guilhem Barthés --- docs/source/reference/volumes.rst | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/source/reference/volumes.rst b/docs/source/reference/volumes.rst index b31278fb..46c4fb5c 100644 --- a/docs/source/reference/volumes.rst +++ b/docs/source/reference/volumes.rst @@ -1,9 +1,13 @@ Volumes ------- +This section lists the different persistent volume claims that are created during a standard deployment of Substra in a cluster. + Orchestrator ************ +The orchestrator claims are only linked with its database. This database (and the underlying volume) is important has it is where we store all the events that happen in the network. + +--------------+--------------+----------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+---------+ | PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | Comment | +==============+==============+==========+=================================+==========================+===============+================+===============+==============================================+=========+ @@ -14,6 +18,8 @@ Orchestrator Backend ******* +The backend is the more compolex component and requires different volumes for functioning. The ones that requires to be kept are re-usable and can be created outside of the deployment of the Substra stack. The other ones act as cache, and there is no way for now to re-use existing volumes using the Helm chart. + +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ | PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | Comment | +=====================+===========+============+=================================+==========================+===============+================+===============+==============================================+====================================================+ @@ -38,4 +44,4 @@ Backend Frontend ******** -no volume used \ No newline at end of file +The frontend does not need any persistent volume claim. \ No newline at end of file From 2b1dd19b905dcb29ccdb39b752a0000e25a56f7e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guilhem=20Barth=C3=A9s?= Date: Tue, 9 Jul 2024 11:41:42 +0200 Subject: [PATCH 5/7] fix: apply feedback on tables MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Guilhem Barthés --- docs/source/reference/volumes.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/source/reference/volumes.rst b/docs/source/reference/volumes.rst index 46c4fb5c..a0f6034e 100644 --- a/docs/source/reference/volumes.rst +++ b/docs/source/reference/volumes.rst @@ -8,11 +8,11 @@ Orchestrator The orchestrator claims are only linked with its database. This database (and the underlying volume) is important has it is where we store all the events that happen in the network. -+--------------+--------------+----------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+---------+ -| PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | Comment | -+==============+==============+==========+=================================+==========================+===============+================+===============+==============================================+=========+ -| Postgres PVC | orchestrator | postgres | ReadWriteOnce (can be modified) | 8 | | default | yes | postgresql.primary.persistence.existingClaim | | -+--------------+--------------+----------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+---------+ ++--------------+--------------+----------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+ +| PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | ++==============+==============+==========+=================================+==========================+===============+================+===============+==============================================+ +| Postgres PVC | orchestrator | postgres | ReadWriteOnce (can be modified) | 8 | | default | yes | postgresql.primary.persistence.existingClaim | ++--------------+--------------+----------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+ Backend @@ -25,17 +25,17 @@ The backend is the more compolex component and requires different volumes for fu +=====================+===========+============+=================================+==========================+===============+================+===============+==============================================+====================================================+ | Postgres PVC | backend | postgres | ReadWriteOnce (can be modified) | 8 | | default | yes | postgresql.primary.persistence.existingClaim | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ -| localstack PVC | backend | localstack | ReadWriteOnce (can be modified) | 5 | | default | yes | localstack.persistence.existingClaim | Only created when `localstack.enabled = true` | +| Localstack PVC | backend | localstack | ReadWriteOnce (can be modified) | 5 | | default | yes | localstack.persistence.existingClaim | Only created when `localstack.enabled = true` | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ -| minio PVC | backend | minio | ReadWriteOnce (can be modified) | 8 | | default | yes | minio.persistence.existingClaim | Only created when `minio.enabled = true` | +| Minio PVC | backend | minio | ReadWriteOnce (can be modified) | 8 | | default | yes | minio.persistence.existingClaim | Only created when `minio.enabled = true` | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ | Redis PVC | backend | redis | ReadWriteOnce (can be modified) | 8 | | default | yes | redis.master.persistence.existingClaim | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ | Kaniko cache warmer | backend | builder | ReadWriteOnce | 10 | | default | No | | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ -| builder PVC | backend | builder | ReadWriteOnce | 10 | | default | No | | | +| Builder PVC | backend | builder | ReadWriteOnce | 10 | | default | No | | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ -| worker subtuple | backend | worker | ReadWriteOnce (can be modified) | 10 | | default | No | | | +| Worker subtuple | backend | worker | ReadWriteOnce (can be modified) | 10 | | default | No | | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ | Docker registry | backend | registry | ReadWriteOnce (can be modified) | 10 | | default | yes | docker-registry.persistence.existingClaim | Only created when `docker-registry.enabled = true` | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ From 792b93163a9ee539ce4f49a93eafc0a2f11989d2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guilhem=20Barth=C3=A9s?= Date: Tue, 9 Jul 2024 11:46:45 +0200 Subject: [PATCH 6/7] fix: change description volumes in substra backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Guilhem Barthés --- docs/source/reference/volumes.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/reference/volumes.rst b/docs/source/reference/volumes.rst index a0f6034e..ba3c6de7 100644 --- a/docs/source/reference/volumes.rst +++ b/docs/source/reference/volumes.rst @@ -18,7 +18,7 @@ The orchestrator claims are only linked with its database. This database (and th Backend ******* -The backend is the more compolex component and requires different volumes for functioning. The ones that requires to be kept are re-usable and can be created outside of the deployment of the Substra stack. The other ones act as cache, and there is no way for now to re-use existing volumes using the Helm chart. +The backend is the most complex component and it requires different volumes for functioning. Volumes that should be persisted on the long term can be created outside of the deployment of the Substra stack. It is not currently possible to re-use existing volumes for the other ones (acting as cache). +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ | PVC name | Component | Pod | access mode | Volume Size default (Gi) | storage class | reclaim policy | Can be reused | How to re use | Comment | From 2ab5c1d9142ba1ed2989a5ee1b4a650ab1d906c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Guilhem=20Barth=C3=A9s?= Date: Tue, 9 Jul 2024 14:05:22 +0200 Subject: [PATCH 7/7] fix: capitalization MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Guilhem Barthés --- docs/source/reference/volumes.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/reference/volumes.rst b/docs/source/reference/volumes.rst index ba3c6de7..53692a45 100644 --- a/docs/source/reference/volumes.rst +++ b/docs/source/reference/volumes.rst @@ -31,11 +31,11 @@ The backend is the most complex component and it requires different volumes for +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ | Redis PVC | backend | redis | ReadWriteOnce (can be modified) | 8 | | default | yes | redis.master.persistence.existingClaim | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ -| Kaniko cache warmer | backend | builder | ReadWriteOnce | 10 | | default | No | | | +| Kaniko cache warmer | backend | builder | ReadWriteOnce | 10 | | default | no | | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ -| Builder PVC | backend | builder | ReadWriteOnce | 10 | | default | No | | | +| Builder PVC | backend | builder | ReadWriteOnce | 10 | | default | no | | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ -| Worker subtuple | backend | worker | ReadWriteOnce (can be modified) | 10 | | default | No | | | +| Worker subtuple | backend | worker | ReadWriteOnce (can be modified) | 10 | | default | no | | | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+ | Docker registry | backend | registry | ReadWriteOnce (can be modified) | 10 | | default | yes | docker-registry.persistence.existingClaim | Only created when `docker-registry.enabled = true` | +---------------------+-----------+------------+---------------------------------+--------------------------+---------------+----------------+---------------+----------------------------------------------+----------------------------------------------------+