feat: add analytics example #311

ThibaultFy · 2023-05-17T10:13:40Z

Context and user need:

Documentation example on how to create a simple Analytics using the ComputePlanBuilder class.

See https://www.notion.so/owkin-fdn/Federated-Analytics-in-SubstraFL-5685c4f2549748a89a3ad2e314cde886?pvs=4 for more info, especially the Mean Analytics example.

Functional spec:

Write the example in the SubstraFL "get started" section.
Mirror of the diabetes substra example

closes FL-631

linear · 2023-06-15T15:32:10Z

FL-631 [Flex API] - Documentation on how to write FA

Context and user need:
Documentation example on how to create a simple Analytics using the ComputePlanBuilder class.

See https://www.notion.so/owkin-fdn/Federated-Analytics-in-SubstraFL-5685c4f2549748a89a3ad2e314cde886?pvs=4 for more info, especially the Mean Analytics example.

Functional spec:
Write the example in the SubstraFL "get started" section.
Mirror of the diabetes substra example

Technical spec:

Acceptance criteria:

RomainGoussault

Thanks for the PR, this is great for better understanding SubstraFL.
I will come back have another read later on.

Also I think I am not sure if we should keep it in the get started section as it's still quite complex.

RomainGoussault · 2023-06-15T17:05:12Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+===========================================
+Federated Analytics on the diabetes dataset
+===========================================


I would add a few sentences explaining that is the same example as the Substra (no FL) one.

RomainGoussault · 2023-06-15T17:06:28Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+# Registering data samples and dataset
+# ------------------------------------
+#
+# Every asset will be created in respect to predefined schemas (Spec) previously imported from


Suggested change

# Every asset will be created in respect to predefined schemas (Spec) previously imported from

# Every asset will be created in respect to predefined specifications previously imported from

maybe not, I am confused, what is the difference between Schema and Specs?

In my mind it's quite the same...

RomainGoussault · 2023-06-15T17:09:58Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+# SubstraFL provides different type of Nodes to ingest these data, a Node being an object that will create an link
+# the different tasks with each other.


I have trouble understanding this part, but more generally I believe this outlines the half broken concept of node.

I must admit, I had to re-read twice (admittedly, it's Friday and I got up waaay too early), and I contributed to the definition of those concepts 🙃

It's just a suggestion but I think I would do the following:

I would start a new section at this point, after the data registration (which should be pretty straightforward for anyone familiar with Susbtra); this is the important part of the tutorial, it should be easy to refer to it; this section will include the discussion of Nodes and of ComputePlanBuilder, maybe with two subsections.

Move part of the "theoretical" explanation about compute graphs at the top of the section, to give some contexts why we are caring about nodes. Something like "a Federated computation can be represented as a graph of tasks, some of them executing on the data nodes (training tasks) and others aggregating local results (aggregation tasks). Substra does not store an explicit definition of this graph; instead, it gives the user full flexibility to define the computation graph they need, by linking a task to its parents (tasks that needs to be executed before this one, for example because it needs the outputs of the previous tasks). "
And then proceed with the discussion of the type of Nodes, and after of the ComputePlanBuilder.

(I hope this is not too confusing, we can discuss it live if you want. Basically the idea is to first explain why it's important to set up the nodes, and how they relate with what the user really wants to do, that is building a Compute Graph. Then present the technical details. )

RomainGoussault · 2023-06-15T17:10:43Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+# A third type of node exists in SubstraFL, called :ref:`Test data node<substrafl_doc/api/nodes:TestDataNode>`.
+# We will not need it in the current example. See the `MNIST Example
+# <https://docs.substra.org/en/stable/substrafl_doc/examples/get_started/run_mnist_torch.html#sphx-glr-substrafl-doc-examples-get-started-run-mnist-torch-py>`__
+# to learn how to use this type of Nodes.


Let's not mention it then?

RomainGoussault · 2023-06-15T17:13:42Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+#
+# The ``update_state`` method outputs the new state of the node, that can be passed as an argument to a following one.
+# This succession of ``next_state`` pass to new ``node.update_state`` is how Substra create the graph of the
+# ``ComputePlan``.


Let's be consistent with either ComputePlan or .ComputePlan.
And if we do use ComputePlan then I would use Node as well

I'm not sure to understand what you meant :/

SdgJlbl

Thanks a lot for the PR
Committing comments before I loose them to my flaky internet connection 😅

substrafl_examples/get_started/run_diabetes_substrafl.py

SdgJlbl · 2023-06-16T06:13:36Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+# SubstraFL provides different type of Nodes to ingest these data, a Node being an object that will create an link
+# the different tasks with each other.


I must admit, I had to re-read twice (admittedly, it's Friday and I got up waaay too early), and I contributed to the definition of those concepts 🙃

It's just a suggestion but I think I would do the following:

I would start a new section at this point, after the data registration (which should be pretty straightforward for anyone familiar with Susbtra); this is the important part of the tutorial, it should be easy to refer to it; this section will include the discussion of Nodes and of ComputePlanBuilder, maybe with two subsections.

Move part of the "theoretical" explanation about compute graphs at the top of the section, to give some contexts why we are caring about nodes. Something like "a Federated computation can be represented as a graph of tasks, some of them executing on the data nodes (training tasks) and others aggregating local results (aggregation tasks). Substra does not store an explicit definition of this graph; instead, it gives the user full flexibility to define the computation graph they need, by linking a task to its parents (tasks that needs to be executed before this one, for example because it needs the outputs of the previous tasks). "
And then proceed with the discussion of the type of Nodes, and after of the ComputePlanBuilder.

(I hope this is not too confusing, we can discuss it live if you want. Basically the idea is to first explain why it's important to set up the nodes, and how they relate with what the user really wants to do, that is building a Compute Graph. Then present the technical details. )

SdgJlbl · 2023-06-16T06:21:06Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+# Substra. Using the different nodes we created, we will update there states by applying user defined methods,
+# called ``RemoteMethod`` or ``RemoteDataMethod``, created using simply decorators, such as ``@remote`` or ``@remote_data``.


Maybe it's just me, but I will present first what we want each method to do. And then, after the code, add a note about "oh, look, there are some decorators @remote and @remote_data on some of the functions, it's very important to use them. This is because these functions are not going to be executed locally, but on a distant node, and they either need access to data or not. It's a bit of magic Substra provides to help the user."

Thanks for the good advice. I tried a new configuration, let me know what it looks like :)

SdgJlbl · 2023-06-16T06:21:48Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+# The ``load_local_state`` and ``save_local_state`` are two methods used at each new iteration on a Node, in order to
+# retrieve a the previous local state that have not been shared with the other nodes.
+# For instance, after updating a :ref:`Train data node<substrafl_doc/api/nodes:TrainDataNode>` using its
+# ``update_state`` method, we will have access to its next local state, that we will pass as argument to the
+# next ``update_state`` we will apply on this :ref:`Train data node<substrafl_doc/api/nodes:TrainDataNode>`.


substrafl_examples/get_started/run_diabetes_substrafl.py

SdgJlbl · 2023-06-16T06:41:22Z

substrafl_examples/get_started/run_diabetes_substrafl.py

+#
+# As a last step before launching our experiment, we need to specify the third parties dependencies required to run it.
+# The :ref:`substrafl_doc/api/dependency:Dependency` object is instantiated in order to install the right libraries in
+# the Python environment of each organization.


Shall the Dependency object definition be moved to its own cell here?

If we move the example to "go further", I would say no

substrafl_examples/get_started/run_diabetes_substrafl.py

ThibaultFy · 2023-06-16T07:40:21Z

Thanks for the PR, this is great for better understanding SubstraFL. I will come back have another read later on.

Also I think I am not sure if we should keep it in the get started section as it's still quite complex.

I changed the example to the go_further section. Should I remove some of the explanation then ?
On an other subject, do we keep the exact same image as the diabetes example ?

SdgJlbl · 2023-06-16T12:05:46Z

I changed the example to the go_further section. Should I remove some of the explanation then ?

Maybe a few small things, but most of it is not too cumbersome and I think it's important that tutorials remain relatively self-contained and can be followed in any order.

RomainGoussault · 2023-06-20T16:11:12Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+    clean_models=False,
+)


Suggested change

clean_models=False,

)

clean_models=False,

name="Federated Analytics with SubstraFL documentation example",

)

SdgJlbl

Thanks a lot for the PR.
A few comments, but mostly typos. Well done 🚀

SdgJlbl · 2023-06-26T15:32:59Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+# To be able to create this graph of computation, SubstraFL provides the ``Node`` abstraction. A ``Node`` is used to
+# attached an organization (aka a Client) to a certain type of task depending on the computation we expect to happen on
+# this organization.


Suggested change

# To be able to create this graph of computation, SubstraFL provides the ``Node`` abstraction. A ``Node`` is used to

# attached an organization (aka a Client) to a certain type of task depending on the computation we expect to happen on

# this organization.

# To create this graph of computations, SubstraFL provides the ``Node`` abstraction. A ``Node``

# assigns to an organization (aka a Client) tasks of a given type. The type of the ``Node`` depends on the type of tasks

# we want to run on this organization (training or aggregation tasks).

I find Sarah version easier to understand

SdgJlbl · 2023-06-26T15:34:32Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+
+aggregation_node = AggregationNode(ANALYTICS_PROVIDER_ORG_ID)
+
+# Create the Train Data Nodes (or training tasks) and save them in a list


I think we can remove this comment, and instead add in the explanation (l 187) that we need to create a Train Node for each organization which has data.

SdgJlbl · 2023-06-26T15:35:17Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+# - ``save_local_state(...)``
+#
+# The ``build_compute_plan`` method is essential to create the graph of the compute plan that will be executed on
+# Substra. Using the different ``Nodes`` we created, we will update there states by applying user defined methods.


Suggested change

# Substra. Using the different ``Nodes`` we created, we will update there states by applying user defined methods.

# Substra. Using the different ``Nodes`` we created, we will update their states by applying user defined methods.

It doesn't look resolved

SdgJlbl · 2023-06-26T15:35:46Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+
+    @remote_data
+    def local_first_order_computation(self, datasamples: pd.DataFrame, shared_state=None):
+        """Compute from the data samples, expected to be pandas dataframe,


Suggested change

"""Compute from the data samples, expected to be pandas dataframe,

"""Compute from the data samples, expected to be a pandas dataframe,

SdgJlbl · 2023-06-26T15:36:42Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+        evaluation_strategy=None,
+        clean_models=False,
+    ):
+        """Method to build and link the different operation to execute with each other.


Suggested change

"""Method to build and link the different operation to execute with each other.

"""Method to build and link the different operations to execute with each other.

or maybe even "computations"?

substrafl_examples/go_further/run_diabetes_substrafl.py

SdgJlbl · 2023-06-26T15:39:47Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+# compute plan.
+#
+# The ``load_local_state`` and ``save_local_state`` are two methods used at each new iteration on a Node, in order to
+# retrieve a the previous local state that have not been shared with the other ``Nodes``.


a / the ⚔️

Sorry I don't get the comment 😶

sorry I wasn't very clear. It's written retrieve a the previous local state, and I think one of the two articles is enough :)

Aaaaaah ahah :D thanks !!

substrafl_examples/go_further/run_diabetes_substrafl.py

ThibaultFy · 2023-06-27T07:08:23Z

@RomainGoussault should we keep the same image as the Substra example ?

RomainGoussault · 2023-06-27T07:33:14Z

@RomainGoussault should we keep the same image as the Substra example ?

yes

guilhem-barthes

Thanks for your work 🙏 However, some of the conversations have been marked as resolved although the point didn't seem to be actually resolved.

guilhem-barthes · 2023-06-27T14:46:13Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+The goal of this example is to compute some analytics such as Age mean, Blood pressure standard deviation or Sex percentage.
+
+We simulate having two different data organizations, and a third organization which wants to compute aggregated analytics
+without having access to the raw data. The example here runs everything locally; however there is only one parameter to


Should we mention which one?

guilhem-barthes · 2023-06-27T14:52:29Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+    logs_permission=permissions_dataset,
+)
+
+# We register the dataset for each of the organizations


nitpicking

Suggested change

# We register the dataset for each of the organizations

# We register the dataset for each organization

substrafl_examples/go_further/run_diabetes_substrafl.py

guilhem-barthes · 2023-06-27T15:00:18Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+# the compute plan (or computation graph) they need, by linking a task to its parents.
+#
+# To be able to create this graph of computation, SubstraFL provides the ``Node`` abstraction. A ``Node`` is used to
+# attached an organization (aka a Client) to a certain type of task depending on the computation we expect to happen on


Suggested change

# attached an organization (aka a Client) to a certain type of task depending on the computation we expect to happen on

# attach an organization (aka a Client) to a certain type of task depending on the computation we expect to happen on

guilhem-barthes · 2023-06-27T15:02:24Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+# To be able to create this graph of computation, SubstraFL provides the ``Node`` abstraction. A ``Node`` is used to
+# attached an organization (aka a Client) to a certain type of task depending on the computation we expect to happen on
+# this organization.


I find Sarah version easier to understand

guilhem-barthes · 2023-06-27T15:16:27Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+    def aggregation(self, shared_states: List[Dict]):
+        """Aggregation function that receive a list on locally computed analytics in order to
+        aggregate them.
+        The aggregation will be a weighted average using "n_samples" as weighted coefficient.


Suggested change

The aggregation will be a weighted average using "n_samples" as weighted coefficient.

The aggregation will be a weighted average using "n_samples" as weight coefficient.

substrafl_examples/go_further/run_diabetes_substrafl.py

guilhem-barthes · 2023-06-27T15:18:54Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+            train_data_nodes (TrainDataNode): Nodes linked to the data samples on which
+                to compute analytics.


Suggested change

train_data_nodes (TrainDataNode): Nodes linked to the data samples on which

to compute analytics.

train_data_nodes (List[TrainDataNode]): Nodes linked to the data samples on which

to compute analytics.

guilhem-barthes · 2023-06-27T15:19:42Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+                to compute analytics.
+            aggregation_node (AggregationNode): Node on which to compute the aggregation
+                of the analytics extracted from the train_data_nodes.
+            num_rounds (int): Num rounds to be used to iterate on recurrent part of


I think this is Optional as it defaults to None

guilhem-barthes · 2023-06-27T15:20:39Z

substrafl_examples/go_further/run_diabetes_substrafl.py

+
+# %%
+# Now that we saw the implementation of the custom ``Analytics`` class, we can add detailed to some of the previously
+# introduced concept.


Suggested change

# introduced concept.

# introduced concepts.

ThibaultFy · 2023-06-27T15:56:45Z

Thanks for your work 🙏 However, some of the conversations have been marked as resolved although the point didn't seem to be actually resolved.

Wouah thanks a lot for catching this... I must have completely messed up the reset to sign up the automated commit from suggestions... And thanks a lot for your comments !

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

ThibaultFy force-pushed the substrafl-analytics-example branch 2 times, most recently from 86c72f1 to 9d37d58 Compare May 22, 2023 13:19

ThibaultFy force-pushed the substrafl-analytics-example branch from 9d37d58 to dcc4b55 Compare June 8, 2023 08:45

ThibaultFy force-pushed the substrafl-analytics-example branch from 5db7ad6 to f6cfb7a Compare June 15, 2023 12:31

ThibaultFy marked this pull request as ready for review June 15, 2023 15:13

ThibaultFy requested review from a team and RomainGoussault as code owners June 15, 2023 15:13

RomainGoussault reviewed Jun 15, 2023

View reviewed changes

SdgJlbl reviewed Jun 16, 2023

View reviewed changes

ThibaultFy force-pushed the substrafl-analytics-example branch from b10abea to 6a75c22 Compare June 16, 2023 06:54

ThibaultFy requested review from RomainGoussault and SdgJlbl June 16, 2023 07:50

RomainGoussault reviewed Jun 20, 2023

View reviewed changes

ThibaultFy requested a review from RomainGoussault June 21, 2023 12:26

SdgJlbl approved these changes Jun 26, 2023

View reviewed changes

ThibaultFy force-pushed the substrafl-analytics-example branch from a51674b to 8c3f032 Compare June 26, 2023 15:50

guilhem-barthes reviewed Jun 27, 2023

View reviewed changes

oleobal and others added 7 commits June 30, 2023 14:54

feat: add analytics example

0136224

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

feat: add compute plan builder

5150800

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

update example

9cc52ee

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

update example

bd6dc7b

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

update example

f18854b

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

update example

c7ea25e

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

update example

97c707f

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

ThibaultFy added 27 commits June 30, 2023 14:54

add comment

6d3d711

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

rm num round from metadata

f4cea33

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

rm num round from metadata

76402b3

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

type fail

3f59175

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

type fail

9810762

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

type fail

75f407d

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

update example

6accab3

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

update branch

9ade74a

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

update branch

cadab6b

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

correct rst

f4592dc

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

correct rst

912712d

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

correct rst

47129d8

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

correct rst

232cd30

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

correct rst

10361f7

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

correct rst

8386b97

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

correct rst

36b03e1

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

correct rst

193beb3

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

sarah review

4854a0b

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

moove to go further

fd1ef00

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

docstring

13dd8eb

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

test

b151e4f

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

text

732cabc

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

text

1922f69

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

cp name

e0c0107

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

sarah review

100cf41

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

docstring

e87262f

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

guilhem review + sarah review

8329154

Signed-off-by: ThibaultFy <50656860+ThibaultFy@users.noreply.github.com>

ThibaultFy force-pushed the substrafl-analytics-example branch from 968e7bf to 8329154 Compare June 30, 2023 12:55

ThibaultFy merged commit b2733ae into main Jun 30, 2023

ThibaultFy deleted the substrafl-analytics-example branch June 30, 2023 13:10

	# Every asset will be created in respect to predefined schemas (Spec) previously imported from
	# Every asset will be created in respect to predefined specifications previously imported from

		# SubstraFL provides different type of Nodes to ingest these data, a Node being an object that will create an link
		# the different tasks with each other.

		# Substra. Using the different nodes we created, we will update there states by applying user defined methods,
		# called ``RemoteMethod`` or ``RemoteDataMethod``, created using simply decorators, such as ``@remote`` or ``@remote_data``.


		aggregation_node = AggregationNode(ANALYTICS_PROVIDER_ORG_ID)

		# Create the Train Data Nodes (or training tasks) and save them in a list

	# Substra. Using the different ``Nodes`` we created, we will update there states by applying user defined methods.
	# Substra. Using the different ``Nodes`` we created, we will update their states by applying user defined methods.

	"""Compute from the data samples, expected to be pandas dataframe,
	"""Compute from the data samples, expected to be a pandas dataframe,

	"""Method to build and link the different operation to execute with each other.
	"""Method to build and link the different operations to execute with each other.

	# We register the dataset for each of the organizations
	# We register the dataset for each organization

	# attached an organization (aka a Client) to a certain type of task depending on the computation we expect to happen on
	# attach an organization (aka a Client) to a certain type of task depending on the computation we expect to happen on

	The aggregation will be a weighted average using "n_samples" as weighted coefficient.
	The aggregation will be a weighted average using "n_samples" as weight coefficient.

		train_data_nodes (TrainDataNode): Nodes linked to the data samples on which
		to compute analytics.

feat: add analytics example #311

feat: add analytics example #311

Conversation

ThibaultFy commented May 17, 2023 • edited Loading

Context and user need:

Functional spec:

linear bot commented Jun 15, 2023

RomainGoussault left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SdgJlbl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThibaultFy commented Jun 16, 2023

SdgJlbl commented Jun 16, 2023

Choose a reason for hiding this comment

SdgJlbl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThibaultFy commented Jun 27, 2023

RomainGoussault commented Jun 27, 2023

guilhem-barthes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThibaultFy commented Jun 27, 2023

ThibaultFy commented May 17, 2023 •

edited

Loading