Build collaborative Experiment Tracking feature

[rashidakanchwala]

Description:

This ticket aims to develop a collaborative experiment tracking feature in Kedro-viz. This feature will allow users to save their experiments to a shared cloud storage and view and compare other users' experiments.

Also see user research ticket - #1218

With collaborative experiment tracking enabled on Kedro-viz, users will be able to:

• Specify a shared cloud storage location to save experiment metadata
• Access other users' experiments stored in the same shared cloud storage
• Modify their own experiment by changing titles and adding notes. However, this feature will only be available in their own experiment tracking view. In the first release, users will not be able to view modifications made by other users on different experiments.

Implementation:

• We will use fsspec to upload each user's session_store.db to the cloud storage, assigning a unique name for each user's database based on their computer user name and computer name.
• When Kedro-viz is running, we will read all the databases from the cloud storage and add the experiment data to the user's experiment data table, allowing them to view not only their own experiments but also those of other users.
• We will implement auto-reload functionality that updates Experiment Tracking when new experiments are added while Kedro-viz is running.
• All modifications to titles and notes made by the user will be saved to the cloud. However, during read time, this data will not be merged with other databases, so users will only be able to view their own modifications.

Checklist

• Include labels so that we can categorise your feature request

[rashidakanchwala]

Hi team,
Last week, we scoped all the features that will go in different releases of Collaborative Experiment Tracking. Here are the details :-

Version 1

• We will enable users to read experiments of other users that are stored on s3
• Allow users to make local changes (i.e. modify name, title, bookmark) from Kedro-viz. These modified changes will not be uploaded to s3, and will remain in the local session_store.db
• We will not be doing any caching to optimise the loading of multiple session_store from cloud in v.1
• We will use fsspec but do thorough testing only on s3
• No auto-reload. So if a user wants to see new experiments that were added while they were running Kedro-viz.,they will have to restart Kedro-viz
• No credentials set up for now. AWS will connect based on credentials stored in env variables.

Version 2:

• In this version, we will enable auto-reload. So if a user is running Kedro-viz while other users do a Kedro-run, auto-reload will scan the s3 bucket every 3 seconds and add any newly added experiments on the user's Kedro-viz without them having to restart Viz

Version 2.5 :

• In this version, we will enable caching so we can optimise and load only from databases that have new experiments.

Version 3:

• We will allow users to make modifications to runs from Kedro-viz. In this version, we will upload all user modifications (change name, title etc) on cloud.
• We will also do support for other cloud storage like azure, gcs etc.

[rashidakanchwala]

@antonymilne FYI

[tynandebold]

Closing this in favor of the other tickets that we have going.

Track the progress of this effort in the milestone here.