You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As discussed in technical design for #1218, we want to add functionality to allow users to store their experiment tracking related data in a shared storage space.
This requires creating a mechanism where only data is shared and Kedro-Viz is still running locally but has access to the shared data service - An S3 bucket for example, which can provide the new data.
We'll need to investigate how experiment tracking data can be stored and retrieved efficiently in remote storage
The solution should work with different storage types, at least with: S3 bucket, Azure blob, Google storage bucket
Possible solution
Store data in SQLite (just like it is now) and store that database file in the remote storage.
We'd have to make sure multiple users can read and write to this file. It's okay if data is eventually consistent.
merelcht
changed the title
Investigate how to efficiently store Experiment Tracking in a bucket
Investigate how to efficiently store Experiment Tracking remotely
Jan 16, 2023
This is an outcome ticket from the Technical Design session of Allow users to write experiments to a remote server kedro-org/kedro-viz#1218
Description
As discussed in technical design for #1218, we want to add functionality to allow users to store their experiment tracking related data in a shared storage space.
This requires creating a mechanism where only data is shared and Kedro-Viz is still running locally but has access to the shared data service - An S3 bucket for example, which can provide the new data.
Possible solution
Store data in SQLite (just like it is now) and store that database file in the remote storage.
We'd have to make sure multiple users can read and write to this file. It's okay if data is eventually consistent.
Subtask
The text was updated successfully, but these errors were encountered: