Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate how to efficiently store Experiment Tracking remotely #1220

Closed
1 task done
NeroOkwa opened this issue Jan 11, 2023 · 1 comment
Closed
1 task done

Investigate how to efficiently store Experiment Tracking remotely #1220

NeroOkwa opened this issue Jan 11, 2023 · 1 comment

Comments

@NeroOkwa
Copy link
Contributor

NeroOkwa commented Jan 11, 2023

This is an outcome ticket from the Technical Design session of Allow users to write experiments to a remote server kedro-org/kedro-viz#1218

Description

As discussed in technical design for #1218, we want to add functionality to allow users to store their experiment tracking related data in a shared storage space.

This requires creating a mechanism where only data is shared and Kedro-Viz is still running locally but has access to the shared data service - An S3 bucket for example, which can provide the new data.

  • We'll need to investigate how experiment tracking data can be stored and retrieved efficiently in remote storage
  • The solution should work with different storage types, at least with: S3 bucket, Azure blob, Google storage bucket

Possible solution

Store data in SQLite (just like it is now) and store that database file in the remote storage.

We'd have to make sure multiple users can read and write to this file. It's okay if data is eventually consistent.

Subtask

@tynandebold tynandebold transferred this issue from kedro-org/kedro Jan 16, 2023
@merelcht merelcht changed the title Investigate how to efficiently store Experiment Tracking in a bucket Investigate how to efficiently store Experiment Tracking remotely Jan 16, 2023
@tynandebold
Copy link
Member

Closing this as we now have direction, as seen in #1295.

@github-project-automation github-project-automation bot moved this from Inbox to Done in Kedro-Viz Mar 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

No branches or pull requests

2 participants