Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advanced uploads #651

Merged
merged 2 commits into from
Aug 5, 2024
Merged

Advanced uploads #651

merged 2 commits into from
Aug 5, 2024

Conversation

chrisclark
Copy link
Collaborator

@chrisclark chrisclark commented Jul 24, 2024

  • Ability to append an upload to a previously uploaded file/sqlite DB as a new table
  • Good cache busting and detection of file changes on uploads
  • Separate the upload UI from the 'add connection' UI, as they are materially different
  • Fix a small bug with bar chart generation, when values are null
  • Ability to refresh a connection's schema and data (if it's an upload) from the connections list view

TODO

  • Increase test coverage on append (e.g. does the table get overwritten when it already exists? what happens when the append ID val in the POST request is invalid? Are there race conditions that could happen?)
  • Create some kind of check that invalidates the downloaded SQLite files. If a server has a copy of the original SQLite uploaded file, but then someone appends a table to it (on a different node/server), the first server will never go get the updated version from s3.
  • Similarly, we might need to invalidate schema data when databases have a file appended.
  • Documentation

explorer/ee/db_connections/utils.py Fixed Show fixed Hide fixed
explorer/ee/db_connections/utils.py Dismissed Show dismissed Hide dismissed
@chrisclark chrisclark force-pushed the advanced-uploads branch 2 times, most recently from f66c492 to d6c4711 Compare July 29, 2024 13:50
Separate upload UI from regular 'add connection' UI
Good cache busting and detection of file changes
Ability to force a cache refresh and file refresh of data sets
files. See the 'User uploads' section of :doc:`settings`.
files.

File Uploads
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marksweb If you have a moment, I'd love for you to give this a read and let me know what you think. Not asking for a full code review, just looking over the docs and make sure the functionality is clear to you. If there are outstanding questions, I can address them in the docs, or perhaps just by reading the docs you will think of some edge case or test I haven't thought of! I would welcome a brain-dump from you on this functionality.

@chrisclark chrisclark merged commit 841549d into master Aug 5, 2024
11 checks passed
@chrisclark chrisclark deleted the advanced-uploads branch August 5, 2024 18:55
@chrisclark chrisclark mentioned this pull request Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant