Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement multi-file data retrieval #45

Open
gareth-j opened this issue Sep 1, 2023 · 0 comments · May be fixed by #62
Open

Implement multi-file data retrieval #45

gareth-j opened this issue Sep 1, 2023 · 0 comments · May be fixed by #62
Assignees
Labels
enhancement New feature or request

Comments

@gareth-j
Copy link
Contributor

gareth-j commented Sep 1, 2023

As we'll have larger amounts of data with the newer dashboards we could do with being able to retrieve data depending on what the user wants to look at. Having run some tests it looks like writing the data out to a separate file for each ObsData object returned by OpenGHG works quite well. These are standard JSON files that are accompanied by a separate JSON file that provides a lookup table for them and all the metadata required to build the interface for the site.

NOTE: I'll use the term source here to mean an inlet at a site for a specific species

My idea is to:

  • build the interface from the exported metadata file (containing filenames and metadata)
  • implement dynamic retrieval of data based on source selection on the dashboard
  • retrieve some data on first load so an initial plot is created
  • retrieve the data each time they click on a new source
  • look at gzip compression, this seems to perform favourably when compared with parquet, reducing the data to around 1/4 of the size of the exported JSON.
@gareth-j gareth-j added the enhancement New feature or request label Sep 1, 2023
@gareth-j gareth-j self-assigned this Sep 1, 2023
@gareth-j gareth-j linked a pull request Sep 5, 2023 that will close this issue
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant