Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement reusable data service #2714

Open
iamleeg opened this issue Jun 1, 2022 · 3 comments
Open

Implement reusable data service #2714

iamleeg opened this issue Jun 1, 2022 · 3 comments
Assignees
Labels
P0: Critical path blocker Blocking testing or the launch critical path, drop everything to investigate & fix turnkey

Comments

@iamleeg
Copy link
Contributor

iamleeg commented Jun 1, 2022

See Moritz's day zero schema on globaldothealth/monkeypox#21 (comment)

@iamleeg iamleeg added this to the Oberon milestone Jun 1, 2022
@iamleeg
Copy link
Contributor Author

iamleeg commented Jun 14, 2022

Moritz and the curation team are reviewing this schema. Based on discussion including @abhidg and leads/curators yesterday, I think we want to treat the covid-19 collection and schema as its own thing and start from scratch with the day-zero schema (maybe just renaming the database from covid19 to $newdisease) in new instances, then configure them to support additional fields as the pandemic develops.

@iamleeg iamleeg self-assigned this Jun 16, 2022
@iamleeg
Copy link
Contributor Author

iamleeg commented Jun 17, 2022

Here's the plan @abhidg and I discussed yesterday.

#2714 architecture

So the curator API has a configurable connection to a data service (which it already does), and rather than having a strict OpenAPI validator for cases accepts any object as a case and lets the data service validate its input and report errors: the curator API is responsible for user management, data source/ingestion management and access control.

Existing UI has a lot of connection to new schema, but we can re-use automated source configuration, user profiles/management, overall structure so add new case list/forms for the new flexible schema and choose which to use per deployment.

Geocoding API remains unchanged.

The data API is completely replaced: the existing one has multiple constraints coupling it to the COVID-19 schema, so we'll have a new service for new diseases which encodes the day 0 schema plus custom fields. The covid-19 instances of G.h use the existing data service, and hMPXV/emerging outbreak instances use the new one instead.
New data service can be based on Python, and should not use a schema-defining library like Mongoose so that validation can be done in code to take into account custom fields defined by curators. It should use standard representations of all fields where possible, e.g. ISO country codes, ISO date formats, GeoJSON for locations. OpenAPI documentation should be derived from the code and not a separate, redundant statement of the data schema. Consider whether it is possible/desirable to switch database to postgres or appropriate to continue using mongoDB.

iamleeg added a commit that referenced this issue Jun 17, 2022
iamleeg added a commit that referenced this issue Jun 17, 2022
iamleeg added a commit that referenced this issue Jun 17, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 20, 2022
iamleeg added a commit that referenced this issue Jun 21, 2022
…rialise such dates #2714

I chose not to handle this in the mongo_store because it would have been (cognitively and computationally) inefficient: it would mean dumping the BSON, loading the JSON, patching it, dumping that, then loading in the model. Nein, danke!
iamleeg added a commit that referenced this issue Jul 27, 2022
iamleeg added a commit that referenced this issue Jul 27, 2022
iamleeg added a commit that referenced this issue Jul 27, 2022
iamleeg added a commit that referenced this issue Jul 27, 2022
The free text is stored in an associated string field
iamleeg added a commit that referenced this issue Jul 27, 2022
iamleeg added a commit that referenced this issue Jul 27, 2022
iamleeg added a commit that referenced this issue Jul 28, 2022
The free text is stored in an associated string field
iamleeg added a commit that referenced this issue Jul 28, 2022
iamleeg added a commit that referenced this issue Jul 28, 2022
iamleeg added a commit that referenced this issue Jul 28, 2022
@ksewalk ksewalk changed the title Configure form/list for monkeypox data schema Configure form/list for pathogen data schema Sep 14, 2022
abhidg pushed a commit that referenced this issue Sep 19, 2022
@abhidg abhidg changed the title Configure form/list for pathogen data schema Implement reusable data service Oct 8, 2022
@abhidg abhidg removed the Monkeypox label Oct 8, 2022
@abhidg abhidg removed this from the Oberon milestone Oct 8, 2022
@jim-sheldon
Copy link
Collaborator

Work on the curator UI captured in this issue: #2902

@abhidg abhidg added the turnkey label Oct 11, 2022
@jim-sheldon jim-sheldon removed their assignment Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P0: Critical path blocker Blocking testing or the launch critical path, drop everything to investigate & fix turnkey
Projects
None yet
Development

No branches or pull requests

3 participants