Rework aggregate script, fixes #2495 #2507
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
abhidg
force-pushed
the
2495-aggregates
branch
from
7183f4f
to
54ede32
Compare
jim-sheldon
reviewed
Feb 10, 2022
jim-sheldon
reviewed
Feb 10, 2022
jim-sheldon
reviewed
Feb 10, 2022
jim-sheldon
reviewed
Feb 10, 2022
jim-sheldon
reviewed
Feb 10, 2022
* Make script modular by splitting transformation, upload, and aggregation pipeline steps. * Simplify merge with JHU data by translating country names to country codes and using country codes instead of mapping different naming conventions for countries. * Drop count_documents() calculation as that can be obtained from the casecount aggregation * Drop requests dependency as pandas.read_csv() can fetch from URLs * Add pytest as dev dependency for CI * Use local geocoding dictionary instead of fetching from GitHub * Add unit tests This fixes #2495, which was caused by a mismatch between country names. This led to missing lat/lon information which was then ignored by the map. Also fixes a bug where country values were set incorrectly in regional/latest.json.
abhidg
force-pushed
the
2495-aggregates
branch
from
54ede32
to
f552892
Compare
jim-sheldon
approved these changes
Feb 10, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
aggregation pipeline steps.
country codes and using them instead of mapping different naming
conventions.
the casecount aggregation
This fixes #2495, which was caused by a mismatch between country
names which led to missing lat/lon information which was then
ignored by the map. Also fixes a bug where country values were set
incorrectly in regional/latest.json.