Skip to content

Commit

Permalink
Rename the aggregate script to aggregate-covid19 #2712
Browse files Browse the repository at this point in the history
  • Loading branch information
iamleeg committed Jun 14, 2022
1 parent 6ab72f6 commit d52cced
Show file tree
Hide file tree
Showing 12 changed files with 12 additions and 8 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/aggregate-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,19 @@ on:
branches: [main, '*-stable']
paths:
- '.github/workflows/aggregate-ci.yml'
- 'data-serving/scripts/aggregate/aggregate/**'
- 'data-serving/scripts/aggregate-covid19/aggregate/**'
pull_request:
paths:
- '.github/workflows/aggregate-ci.yml'
- 'data-serving/scripts/aggregate/aggregate/**'
- 'data-serving/scripts/aggregate-covid19/aggregate/**'
workflow_dispatch:

jobs:
ci:
runs-on: ubuntu-20.04
defaults:
run:
working-directory: data-serving/scripts/aggregate/aggregate
working-directory: data-serving/scripts/aggregate-covid19/aggregate
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.9
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/aggregate-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ on:
branches: [main]
paths:
- '.github/workflows/aggregate-deploy.yml'
- 'data-serving/scripts/aggregate/**'
- '!data-serving/scripts/aggregate/README.md'
- 'data-serving/scripts/aggregate-covid19/**'
- '!data-serving/scripts/aggregate-covid19/README.md'
workflow_dispatch:

jobs:
Expand All @@ -26,7 +26,7 @@ jobs:
uses: aws-actions/amazon-ecr-login@v1

- name: Build, tag, and push image to Amazon ECR (latest)
working-directory: data-serving/scripts/aggregate/aggregate
working-directory: data-serving/scripts/aggregate-covid19/aggregate
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPO: gdh-map-aggregator
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,13 @@
# Aggregating data

This is a AWS batch script built to aggregate data that is used by Map.
This is a AWS batch script built to aggregate data that is used by COVID-19 Map.
Previously Lambda functions were used for data export, that is now handled
by AWS Batch in [../export-data](../export-data/README.md)

The Batch job (actually two; one for the [dev map](https://dev-map.covid-19.global.health) and one for [prod](https://map.covid-19.global.health)) pulls the code from an image on the Amazon container repo. That image is prepared from the `main` branch by a github action.

**Note for Country level**: Mapbox requires that the lat/long coordinates remain static in order to return the shapes for the choropleth, so this is standardized using the [Google Canonical DSPL countries.csv](https://developers.google.com/public-data/docs/canonical/countries_csv) dataset.

### COVID-19 specific aggregation

This script is only appropriate for aggregating COVID-19 data, as it merges in information from the John Hopkins University to report on completeness.
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ def transform_counts(
counts: list[dict[str, Any]], jhu_counts: dict[str, int] = None
) -> list[dict[str, Any]]:
"""
Transforms aggregated casecount data for map
Transforms aggregated COVID-19 casecount data for map
counts -- Case counts as a list of records of the form
[{"_id": "two-letter country code", "casecount": N}, ... ]
Expand Down

0 comments on commit d52cced

Please sign in to comment.