globaldothealth · iamleeg · Jun 14, 2022 · Jun 14, 2022 · Jun 14, 2022 · Jun 14, 2022
diff --git a/.github/workflows/aggregate-ci.yml b/.github/workflows/aggregate-ci.yml
@@ -5,19 +5,19 @@ on:
     branches: [main, '*-stable']
     paths:
     - '.github/workflows/aggregate-ci.yml'
-    - 'data-serving/scripts/aggregate/aggregate/**'
+    - 'data-serving/scripts/aggregate-covid19/aggregate/**'
   pull_request:
     paths:
     - '.github/workflows/aggregate-ci.yml'
-    - 'data-serving/scripts/aggregate/aggregate/**'
+    - 'data-serving/scripts/aggregate-covid19/aggregate/**'
   workflow_dispatch:
 
 jobs:
   ci:
     runs-on: ubuntu-20.04
     defaults:
       run:
-        working-directory: data-serving/scripts/aggregate/aggregate
+        working-directory: data-serving/scripts/aggregate-covid19/aggregate
     steps:
     - uses: actions/checkout@v2
     - name: Set up Python 3.9

diff --git a/.github/workflows/aggregate-deploy.yml b/.github/workflows/aggregate-deploy.yml
@@ -5,8 +5,8 @@ on:
     branches: [main]
     paths:
     - '.github/workflows/aggregate-deploy.yml'
-    - 'data-serving/scripts/aggregate/**'
-    - '!data-serving/scripts/aggregate/README.md'
+    - 'data-serving/scripts/aggregate-covid19/**'
+    - '!data-serving/scripts/aggregate-covid19/README.md'
   workflow_dispatch:
 
 jobs:
@@ -26,7 +26,7 @@ jobs:
       uses: aws-actions/amazon-ecr-login@v1
 
     - name: Build, tag, and push image to Amazon ECR (latest)
-      working-directory: data-serving/scripts/aggregate/aggregate
+      working-directory: data-serving/scripts/aggregate-covid19/aggregate
       env:
         REGISTRY: ${{ steps.login-ecr.outputs.registry }}
         REPO: gdh-map-aggregator

diff --git a/data-serving/README.md b/data-serving/README.md
@@ -4,15 +4,16 @@
 
 G.h case data, as well as the source, user, and session data for the curator portal, is stored in a MongoDB database.
 
-We have multiple instances of MongoDB, ranging from local instances for development, to dev (for
-https://dev-data.covid-19.global.health/), to prod (for https://data.covid-19.global.health/).
+We have multiple instances of MongoDB, ranging from local instances for development, to dev and qa (for
+https://dev-data.covid-19.global.health/ and https://qa-data.covid-19.global.health in the case of COVID-19 data),
+and prod (for https://data.covid-19.global.health/ for COVID-19).
 
 Each instance has a `covid19` database, which in turn has collections for each type of data, e.g. `cases`, `users`, etc.
 
 ## Case data
 
 The data in the `cases` collection is the primary data that G.h collects, verifies, and shares. Each document in the
-`cases` collection represents a single COVID-19 case.
+`cases` collection represents a single disease case.
 
 ### Shape of the data
 
@@ -33,7 +34,7 @@ indexes the id of the `case` and its revision for quick lookups.
 
 ### Importing cases
 
-G.h has millions of case records that predate the new curator portal. These are exported to a
+G.h has millions of COVID-19 case records that predate the new curator portal. These are exported to a
 [gzipped CSV](https://github.com/beoutbreakprepared/nCoV2019/tree/master/latest_data) in a
 [separate repo](https://github.com/beoutbreakprepared/nCoV2019).
 

diff --git a/data-serving/scripts/aggregate/README.md → ...rving/scripts/aggregate-covid19/README.md b/data-serving/scripts/aggregate/README.md → ...rving/scripts/aggregate-covid19/README.md
@@ -1,9 +1,13 @@
 # Aggregating data
 
-This is a AWS batch script built to aggregate data that is used by Map.
+This is a AWS batch script built to aggregate data that is used by COVID-19 Map.
 Previously Lambda functions were used for data export, that is now handled
 by AWS Batch in [../export-data](../export-data/README.md)
 
 The Batch job (actually two; one for the [dev map](https://dev-map.covid-19.global.health) and one for [prod](https://map.covid-19.global.health)) pulls the code from an image on the Amazon container repo. That image is prepared from the `main` branch by a github action.
 
 **Note for Country level**: Mapbox requires that the lat/long coordinates remain static in order to return the shapes for the choropleth, so this is standardized using the [Google Canonical DSPL countries.csv](https://developers.google.com/public-data/docs/canonical/countries_csv) dataset.
+
+### COVID-19 specific aggregation
+
+This script is only appropriate for aggregating COVID-19 data, as it merges in information from the John Hopkins University to report on completeness.
diff --git a/...ng/scripts/aggregate/aggregate/Dockerfile → ...ts/aggregate-covid19/aggregate/Dockerfile b/...ng/scripts/aggregate/aggregate/Dockerfile → ...ts/aggregate-covid19/aggregate/Dockerfile
diff --git a/...erving/scripts/aggregate/aggregate/app.py → ...cripts/aggregate-covid19/aggregate/app.py b/...erving/scripts/aggregate/aggregate/app.py → ...cripts/aggregate-covid19/aggregate/app.py
@@ -200,7 +200,7 @@ def transform_counts(
     counts: list[dict[str, Any]], jhu_counts: dict[str, int] = None
 ) -> list[dict[str, Any]]:
     """
-    Transforms aggregated casecount data for map
+    Transforms aggregated COVID-19 casecount data for map
 
     counts -- Case counts as a list of records of the form
               [{"_id": "two-letter country code", "casecount": N}, ... ]

diff --git a/...regate/aggregate/geocoding_countries.json → ...ovid19/aggregate/geocoding_countries.json b/...regate/aggregate/geocoding_countries.json → ...ovid19/aggregate/geocoding_countries.json
diff --git a/...g/scripts/aggregate/aggregate/poetry.lock → ...s/aggregate-covid19/aggregate/poetry.lock b/...g/scripts/aggregate/aggregate/poetry.lock → ...s/aggregate-covid19/aggregate/poetry.lock
diff --git a/...cripts/aggregate/aggregate/pyproject.toml → ...ggregate-covid19/aggregate/pyproject.toml b/...cripts/aggregate/aggregate/pyproject.toml → ...ggregate-covid19/aggregate/pyproject.toml
diff --git a/.../aggregate/tests/casecount-aggregate.json → .../aggregate/tests/casecount-aggregate.json b/.../aggregate/tests/casecount-aggregate.json → .../aggregate/tests/casecount-aggregate.json
diff --git a/...regate/aggregate/tests/jhu-02-02-2022.csv → ...ovid19/aggregate/tests/jhu-02-02-2022.csv b/...regate/aggregate/tests/jhu-02-02-2022.csv → ...ovid19/aggregate/tests/jhu-02-02-2022.csv
diff --git a/...te/aggregate/tests/regions-aggregate.json → ...19/aggregate/tests/regions-aggregate.json b/...te/aggregate/tests/regions-aggregate.json → ...19/aggregate/tests/regions-aggregate.json
diff --git a/...pts/aggregate/aggregate/tests/test_app.py → ...egate-covid19/aggregate/tests/test_app.py b/...pts/aggregate/aggregate/tests/test_app.py → ...egate-covid19/aggregate/tests/test_app.py
diff --git a/data-serving/scripts/convert-data/README.md → ...ng/scripts/convert-covid19-data/README.md b/data-serving/scripts/convert-data/README.md → ...ng/scripts/convert-covid19-data/README.md
@@ -1,6 +1,6 @@
 # Working with line-list data
 
-This directory contains scripts for converting, ingesting, and otherwise munging line-list data.
+This directory contains scripts for converting, ingesting, and otherwise munging COVID-19 line-list data.
 
 ## Prerequisites
 

diff --git a/...-serving/scripts/convert-data/__init__.py → .../scripts/convert-covid19-data/__init__.py b/...-serving/scripts/convert-data/__init__.py → .../scripts/convert-covid19-data/__init__.py
diff --git a/...serving/scripts/convert-data/constants.py → ...scripts/convert-covid19-data/constants.py b/...serving/scripts/convert-data/constants.py → ...scripts/convert-covid19-data/constants.py
diff --git a/...ving/scripts/convert-data/convert_data.py → ...ipts/convert-covid19-data/convert_data.py b/...ving/scripts/convert-data/convert_data.py → ...ipts/convert-covid19-data/convert_data.py
diff --git a/...erving/scripts/convert-data/converters.py → ...cripts/convert-covid19-data/converters.py b/...erving/scripts/convert-data/converters.py → ...cripts/convert-covid19-data/converters.py
diff --git a/...ving/scripts/convert-data/geocode_util.py → ...ipts/convert-covid19-data/geocode_util.py b/...ving/scripts/convert-data/geocode_util.py → ...ipts/convert-covid19-data/geocode_util.py
diff --git a/data-serving/scripts/convert-data/logger.py → ...ng/scripts/convert-covid19-data/logger.py b/data-serving/scripts/convert-data/logger.py → ...ng/scripts/convert-covid19-data/logger.py
diff --git a/...erving/scripts/convert-data/move_notes.py → ...cripts/convert-covid19-data/move_notes.py b/...erving/scripts/convert-data/move_notes.py → ...cripts/convert-covid19-data/move_notes.py
diff --git a/data-serving/scripts/convert-data/parsers.py → ...g/scripts/convert-covid19-data/parsers.py b/data-serving/scripts/convert-data/parsers.py → ...g/scripts/convert-covid19-data/parsers.py
diff --git a/...ing/scripts/convert-data/requirements.txt → ...pts/convert-covid19-data/requirements.txt b/...ing/scripts/convert-data/requirements.txt → ...pts/convert-covid19-data/requirements.txt
diff --git a/...ng/scripts/convert-data/tests/__init__.py → ...ts/convert-covid19-data/tests/__init__.py b/...ng/scripts/convert-data/tests/__init__.py → ...ts/convert-covid19-data/tests/__init__.py
diff --git a/...s/convert-data/tests/test_geocode_util.py → ...t-covid19-data/tests/test_geocode_util.py b/...s/convert-data/tests/test_geocode_util.py → ...t-covid19-data/tests/test_geocode_util.py
diff --git a/data-serving/scripts/convert-data/utils.py → ...ing/scripts/convert-covid19-data/utils.py b/data-serving/scripts/convert-data/utils.py → ...ing/scripts/convert-covid19-data/utils.py
diff --git a/data-serving/scripts/export-data/README.md b/data-serving/scripts/export-data/README.md
@@ -63,7 +63,7 @@ data-export/
 └── latest
 ```
 
-Currently the corresponding S3 buckets according to environment are:
+Currently the corresponding S3 buckets for the COVID-19 data according to environment are:
 * **prod**: covid-19-country-export, covid-19-data-export
 * **dev**: covid-19-country-export-dev, covid-19-data-export-dev