Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix KeyError parsing for Cuba #2778

Merged
merged 4 commits into from
Aug 16, 2022
Merged

Fix KeyError parsing for Cuba #2778

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7ccdca8
new_zealand.py
rbevansp Jul 18, 2022
3d9c24c
Fix parser error NZ
rbevansp Jul 21, 2022
8c73a09
Fix the parsing KeyError
rbevansp Jul 21, 2022
6c96b56
Except only for KeyError and log the error
rbevansp Jul 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
22 changes: 14 additions & 8 deletions ingestion/functions/parsing/cuba/cuba.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,16 +161,22 @@ def parse_cases(raw_data_file, source_id, source_url):
# First make dict mapping code names of diagnostic and treament centers
# to actual locations
hospital_map = {}
for centre_type in ['centros_aislamiento', 'centros_diagnostico']:
for centre in json_data[centre_type]:
hospital_map[centre] = json_data[centre_type][centre]['nombre'] + \
", " + json_data[centre_type][centre]['provincia']
try:
for centre_type in ['centros_aislamiento', 'centros_diagnostico']:
for centre in json_data[centre_type]:
hospital_map[centre] = json_data[centre_type][centre]['nombre'] + \
", " + json_data[centre_type][centre]['provincia']
except KeyError:
logger.error(f"KeyError in Cuba parser")

# Get schema_version
schema_version = json_data['schema-version']
if schema_version != 7:
logger.warning(
f'Schema version has been updated from 7 to {schema_version}')
try:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was schema-version removed? A change in schema-version should break the script, as we might not be capturing the correct fields.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

schema-version was removed. There is no mention of schema-version in covid19-cuba-1.json.

schema_version = json_data['schema-version']
if schema_version != 7:
logger.warning(
f'Schema version has been updated from 7 to {schema_version}')
except:
schema_version = "(undef)"

for day in json_data['casos']['dias']:
if 'diagnosticados' in json_data['casos']['dias'][day]:
Expand Down
5 changes: 3 additions & 2 deletions ingestion/functions/parsing/new_zealand/new_zealand.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,9 @@ def convert_demographics(entry):
'''
demo = {}
if (age := entry[_AGE]) != "NA":
if '90+' in age:
demo["ageRange"] = {"start": 90, "end": 120}
if '+' in age:
start = int(age.rstrip('+'))
demo["ageRange"] = {"start": start, "end": 120}
else:
start, end = list(map(int, age.split(' to ')))
demo["ageRange"] = {"start": start, "end": end}
Expand Down