Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add default geolocation rules #1744

Draft
wants to merge 24 commits into
base: master
Choose a base branch
from
Draft

Add default geolocation rules #1744

wants to merge 24 commits into from
+45,108 −4

Conversation

joverlee521
Copy link
Contributor

Description of proposed changes

Add default geolocation rules that can be used for augur curate apply-geolocation-rules. This PR only adds the default rules, but does not incorporate them into the augur curate command. I plan to do that separately.

The rules were originally copied from ncov-ingest and then sorted and modified to match places to the geographic location rather than diplomatic semantics (as discussed on Slack).

Welcome any suggestions on organization and any sharp eyes that can flag weird rules that should be removed/modified.

Related issue(s)

Resolves #1488

Checklist

  • Automated checks pass
  • Check if you need to add a changelog message
  • Check if you need to add tests
  • Check if you need to update docs

@joverlee521
Add geolocation rules
c828ef7 
This commit copies the latest geolocation rules from ncov-ingest¹
which we've been using as our "central" geolocation rules across
pathogen ingest workflows. The subsequent commits will modify the rules
to remove GISAID specific entries and incorporate the rules as the
default rules for the `augur curate apply-geolocation-rules` command.

¹ <https://github.com/nextstrain/ncov-ingest/blob/71ff771dc83ca5c5d14ea6de70132ea2e52a2ab6/source-data/gisaid_geoLocationRules.tsv>
@joverlee521
Sort geolocation_rules.tsv
237e16e 
Sort geolocation rules by the annotated country to make manual curation
easier.

I did this using visidata and the replayable commands are saved at
<https://gist.github.com/joverlee521/6d257c5b5045dd928dd374556baada9a>
@joverlee521
geolocation_rules: Label sections
72651fd 
Add clear demarcation of sections of geolocation rules
Within each section, the rules are sorted by alphabetical order of the
raw geolocation values.

Subsequent commits will clean up the general geolocation rules
since they seem to be specific to the ncov-ingest data.
@joverlee521
Remove general geolocation rules
74d2546 
Remove multiple general geolocation rules that are specific to the
ncov-ingest data.

1. Removed rule matching a date string `24 de Diciembre`
2. Removed rules that use abbreviations that seem too general to
match to specific locations.
3. Removed the general rule for "Milwaukee" because there are multiple
locations named Milwaukee and not all of them are counties.
@joverlee521
Match "French Guiana" to geographic location
87aea75 
Part of effort to match places to the geographic location rather than
diplomatic semantics.¹ Similar to the geolocation rules used in
rabies,² match "French Guiana" to its geographic location:

    region = South America
    country = French Guiana
    division = French Guiana

¹ <https://bedfordlab.slack.com/archives/CFEU0GNNS/p1736375470477489>
² <nextstrain/rabies#22>
@joverlee521
Match "Réunion" to geographic location
16d435a 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Réunion" to its geographic location:

    region = Africa
    country = Réunion
    division = Réunion
@joverlee521
Match "Guadeloupe" to geographic location
41d03ff 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Guadeloupe" to its geographic location:

    region = North America
    country = Guadeloupe
    division = Guadeloupe
@joverlee521
Match "Martinique" to geographic location
b746428 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Martinique" to its geographic location:

    region = North America
    country = Martinique
    division = Martinique
@joverlee521
Match "Mayotte" to geographic location
eeb13ba 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Mayotte" to its geographic location:

    region = Africa
    country = Mayotte
    division = Mayotte
@joverlee521
Match "New Caledonia" to geographic location
ec1aab9 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "New Caledonia" to its geographic location:

    region = Oceania
    country = New Caledonia
    division = New Caledonia
@joverlee521
Match "Wallis and Futuna" to geographic location
f69333c 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Wallis and Futuna" to its geographic location:

    region = Oceania
    country = Wallis and Futuna
    division = Wallis and Futuna

Also consolidates "Wallis and Futuna" and "Wallis and Futuna Islands".
@joverlee521
Match "Canary Islands" to geographic location
0471ee5 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Canary Islands" to its geographic location:

    region = Africa
    country = Canary Islands
    division = Canary Islands
@joverlee521
Match "Sint Eustatius" to geographic location
69c7dad 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Sint Eustatius" to its geographic location:

    region = North America
    country = Sint Eustatius
    division = Sint Eustatius

Also updates location to use the fully spelled out name instead of
the abbreviated "St Eustatius" to match other locations.
@joverlee521
Match "Anguilla" to geographic location
d3274d6 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Anguilla" to its geographic location:

    region = North America
    country = Anguilla
    division = Anguilla
@joverlee521
Match "British Virgin Islands" to geographic location
5615f3b 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "British Virgin Islands" to its geographic location:

    region = North America
    country = British Virgin Islands
    division = British Virgin Islands
@joverlee521
Match "Cayman Islands" to geographic location
c3a123a 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Cayman Islands" to its geographic location:

    region = North America
    country = Cayman Islands
    division = Cayman Islands
@joverlee521
Match "Montserrat" to geographic location
1a710bf 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Montserrat" to its geographic location:

    region = North America
    country = Montserrat
    division = Montserrat
@joverlee521
Match "Turks and Caicos Islands" to geographic location
2bec1da 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Turks and Caicos Islands" to its geographic location:

    region = North America
    country = Turks and Caicos Islands
    division = Turks and Caicos Islands
@joverlee521 joverlee521 linked an issue Feb 4, 2025 that may be closed by this pull request
Curate standard geolocation rules #1488
Open
@joverlee521
Copy link
Contributor Author

Thanks @kimandrews for flagging the USA territories! Will update rules for those in new commits.

@joverlee521
Match "American Samoa" to geographic location
51b95b0 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "American Samoa" to its geographic location:

    region = Oceania
    country = American Samoa
    division = American Samoa
@joverlee521
Match "Guam" to geographic location
043700a 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Guam" to its geographic location:

    region = Oceania
    country = Guam
    division = Guam
@joverlee521
Match "Northern Mariana Islands" to geographic location
9e6bb27 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Northern Mariana Islands" to its geographic location:

    region = Oceania
    country = Northern Mariana Islands
    division = Northern Mariana Islands
@joverlee521
Match "Puerto Rico" to geographic location
55a8acb 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "Puerto Rico" to its geographic location:

    region = North America
    country = Puerto Rico
    division = Puerto Rico
@joverlee521
Match "US Virgin Islands" to geographic location
f71e277 
Similar to previous commit, this is part of effort to match places to
the geographic location rather than diplomatic semantics.
Match "US Virgin Islands" to its geographic location:

    region = North America
    country = US Virgin Islands
    division = US Virgin Islands
@joverlee521
Add test for checking geolocation rules
17b54ae 
Adds unit test to check the default geolocation rules.
Currently checks for any "cyclic" rules where the annotation exists
in the raw matching column. This won't catch all cyclic rules because
of wildcard matching, but it's better than nothing.

This already flags some cyclic rules in the file that will be fixed in
subsequent commits.
@joverlee521 joverlee521 marked this pull request as draft February 5, 2025 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Curate standard geolocation rules
1 participant
@joverlee521