add metadata to existing datasets

[sjsrey]

It would be helpful to document the original source for the cbsa definitions as these can change over time and we may need to cross-reference different time slices.

For example, we don't have parquet files for the following:

• 41900 San Germán-Cabo Rojo, PR Metropolitan Statistical Area
• 41980 San Juan-Caguas-Guaynabo, PR Metropolitan Statistical Area
• 39140 Prescott, AZ Metropolitan Statistical Area
• 38660 Ponce, PR Metropolitan Statistical Area
• 10380 Aguadilla-Isabela-San Sebastián, PR Metropolitan Statistical Area
• 11640
• 32420 Mayagüez, PR Metropolitan Statistical Area
• 25020 Guayama, PR Metropolitan Statistical Area
• 19380 Dayton, OH Metropolitan Statistical Area

It makes sense that ltdb is not covering PR areas, but the AZ, OH and 11640 are less clear why they are not in the data.

[knaaptime]

thats a good point. The shape data come directly from census, but the table that osnap refers to as cbsa is the county crosswalk, probably from here

[knaaptime]

rather than storing the CBSA crosswalk, it might be better to just read it directly when necessary (like we do with the inflation table)

the benefits would be

1. the documentation is implicit since the URL defines the original source (though we'd want to document explicitly as well)
2. we could provide methods for users to call MSAs as defined in other years

2019 cbsa-county definitions

[sjsrey]

And an older source for 2003 cbsa definitions

[knaaptime]

workflow:

1. store "appropriate version in osnap repo
2. when user calls a dataset, osnap checks the canonical dataset by downloading from the URL
3. osnap checks to see whether stored version matches canonical version
4. if versions don't match, raise a warning, etc.

[knaaptime]

if we're checking the dataset integrity with each use, is there any real use in caching it?

[knaaptime]

https://github.com/quiltdata/open-data-registry

[knaaptime]

we should use this source. It might be best to always grab from the web like we do for the inflation adjustment table, though this would be used more and more prone to failure if the census api goes down. That would also mean you always need an internet connection to make MSA queries

[knaaptime]

should be resolved by #215 but give it a look

[knaaptime]

https://spatialucr.github.io/geosnap/data.html