fix: Corrections from Cal-ITP (a selection) #125

e-lo · 2022-04-06T17:14:36Z

Categories of fixes:

Update download URLs to point to transit provider's DNS rather than contractor or transitfeeds
Update transit provider names to be current (e.g. "Santa Maria Regional (not Area) Transit"), consistent (e.g. LA Metro should have same name for Bus and Rail feeds with only difference being bus vs rail), have their brand (e.g. "Muni")
Fix outdated feed URLs
Fix miscategorizations (e.g. Nevada RTC should be in...Nevada)

Still a lot more to add/fix, but this is a start.

Updates to use AC Transit's own URL

> Blue Lake Rancheria is included in Humboldt's feed > Name should be Humboldt, not arcata/mad river (which is a subsystem of humboldt)

- LA Metro should have consistent naming between Rail and Bus (this is same agency, with a split feed only b/c of size)

- Should refer to DASH and Commuter Express bus services which is the brand people will search from - Update URL to be agency's

- add brand: yolobus - update URL to be agency's

- name is Taft Area Transit (no maricopa in any names other than routes)

Soltrans download should be agency DNS

- Should be nevada not california

- not named PASS transit

- Update URL to be agency DNS - Add brand: SacRT to name for searchability

- Should be San Mateo (or at least a city in San Mateo county) not San Francisco as municipality

- Download should be agency DNS

Add acronym

- Update download link to be current (previous one is several years outdated) - Update name to be accurate

- add acronym

- name isn't SMAT anymore - its SMRT / santa maria regional transit NOTE: this is an outdated download URL!

CLAassistant · 2022-04-06T17:14:40Z

All committers have signed the CLA.

emmambd

Hi @e-lo,

Thank you for our first community source PR to the catalogs! It’s exciting to see some changes here that aren’t available on other aggregators yet.

I’ve left some comments in the review for specific instances where we can’t make an alteration, mainly due to current limitations of the catalogs. I’ve created issues for the relevant changes where they seem clear.
In order to trigger the pipeline, source additions/changes need to have [SOURCES] at the end of the PR name.

emmambd · 2022-04-11T14:11:11Z

.../gtfs/schedule/us-california-alameda-contra-costa-transit-district-ac-transit-gtfs-1244.json

@@ -15,7 +15,7 @@
        }
    },
    "urls": {
-        "direct_download": "https://transitfeeds.com/p/ac-transit/1269/latest/download",
+        "direct_download": "https://api.actransit.org/transit/gtfs/download?token={{ API_KEY }}",


As highlighted in the release plan, right now API-based sources can’t be added since we’re running checks to see if a source is a valid ZIP file. I’ve created this issue to make the checks conditional so API sources can be added and we can define a convention for including API keys and authorization methods in the schema.

I’ve also added this info to CONTRIBUTING.MD so the constraint is clearer.

Addressed in updated PR.

emmambd · 2022-04-11T14:12:06Z

catalogs/sources/gtfs/schedule/us-california-blue-lake-rancheria-transit-system-gtfs-598.json

@@ -1,21 +0,0 @@
-{
-    "mdb_source_id": 598,


In order to preserve the Mobility Database IDs, we can’t delete sources. This issue is here so we can add deprecation info to the schema.

This is unfortunate as I don't think the definition of what a unique ID even represents has been defined (see comment in I believe #36 ) - so it seems impractical /impossible to "hold them constant" if we don't know what they represent.

I would suggest we don't "hold anything to be frozen/backwards compatible until this issue is resolved/closed as it will end up in a LOT of messiness.

I backed out this change in the PR.

emmambd · 2022-04-11T14:14:18Z

catalogs/sources/gtfs/schedule/us-california-santa-maria-regional-transit-gtfs-26.json

@@ -15,7 +15,7 @@
        }
    },
    "urls": {
-        "direct_download": "http://data.trilliumtransit.com/gtfs/smat-ca-us/smat-ca-us.zip",
+        "direct_download": "",


Currently, changing the name of a file would also trigger changing the direct download URL and break pipeline workflows. We could change the direct download URL structure so it’s non-descriptive (only the source ID for instance), or discuss other potential solutions.

A direct download URL needs to be available for the source to be valid. I’m assuming there’s no available replacement since you left it empty? We have talked about adding feed_end_date to the schema to make it easier to see if the most recently available URL is still in use.

This is unfortunate. So what happens when an agency changes their name? When a URL changes what they represent? If this is the case, I would suggest only having the ID and/or geography (something that isn't going to change) in the name.

A direct download URL needs to be available for the source to be valid.

That is correct. What should be the course of action - since the URL isn't valid and therefore doesn't conform to the schema?

Agreed. Issue created here.

The schema states that the URL should automatically open the source information, and doesn't describe anything regarding data recency. In instances where the URL is not up-to-date but there is no known available alternative, we've still been including it in the catalogs, as this is a very common occurrence and deleting the source entirely seemed counterproductive for data access. Once historical URLs are implemented in the schema, we could include these URLs there instead for clarity.

Currently, changing the name of a file would also trigger changing the direct download URL and break pipeline workflows. We could change the direct download URL structure so it’s non-descriptive (only the source ID for instance), or discuss other potential solutions.

Here, it's the latest dataset URL that is affected, not the direct download one (so it's system generated not provided by the user), but I agree that in any cases it should be stable enough that changing the filename won't break the pipeline.

A direct download URL needs to be available for the source to be valid. I’m assuming there’s no available replacement since you left it empty? We have talked about adding feed_end_date to the schema to make it easier to see if the most recently available URL is still in use.

I think adding feed_end_date and feed_start_date would be a good idea. It would allow us not to update the latest url if the feed expiration date is reached. Also, we could always keep the direct download url after adding the historical data.

…y-database-catalogs into cal-itp-corrections

e-lo · 2022-04-12T14:44:58Z

General suggestion:

requirements about not deleting or renaming files should be added as a github action "check"

This should still be deleted - but is a limitation of the definition.

* beaumont transit fix * humboldt county fix * santa maria regional transit name fix * taft area transit fix * southern nevada rtc fix * la county shuttles agency name fix * ladot fix * sacramento regional tansit fix * sf bay area water emergency fix * sfmta fix * samtrans fix * santa barbara fix * scmtd fix * tulare county area fix * yolo county fix * add fares features * replace duplicate go bus file * switch replacements of 561 and 28 so URL of 28 is consistent

emmambd · 2022-11-23T16:46:35Z

@e-lo These data updates were blocked until we found a safe method to change file names. Now that we know consumers are systematically refreshing the CSV, we can modify file names confidently and have updated the documentation to highlight file name changes. Because of this, these changes are now unblocked and have been merged in PR #239. (I made a separate PR for the sake of ease.) I am closing this one since the changes are already reflected.

e-lo added 18 commits April 6, 2022 09:55

AC Transit URL

8b3338c

Updates to use AC Transit's own URL

Fix Humboldt

58aaa33

> Blue Lake Rancheria is included in Humboldt's feed > Name should be Humboldt, not arcata/mad river (which is a subsystem of humboldt)

Fix LA Metro

822cc1a

- LA Metro should have consistent naming between Rail and Bus (this is same agency, with a split feed only b/c of size)

Fix LA DOT

7a45e19

- Should refer to DASH and Commuter Express bus services which is the brand people will search from - Update URL to be agency's

Fix Yolo

d2f1ade

- add brand: yolobus - update URL to be agency's

Fix taft

eb9b702

- name is Taft Area Transit (no maricopa in any names other than routes)

Fix soltrans

05d9e43

Soltrans download should be agency DNS

Fix Nevada RTC

837e215

- Should be nevada not california

Fix beaumont

6b51f7b

- not named PASS transit

Fix SacRT

36ef570

- Update URL to be agency DNS - Add brand: SacRT to name for searchability

Fix SamTrans

f90d0e3

- Should be San Mateo (or at least a city in San Mateo county) not San Francisco as municipality

Fix SBMTD

bdb387f

- Download should be agency DNS

Fix Santa Cruz

8780290

- Download should be agency DNS

Fix SF Muni

dc89940

Add acronym

Fix WETA

7c0925d

Add acronym

Fix LADPW

3d42364

- Update download link to be current (previous one is several years outdated) - Update name to be accurate

update tcat

e80874c

- add acronym

Update Santa Maria name

8d1f3b0

- name isn't SMAT anymore - its SMRT / santa maria regional transit NOTE: this is an outdated download URL!

e-lo changed the title ~~Corrections from Cal-ITP (a selection)~~ Fix: Corrections from Cal-ITP (a selection) Apr 6, 2022

e-lo changed the title ~~Fix: Corrections from Cal-ITP (a selection)~~ fix: Corrections from Cal-ITP (a selection) Apr 6, 2022

emmambd self-requested a review April 6, 2022 18:05

emmambd reviewed Apr 11, 2022

View reviewed changes

emmambd and others added 3 commits April 11, 2022 10:16

Merge branch 'main' into cal-itp-corrections

5d241ed

Back out API Key changes

e47ac20

Merge branch 'cal-itp-corrections' of https://github.com/e-lo/mobilit…

ecb4378

…y-database-catalogs into cal-itp-corrections

Back out deletion of Blue Lake Rancheria.

bb78c76

This should still be deleted - but is a limitation of the definition.

emmambd mentioned this pull request Nov 18, 2022

fix: Update california feeds from PR #125 #238

Closed

emmambd mentioned this pull request Nov 18, 2022

fix: Update california feeds from PR #125 [SOURCES] #239

Merged

emmambd closed this Nov 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Corrections from Cal-ITP (a selection) #125

fix: Corrections from Cal-ITP (a selection) #125

e-lo commented Apr 6, 2022

CLAassistant commented Apr 6, 2022 •

edited

Loading

emmambd left a comment

emmambd Apr 11, 2022

e-lo Apr 12, 2022

emmambd Apr 11, 2022

e-lo Apr 12, 2022

e-lo Apr 12, 2022

emmambd Apr 11, 2022

e-lo Apr 12, 2022 •

edited

Loading

emmambd Apr 14, 2022

maximearmstrong May 19, 2022

e-lo commented Apr 12, 2022

emmambd commented Nov 23, 2022

fix: Corrections from Cal-ITP (a selection) #125

fix: Corrections from Cal-ITP (a selection) #125

Conversation

e-lo commented Apr 6, 2022

CLAassistant commented Apr 6, 2022 • edited Loading

emmambd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e-lo Apr 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

e-lo commented Apr 12, 2022

emmambd commented Nov 23, 2022

CLAassistant commented Apr 6, 2022 •

edited

Loading

e-lo Apr 12, 2022 •

edited

Loading