Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Align speed scripts part 2 / Jul open data part 2 #1192

Merged
merged 12 commits into from
Jul 31, 2024

Conversation

tiffanychu90
Copy link
Member

@tiffanychu90 tiffanychu90 commented Jul 31, 2024

segment_speeds

  • Test on Jul 2024 for all segments, run all scripts from nearest_vp_to_stop to average_segment/summary_speeds
  • Backfill all dates in rt_dates.y2024_dates, rt_dates.y2023_dates, rt_dates.apr2024_week, rt_dates.oct2023_week, rt_dates.apr2023_week...35 dates in total
  • within pipeline_* scripts, there's a spot where kernel dies
    • start profiling the code again
    • wrapping all the dates in dask.delayed and then computing them seems to be ok.
    • only 1 spot where kernel will need to be restarted. after running nearest_vp_to_stop and after the 1st date in filter_to_two_vp. restarting allows the rest to run.
  • multi-day averages need to be injected 1 week at a time, or kernel will crash
  • what's probably happening is some X number of objects aren't getting trashed before the next date is computed, and at some point, some of the larger files overflow. restarting trashes everything and subsequent steps are small enough where this isn't an issue.
  • Research Request - Align segment_speeds pipeline scripts to use vp with dwell time #1183
  • run Jul 2024 for Epic - Open Data Publishing 2024 #991

gtfs_funnel

  • Update README and add a table to reflect what dates are supported across these workstreams

rt_vs_schedule

  • overwrite and backfill rt_stop_times tables for all dates
  • rest of rt_vs_schedule metrics does not change -- use vp_usable without the dwell times, because we want to know how many vp are produced each minute, and we don't want a vp that's been rolled up across minutes
  • run jul 2024 for monthly publishing

@tiffanychu90 tiffanychu90 merged commit 93bfcdb into main Jul 31, 2024
2 checks passed
@tiffanychu90 tiffanychu90 deleted the align-speed-scripts-p2 branch July 31, 2024 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Research Request - Align segment_speeds pipeline scripts to use vp with dwell time
1 participant