Use route APIs instead of web scraping directions links

[sgoodm]

Current approach:

We currently use a headless browser (implemented with Selenium and Firefox) to load the Javascript web map for OSM directions links. This allows us to extract the SVG path from the web map and utilize the start/end coordinates for the route to georeference the SVG path and create a GeoJSON.

Issue:

The basic implementation of Selenium does not appear to be threadsafe and can produce a range of errors when parallelized. Extracting all SVG path data in serial is by far the longest running portion of the build (takes several hours, whereas the remaining parallelized portion can take under an hour).

Possible solution:

The SVG data for routes produced by the directions links are generating using either OSRM or Grasshoper routing services, through their APIs. See an example OSM link for directions and corresponding API call for each service below

OSRM

• https://www.openstreetmap.org/directions?engine=fossgis_osrm_car&route=7.2534%2C80.4464%3B7.1613%2C80.5560
• https://routing.openstreetmap.de/routed-car/route/v1/driving/80.4464,7.2534;80.556,7.1613?overview=full&geometries=geojson

Grasshopper

• https://www.openstreetmap.org/directions?engine=graphhopper_car&route=39.7280%2C73.2477%3B40.5387%2C72.9334
• https://graphhopper.com/api/1/route?vehicle=car&locale=en&key=LijBPDQGfu7Iiq80w3HzwB4RUDJbMbhs6BU0dEnn&point=39.728%2C73.2477&point=40.5387%2C72.9334

Implementation discussion:

1. OSRM has the option to return a GeoJSON of the route directly, but Grasshopper will only return a polyline that must be decoded.

   • OSRM API docs: http://project-osrm.org/docs/v5.5.1/api/#route-service
   • Python package for working with polylines: https://pypi.org/project/polyline/

2. We would need to explore further to determine if paid API keys are required or if the ones used for OSM are considered acceptable for public use. (Note: if they are not and I need to remove the keys from the above links please let me know). Since they are not hidden from publicly-viewable queries by OSM, I am assuming they are public use.

3. Would the amount/rate of API calls be an issue (currently on order of a few hundred per build)?

4. How fast are API results? Giving parallelization issue with SVG-based implementation, I am guessing this will be a net improvement even if the API is not very fast.