-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use JSON for OSM API instead of XML #3765
Comments
The OSM API is XML, not JSON. Some calls have a JSON endpoint, generally those which don't return OSM data. |
Unfortunately, it's as @pnorman says. I'd love to ship a full JSON API, and I've been working away at it (very slowly - sorry!) over at https://github.com/zerebubuth/openstreetmap-cgimap. It already supports JSON as an output format, so it's "just" a matter of getting full API coverage written, tested and debugged, then we can enable JSON by default. If anyone's interested in helping out, that would be greatly appreciated. |
Thanks for the info @pnorman & @zerebubuth and for your work on this 👍 There is no rush, we were just looking into starting work on an OSM Notes layer this week and noticed some JSON support. |
I'm continuing our discussion in #5180 here, as JSON output per se doesn't really have anything to do with Web Workers. Latest status was that cgimap 0.6.2 is deployed on the dev instances, and provides OSM output in Overpass API JSON format: You can try out some JSON output on the dev server for now: https://master.apis.dev.openstreetmap.org/api/0.6/map.json?bbox=-80.28218507766725,25.773537461179274,-80.24941921234132,25.782676914766654 |
Nice! I just clicked on that to take a look.. |
(Json is minified now) Indeed, |
Just do it - any reduction of data sent over the wire is a good thing. It's also probably even slower for whatever library you're using to pretty format the json. |
Just following up on our previous discussion... I did a few measurements to get some idea of XML vs. JSON parsing performance. Data and JSON endpoint: http://api06.dev.openstreetmap.org/ http://localhost:8080/#background=Bing&disable_features=boundaries&map=17.83/49.04575/2.02767
I uploaded larger parts of Halifax to the dev server, so this might also be a good location for testing: http://localhost:8080/#background=Bing&disable_features=boundaries&map=18.36/44.65918/-63.59597 My previous assumption that JSON feels faster seems to be backed by those numbers. I added some code below in case someone wants to try this out. Prototype coding: jsonparser_prototype.txt NB: Most likely that patch won't apply cleanly, as I manually removed some (unrelated) parts. |
@gravitystorm: my measurements seem to indicate, having native JSON support in the API would bring some noticeable parsing speed ups in iD, immediately impacting UX. As we already have JSON support available in cgimap (based on the commonly used Overpass API JSON format as per openstreetmap/operations#244), the question has come up, what steps would be required to add similar support for the Rails port as well. Could you open an issue on the OSM website repo where we can track all required activities, and possibly dispatch this to multiple devs in case this makes sense? As Rails noob I think I'm not the right person to drive this forward in a meaningful way. |
Sure, I can do that.
I'm not sure that we have all that many devs to split the work yet! |
Tag me, maybe I'll JFDI. |
I'd say, most of the work is to move all that XML generation code from the controller to the view where it belongs. Once that bit is done, we could eventually add some JSON related stuff to the view as well and teach teh controller to render either XML or JSON. If I'm not completely mistaken, @woodpeck already started some work to move XML to the views, maybe as part of GDPR (see https://wiki.osmfoundation.org/wiki/Board/Minutes/2019-04-24). I have no idea what his overall plans and timelines are. After all, there may be a few people we could ping to get JSON support in Rails. |
Yes, I started with openstreetmap/openstreetmap-website#2164 and now there's the pending openstreetmap/openstreetmap-website#2207, I was planning to work my way through the stuff from easiest to hardest ;) I cannot devote too much time on it and I'm not a Rails ace to begin with, hence the snail's pace. Also most of the tests currently use the to_xml methods in the models and when XML generation is moved to views, the tests need to receive their own "build an XML thingie for this" methods. Sadly if you want to add JSON you'll probably have to do the same again, i.e. not only generate JSON in the views, but also have your own JSON-generating code in the tests (because it's difficult to co-opt the views from inside the tests, and also bad design to use parts of the code we're testing in building the test case). It is true that my motivation for separating output generation from the models was that eventually I want to be able to restrict parts of the output from showing to anonymous users. If someone wants to help with this or, better still, take it off my plate, I certainly won't object. OSMF would also be willing to pay for (at the least the GDPR related part of) this if a suitable contractor can be found. Nobody has come forward in response to https://blog.openstreetmap.org/2018/11/03/rfq-changes_to_rails_api_and_cgimap/ yet. |
Ok, I gave it a try and added a prototype for the map, nodes, ways and the relations endpoint, both producing XML and JSON (Overpass API style) output: mmd-osm/openstreetmap-website@913dd07 Those should be the basic building blocks of what we need for any of the other related calls. Test cases are also deliberately left out. Same applies to user_display_name_cache and changeset_cache |
Hm yes, if you only want JSON as an output format and not for input then it might not be all that bad with the tests. |
About the only reason why I'm looking into this is JSON deployment as part of CGImap 0.6.2, which faced so much opposition, that I had to cancel it altogether (see openstreetmap/operations#244 (comment)). I can fully understand those concerns, but as we're so extremely limited in available devs on the Rails side, this increasingly starts impacting other parts. Anyway, let's don't look back, get this thing done and move on! |
Follow ups: openstreetmap/openstreetmap-website#2485, #7188 |
Json is in production, let’s close this issue. |
@mmd-osm Thanks, this should have been closed months ago 😆 |
We could simplifiy
osm.js
by using the JSON calls instead of the XML ones, which we need to parse with parser functions.@pnorman @zerebubuth is there anything in here we need to watch out for, or are all of the API calls available over JSON nowadays?
The text was updated successfully, but these errors were encountered: