-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Creating OSM Features in iD from ArcGIS Services #4164
Comments
well said @slibby! in general the flow would be:
|
Imports are a tricky subject and can go wrong easily. Are you aware that imports of third-party data must be discussed and documented properly beforehand? The goal of OSM is not to be a large collection of so-called "authoritative" data. Our goal is a community-driven project which picks out the best pieces from all sources. Unfortunately, being an "authoritative" data set does not mean that the data set is free of errors. Importing is not just uploading a shape file. First of all, the license of the data has to be checked. OSM has some requirements to the license of the data. Often some data exists already in that area. This data has to be conflated, the history of the existing features should to be kept. The imported data should be checked if it fits between the existing OSM data. If OSM data has an offest due to the usage of the best available but still suboptimal imagery (Bing with offset of a few meters), either the OSM data or the "authoritative" data has to be modified. For example, most trees do not grow in the middle of an road (except traffic islands ;-)). Those trees have to be checked by the users who import the data. Maybe the roads has been removed or it is a new road and the authoritative data is outdated? There is a wiki page about this problem which focus on mechanical edits but it partially applies to imports, too. From my point of view, importing data should still has a minimum level of difficulty. People should be aware that they are doing something "special". I fear that offering a too easy way to import data will lead towards to much undiscussed imports which have to be reverted. This angers those who care for the data and those whose imports are reverted. I would like to ask you to get in touch with the community on the Imports mailing list before investing to much time into a feature which might attract endless anger. Please also take the time and read a few import discussions on that mailing list beforehand. You will quickly learn what the main problems of imports are. |
This looks like a super powerful feature and i'm excited by how this could greatly enrich the OSM data with unique identifiers and enables more OSM data users to contribute back to the project easily. The biggest risk is probably the potential to replace existing higher quality data that is contributed by volunteers with authoritative but lower quality information as @Nakaner outlines. This is the hardest part, and a good UI or workflow for imports should allow the user to make a clear assessment of data quality of the various sources in view to conflate successfully. |
"authoritative data" in OSM means surveyed data from the local community. You're talking about "government data". Often gov data is high quality, but not always. iD is currently very good for adding authoritative data (in the OSM sense). It's very interesting to see this, and I do think there are ways to improve imports, but I worry that this suggestion will make it too easy for someone to just blitz some data into OSM, without looking at it first. I don't think iD has (any?) good validation functionality, so this could result in a lot of bad data where someone adds the data on top of existing OSM data. I worry about people not knowing about copyright. Many people have some strange (and wrong) ideas about copyright, so a click through "yeah this is OK" could lead to copyright problems in the OSM data. You'll have to find a way of ensuring that there has been a discussion on the OSM imports@ lists for every use of this.. |
Thanks for the immediate feedback, @rory @planemad @Nakaner We have tried to consider the risks of making it "easier" to add existing data to OSM with this feature from the beginning, and to me, it breaks down to several different areas where we might provide a "brake" or impedance that limits users and tries to prevent this feature from being used for that purpose.
Last, personally, I would like to try to keep this feature from being labeled as an "easy-import" feature or being directly associated to the concept of OSM bulk imports though that is likely inevitable. There is a lot of history and discussion around bulk imports with a history of compelling arguments from different points of view, and this is intentionally not a tool to make it easier to execute the kind of imports that get justifiably reviewed by the OSM import list - large geographic areas of a specific feature type (all TIGER Lines, all buildings in NYC, etc.) or specific areas with imports of several feature types (MassGIS, city of Friday etc.). Some example use cases that we have envisioned for this:
|
Isn't the lack of response on Esri/arcgis-osm-editor#104 (comment) at least slightly worrying here? Surely until that is answered any "Arcgis services" should be considered unusable, at least within iD, where we don't expect users to have as much knowledge about the relationships between different bits of OSM data as we might elsewhere. |
Imports can have a number of problems that must not be ignored. The concept of "data import because the OSM community is small" is particularly problematic; for OSM to thrive it is necessary that you have people on the ground who take ownership and who maintain and amend the data. Imports work well when they are done with and for a local community; they are a liability when done in lieu of a local community. This is a social problem rather than a technical one; people often don't understand OSM enough to get that. Care must be taken not to send out the wrong signals. But I would like to focus on the technical side for now and mention a few typical issues that will need addressing in this context. When people have copied information from their GIS systems into OSM in the past, the following problems have occurred frequently:
|
@SomeoneElseOSM Yes, I do lament the lack of response on that issue but I also know it is still being actively pursued on the Esri side. However, that issue is specifically related to a service provided by Esri directly (Esri has to contend with the data license for all the imagery), where this issue's functionality is intended to allow the addition of data provided by other organizations (municipalities, states, etc.) who are simply using Esri software to expose the data. It's akin to adding support for adding WFS layers (from anywhere) to iD. |
@woodpeck, thanks for the specific and candid feedback and concrete examples of challenges with previous imports!
our current design for this only grants an opportunity to map attributes when users have selected an OSM preset and then limits users to mapping to existing relevant tags. the information you've provided about outside identifiers is helpful and we can ensure that they are stripped as well.
we can definitely pursue checking vertex counts and simplifying/squaring when necessary.
The tool in its current form certainly doesn't allow for wholesale replacement and imported features must be reviewed individually prior to inclusion in a changeset. our hope with this was to explicitly require QA with respect to existing data. |
This is an ongoing process, but I'll write about some protections which are baked into this tool for a deliberate and careful process of bringing in this data:
Still in concept phase, but ultimately I would like to have this work a little like HOT's Task Manager - an expert sets up the service, area, license, and data tag mapping, and then individuals can come and edit small areas to that standard. |
Most of the technical and other issues have already been touched on (well maybe nobody has pointed out that adding data also implies cleaning up if the whole thing is a screw up which is a lot more work than in a typical layered GIS system, but I digress). so I just want to touch on point 1. Copyright by @slibby. In, in the mean time many years of, my experience the absolute last people that are able to understand if their own terms of use are compatible with both our contributor and distribution (ODbL) terms are typical government GIS offices (which I assume is the market ESRI is targeting here), self declaration as only verification is not a good idea. Further it is not true that we do not police background layers and similar that can be used indirectly to add data by tracing of features or by individual feature copying (currently mainly limited to JOSM). We both vet the most used sources of such layers and have technical safeguards available in all major editors including logging of used sources. Naturally this is more difficult than detecting direct imports, but that is no reason to take a stance of you can't do it, so we will ignore you. As has already been pointed out ESRI itself is a good example of our processes working. If you have proof of non-approved sources being used, please forward the information to the DWG data@osmfoundation.org which will take the appropriate steps to rectify the situation. |
@mapmeld wrote:
How do you ensure that the users of this feature are aware of the Import Guidelines and don not violate the guideline? If you make a workflow easier, it is your responsibility that you do not lead your users into an area which looks nice but is dangerous. |
@Nakaner the OSM wiki uses the following definition of Automated edit.
(emphasis mine) the protections we are putting in place (to forbid importing overlapping features and require review and approval for each individual new feature) are intended to ensure that the human oversight remains meaningful. is it possible that a user can click buttons and approve individual problematic edits one at a time? or course! but it is also possible for folks to sketch poor edits by hand with equal velocity. yet you're attempting to apply the same standard as you do to bulk edits made without manual review. lots of folks have chimed in here with valid concerns that we all share. we are attempting to articulate our plans to address these things individually, but other specific suggestions would also be sincerely appreciated. |
@simonpoole I had a question about the logging of imagery sources - is the mandatory imagery_used changeset tag the primary method used in iD for this currently, or are there other behind the scenes telemetry or logging? Perhaps this provides a way to save a similar reference from this proposed feature - a recording that the changeset includes features from |
I'd just like to jump in here as someone who has done a lot of OSM editing (mostly with Potlatch2), has lots of experience with open data, and indeed, currently manages open data for a local government, with lots of useful spatial data. I'm really in favour of this kind of tool. There seems to be an assumption in some of the above negative comments that competent OSM users dont need easy-to-use tools, and such tools will only encourage clueless newbies to carry out harmful imports. Currently I don't have any way to import data, since I don't use JOSM. I'd love to be able to import lots of POIs, building outlines etc, on a case-by-case basis. This would be really handy. An easy way of dealing with the copyright issue is by whitelisting a known set of approved endpoints. (The only issue for me with this specific tool is we don't currently expose any ArcGIS services, so this is a bit useless to me. If it could import GeoJSON it would be useful.) |
Umm... are you really sure about that? Because if so, you won't be able to import any buildings. Countries are mapped in OSM. Relation 148,838 is the United States of America. Does that mean you don't import any buildings which overlap with that polygon? You're allowed to import buildings which overlap with Additionally you should probably check for other polygons, not just buildings. What if someone wants to import lakes that overlap existing OSM buildings? What about riverbanks and roads? |
@rory - when writing that I was going back and forth on how generic to make the statement. What the feature actually does we can avoid overlapping duplicate data on OSM. We currently have this enabled for conflicting building polygons, preferring any existing OSM version. If it's relevant we could also apply it to another iD Editor preset, such as water polygons. It's also only capable of working with the data in the browser viewer... we don't search the planet for polygons. |
@jgravois wrote:
Have you read the Import Guidelines? What your tool is going to make easier is an import, not an automated edit. I don't summarize the guideline for you, please read it on your own. |
OK, that makes more sense. 🙂 It's good to ensure you don't put buildings on top of buildings, but there are many other things to be careful with, buildings/roads (w/o layers/tunnels/bridges). If you were to write this a JOSM plugin, instead of for iD, you'd benefit from JOSM's existing validators, which covers "overlapping buildings" and much more. |
Hey @slibby, @mapmeld @jgravois, I am really excited to take a look at what you've built, and I appreciate that you opened this issue for discussion before submitting a PR, especially considering that imports are a controversial topic, and this work has the potential to have a larger effect on OpenStreetMap than a typical change to iD. I'm also glad to hear the comments from @rory, @Nakaner, @woodpeck, @SomeoneElseOSM, @simonpoole, and @stevage (hope I am not forgetting anybody). Many of you are leaders within the OSM community and work with the DWG on policing and cleaning up after imports, so have more experience than anybody when it comes to dealing with the problems caused by bad imports. Here are my overall impressions, (I will follow up with more direct responses to some of the items):
So, I'm looking at this as an opportunity to improve how importing is done. Let's keep the discussion going and I'm looking forward to a PR that we can test out (against the OSM dev server of course!) |
Hi! 👋 Absolutely ❤️ the ideas and discussion here. I've broken up my comments based on my experience / 2 cents.. Adding GeoService LayerI've done quite a few (approved) imports with locality provided data from ArcGIS Online/Open Data. As you can imagine sometimes the data is "interesting" and requires a bit of pre-processing. I've also experimented with using AGS Open Data as reference in updating / verifying military boundaries I've always thought the wealth of authoritative data / information that organizations make available in AGOL could be super useful as reference for normal tracing/verification in iD.
While some of the sources are authoritative, they're not necessarily great for importing due to over-generalization, or way too many vertices for example. But they are super helpful for providing the missing context or reference needed for the casual mapper. TL:DR Please oh please make it easy to Importing GeoService LayerI also think extending the If the implementation is viewed as an import and has been discussed with community and the import guidelines are followed, it's up to the user which tool they would use to perform the import. The discussed functionality here would make iD another import tool and only make the import process smoother when using an AGOL data source. Great work @slibby @mapmeld @jgravois and anyone else i left off... |
@bhousel wrote:
Adding a changeset tag with the URL of the discussion on the Imports mailing list is a great idea. If that is done, @bhousel wrote:
I don't think that it is a blocker but if someone proposes on the Imports mailing list that he is going to use iD for an import, people will tell him that JOSM has a validator. ;-) OT: Is there already an issue for tracking the progress of the validator feature? There is a dozen of issues where people complain about broken multipolygons, broken route relations etc. but no overall validation issue. |
Great discussion and comments so far. Assuming the license behind the data sources of the GeoServices layer are compatible with ODbL and that the process prohibits people from large scale imports or overwriting existing OSM data, this could be a great addition to iD. I don't see much risk in users trying to edit complex boundary relations using a GeoService overlay. Beyond the relatively easy cases of adding missing building footprints or roads, it would be valuable to transfer missing attributes from 'Authoritative' sources to data already in OSM. |
@slibby wrt imagery_used this may or may not be appropriate for non imagery sources, but I would suggest if you don't want to use imagery_used you should use a differently named tag for logging arcgis sources. The related topic is that you either need to support the black list from the OSM API, or a similar mechanism to block sources. @ALL we have a number of people commenting on this issue that are not regular participants in such discussions, so that we can all understand who is coming from where, could everybody state their affiliation, if any, with ESRI when commenting? Further: we should be clear that while improved tooling is great (@stevage ESRI brought the issue of easy imports forward as one of the goals of the PR, not anybody else), this does amount to support of not just any proprietary API, but that of the de facto monopolist in GIS-space. Now while in the end it is up to Mapbox to decide and there are other proprietary APIs supported in iD with no open counterpart, it would seem that this proposed PR should be balanced out with full support for WMS and WTMS layers so that other companies have a fighting (well not really) chance of supporting their customers using iD. Affiliation with ESRI: none, I do live in likely the sole country in which ESRI has only 95% market share in the public administration market segment instead of 100%, so I'm likely biased. |
I am the GIS Coordinator for San Juan County, WA (San Juan islands) and I think this is an amazing idea that I've been hoping for for years! The San Juan Islands are a busy summer tourist destination in the PNW but have a very small year-round population (~16,000). Unfortunately, aggregate mapping services like Google, Bing, and Esri have historically had very bad maps of our region which often leads to confusion for both tourists and locals. The County itself has very accurate maps and the data is public domain, but most map services (Esri being the exception) have not been willing to *mport our authoritative data. Instead, they require us to submit change requests for each road or address. I (with some help from two friends) have corrected ~95% of County roads in OSM by tediously moving TIGER imported nodes or tracing new roads in iD using our street basemap with the goal of having at an openly licensed and reasonable quality mapping service in our rural area that tourists and locals may be able to use. The proposal here and in #2586 would vastly improve the speed and quality of data into OSM for my County and likely many other areas. I hope both proposals can move forward together cohesively and I would be happy to contribute to them. |
I work for Esri as stated in my Github bio, though this work is not part of my paid responsibilities so I would prefer to consider myself part of the OSM community for the purposes of this issue and eventual PR, though that may not be very convincing. @simonpoole is the blacklist that you referred to the elements available here? http://api.openstreetmap.org/api/0.6/capabilities |
Esri has very high penetration in government GIS systems in Australia, but much less in public-facing open data. There are a few government bodies using ArcGIS Open Data (for example) but much more common is uploading spatial datasets to open data platforms such as data.gov.au (example, where they can be accessed by WFS or GeoJSON (no Esri here). Some councils, including where I work, use Socrata, which supports GeoJSON, KML and Shapefile but not WFS (or Esri web services) - example. Of course, there are lots of raster tile endpoints, both Esri and GeoServer, too - see nationalmap.gov.au. So, the situation you describe does not really represent the current state of open spatial data in Australia, at least. So anyway, if this issue could be reframed as "Selectively import features from open data endpoints such as WFS, GeoJSON and ArcGIS Feature Services", with an extensible import mechanism, that would be awesome. |
@Nakaner wrote:
There isn't a single issue tracking the validator feature, but there are a bunch of issues tagged , and #3130 is probably the best one that lists a bunch of things that we could validate. |
It seems like there are some differing viewpoints on this thread about how to define what this tool is actually doing.
And yea, it seems like in most commenters here are treating this feature as an import tool:
But is importing what this tool is actually doing? The import section of the OSM Wiki defines it this way:
Based on this definition I think that there is a case to be made that this is not an import feature at all. It might seem like splitting hairs but there is an important difference between the two. The fact that individual users would be reviewing each addition to the OSM dataset is something that I feel is largely being discounted in this conversation. Certainly, bulk imports can be problematic. There are also a lot of ways people make poor decisions when interacting with OSM on an elemental basis but what we are talking about here is closer to the latter than the former. OSM is an open project I think we all accept that people make both bad and good decisions adding to it. If the argument here is that this tool will make it easier to make poor decisions than we should be up front about that and not just write it off as a bulk import tool. I'm new to this conversation so for transparencies sake I'll say that I do not work for ESRI nor does my job depend on ESRI's good favor in any way. I work for USAID and have been a proponent of OS there for some time. I have also been wanting to see a feature like this for a while now, dont really care where it comes from. |
Yeah. From the initial description:
I would define the two options like:
|
Thanks again to everyone who responded to this Issue in great detail. I will be presenting on this topic tomorrow (Friday) at State of the Map US in Boulder, and our hope is to open a pull request shortly afterward, showing the work we've done to take these comments and suggestions into account. I would recommend we move further comments to that PR thread once it's opened. My slides for the presentation are available here, and a recording of the session will also be made: https://docs.google.com/presentation/d/1oV0gKRiqpINp4frwoMYLzCF2dsR4ZqDDxyRF8xWHTJM/edit?usp=sharing edit: video of the talk is here: https://www.youtube.com/playlist?list=PLqjPa29lMiE2k2Sp5L5rb6ntduG9dt0te |
That is the link for the playlist of all talks of State of the Map US 2017. For easier access, the direct link to the video of the presentation @slibby gave that shows the functionality being discussed in this thread, is here: |
Shouldn't this be closed as ESRI has added the functionality to RapId, circumventing any possible outcome of the discussion on this fork? |
I think that depends on whether RapId is accepted as widely-available editing client. This code is pretty stale and likely VERY out of date from the current state if iD, so I'm inclined to agree with closing the Issue. More info on the RapId work here: https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/arcgis-data-support-in-osm-editors/ Others are welcome to comment on the closed issue if you prefer to re-open it. |
The Goal
Knowledgeable users can easily contribute reviewed and authoritative data to OpenStreetMap with the iD editor using data extracted from an ArcGIS Map or Feature Services. This issue serves to introduce the concept and ask for feedback from the primary iD contributors and maintainers before submitting a PR (as requested in Contributing.md).
Background
(edit: the use of authoritative in this statement should be reconsidered)
Many Esri customers using ArcGIS are authoritative data creators and maintainers. In many areas, densely detailed user-submitted data is available in OpenStreetMap, but in many others, there is a limited density of data even though easily-accessible authoritative data exists. Currently, to make use of these datasets OpenStreetMap contributors would need to export the ArcGIS Map Service to a Shapefile or PostGIS database and then use desktop tools such as JOSM to edit and upload to OpenStreetMap, or go through a lengthy import process to import a selected set of data which is not already available in OSM or which is more precise or more accurate than current OSM features.
This proposed feature aims to make it easier for volunteers to add features queried from the GeoServices REST API (ArcGIS Server Map Services and Feature Services) to OpenStreetMap using iD. By connecting directly to the ArcGIS Service form an iD session, contributors are getting the most up to date data and also can more easily contribute this data without the need for complex workflows. The tool will also allow users to match up existing attributes from the source data (for example, house numbers on building footprints) to relevant OSM tags to make adding authoritative attribution straightforward along with geometries.
Solution
This feature will add the capability for contributors to select “Add GeoService Layer” and then provide the URL to an ArcGIS REST Service or Feature Layer (ex. http://maps2.dcgis.dc.gov/dcgis/rest/services/DCGIS_DATA/Transportation_WebMercator/MapServer/5). Before loading, the contributor will be able to specify whether specific attributes should be mapped from the source layer to specific OSM tags, or whether any other static tags should be added to all features. The iD editor will load the vector features of the current map extent as temporary objects in the browser window (not immediately submitted as edits to OSM).
The contributor can then select objects generated from the ArcGIS Layer, inspect and alter any tag values or geometries and submit them as edits to OSM (additions or replacements for features). They can then save their changeset, contributing these new objects to OpenStreetMap via iD's existing functions.
Implementation
An initial working version of this feature has already been developed using a branch from the iD GitHub repository and is nearing readiness for submitting a PR to the master iD branch. We have attempted to address potential concerns about ease of import and user experience in the initial version.
cc @jgravois @mapmeld @ajturner
The text was updated successfully, but these errors were encountered: