Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basedata import - support polygon and polyline files #697

Closed
3 tasks done
shawnlaffan opened this issue Jul 23, 2018 · 3 comments
Closed
3 tasks done

Basedata import - support polygon and polyline files #697

shawnlaffan opened this issue Jul 23, 2018 · 3 comments
Assignees
Milestone

Comments

@shawnlaffan
Copy link
Owner

shawnlaffan commented Jul 23, 2018

A reasonable amount of distributional data is provided as polygons, for example the IUCN range maps.

It should be possible to import these using a polygon intersection approach, with a fishnet polygon layer used to define the cells, and the coordinate taken from the fishnet centroids.

The fishnet could be dynamically updated to support multiple file imports, e.g. if two data sets have differing extents. This could be done using the bounding boxes.

http://www.gdal.org/classOGRGeometry.html#a83adb426e79958c90aecb418d91c765c

update 2018-11-29
The process can also work with polylines, so include them too.

2018-12-06, remaining work:

  • Add geometry properties area and length to field name options. If not used then use binarise counts approach.
  • Throw exception when basedata has zero cellsize for an axis that uses the shape x or y coordinates. We cannot build a fishnet for zero sized axes.
  • Check that group axes include :shape_x and :shape_y, and build fishnet accordingly (e.g. if user specifies a text axis, then :shape_y then :shape_x).
@shawnlaffan shawnlaffan added this to the Release_3 milestone Jul 23, 2018
@shawnlaffan shawnlaffan self-assigned this Jul 23, 2018
shawnlaffan added a commit that referenced this issue Nov 17, 2018
shawnlaffan added a commit that referenced this issue Nov 17, 2018
was originally adapted from python code

Updates #697
shawnlaffan added a commit that referenced this issue Nov 17, 2018
shawnlaffan added a commit that referenced this issue Nov 20, 2018
shawnlaffan added a commit that referenced this issue Nov 25, 2018
shawnlaffan added a commit that referenced this issue Nov 25, 2018
Currently fails on re-run on windows.
need to test against CI.

Updates #697
shawnlaffan added a commit that referenced this issue Nov 25, 2018
Updates #697
shawnlaffan added a commit that referenced this issue Nov 26, 2018
The DESTROY method appears to be overzealous at version 0.0601

Updates #697
shawnlaffan added a commit that referenced this issue Nov 26, 2018
shawnlaffan added a commit that referenced this issue Nov 26, 2018
Needed when sets of files use different cases

Updates #697
@shawnlaffan
Copy link
Owner Author

shawnlaffan commented Nov 28, 2018

Intersection process is very slow for some data sets, even when spatial indexes are set. This is of the order of minutes, where ArcGIS Pro takes a few seconds.

One cause is multipart polygons with many vertices. Making these single part appears to help, but needs to be done outside the system for now. ogr2ogr has an explodecollections option to do this, but I cannot find it in the OGR API.

Multi-resolution fishnets might be worth trying.

If we only need presence/absence and basedata cells are square then we could also rasterise instead.
https://gdal.org/gdal__alg_8h.html#adfe5e5d287d6c184aab03acbfa567cb1

shawnlaffan added a commit that referenced this issue Nov 28, 2018
Code was originally adapted from python.

Updates #697
shawnlaffan added a commit that referenced this issue Nov 28, 2018
shawnlaffan added a commit that referenced this issue Nov 28, 2018
Speeds up one intersection from 9 minutes to 4 minutes

Updates #697
shawnlaffan added a commit that referenced this issue Nov 28, 2018
Substantially speeds up the overlays.

Updates #697
shawnlaffan added a commit that referenced this issue Nov 28, 2018
shawnlaffan added a commit that referenced this issue Nov 28, 2018
This way we test the SQL statement
quoting from the previous commit.

Updates #697
shawnlaffan added a commit that referenced this issue Nov 28, 2018
The proper version will be in a future
release of Geo::GDAL:FFI.

Updates #697
shawnlaffan added a commit that referenced this issue Nov 29, 2018
@shawnlaffan shawnlaffan changed the title Basedata import - support polygon files Basedata import - support polygon and polyline files Nov 29, 2018
shawnlaffan added a commit that referenced this issue Nov 29, 2018
shawnlaffan added a commit that referenced this issue Nov 29, 2018
We now do both kinds of poly data.

Updates #697
@shawnlaffan
Copy link
Owner Author

shawnlaffan commented Dec 6, 2018

Faster approach for binary (presence only) case in cfbf327

Need to look at using this as the default. Could use special fields for area and length, as per current approach for x, y, z and m coords. If :area and :length are not used for any of the data fields then we don't need geometry properties for the sample counts.

shawnlaffan added a commit that referenced this issue Dec 6, 2018
Fails a test a because the Intersects operator includes
shared boundaries.

Updates #697
shawnlaffan added a commit that referenced this issue Dec 6, 2018
This shrinks the polygons by a small amount to
avoid cases where polygons touch at their edges,
but otherwise do not intersect.

Updates #697
shawnlaffan added a commit that referenced this issue Dec 7, 2018
We need two, and cannot have zero or negative spacings.

Updates #697
shawnlaffan added a commit that referenced this issue Dec 7, 2018
shawnlaffan added a commit that referenced this issue Dec 7, 2018
@shawnlaffan
Copy link
Owner Author

Mark as fixed.
Test failures are appveyor/chocolatey related.

shawnlaffan added a commit that referenced this issue Dec 20, 2018
The filter geometry does it for us.
Not sure why I thought it just used
the bounding box.

A late update to #697
shawnlaffan added a commit that referenced this issue Dec 20, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant