[FEATURE] Other input formats

[msbarry]

Add support for other geographic encoding formats, for example:

• geojson
• newline-separated geojson
• flatgeobuf
• geopackage

[durkie]

do you envision that planetiler might pick up where tippecanoe left off? i regularly use tippecanoe with newline separated geojson, and it's a great piece of software, but it doesn't seem like it's getting much maintenance/development these days.

[msbarry]

Interesting, I didn't realize the last commit to tippecanoe was almost a year ago. My guess is it's deprecated in favor of their new proprietary tiling service.

I think there are some jobs people use tippecanoe for that planetiler could handle, but not sure about all of the density-based geometry filtering. Planetiler actually looks more like an alternative to the tiling service.

The initial motivation was talking with someone trying to generate contour lines from DEM data. They used gdal contour to generate 1x1 degree contour flatgeobuf tiles then they tried to use tippecanoe but it fell over trying to hold the planet in memory. Planetiler should be able to fill that gap since it does the same job but only holds a single input and output tile in memory at a time.

Supporting more input formats, and exposing more through config (#18) so you can do more without writing Java code would open up handling more use-cases from tippecanoe.

[flother]

What do you think about supporting GeoPackage as an input format too? It's essentially a standardised SQLite database format — a bit like MBTiles — and it's supposed to be a replacement for the venerable shapefile. I've seen it used quite a bit for government data.

[msbarry]

Makes sense! As long as you can read shapes with key/value tags from it then we should be able to use it as an input source.

[wipfli]

Can planetiler handle shapefiles (https://en.wikipedia.org/wiki/Shapefile)?

The Swiss government distributes shapefiles (https://data.geo.admin.ch/) for example for natural protected areas. In the past, I converted shapefiles to GeoJSON and then to mbtiles with tippecanoe.

[msbarry]

Can planetiler handle shapefiles (https://en.wikipedia.org/wiki/Shapefile)?

Yes, in fact planetiler currently can only handle shapefiles, natural earth sqlite file, and .osm.pbf files. The openmaptiles profile uses shapfiles for water polygons and lakelines:

planetiler/planetiler-basemap/src/main/java/com/onthegomap/planetiler/basemap/BasemapMain.java

Lines 28 to 50 in 4ad4d95

	Planetiler.create(arguments)
	.setDefaultLanguages(OpenMapTilesSchema.LANGUAGES)
	.fetchWikidataNameTranslations(sourcesDir.resolve("wikidata_names.json"))
	// defer creation of the profile because it depends on data from the runner
	.setProfile(BasemapProfile::new)
	// override any of these with arguments: --osm_path=... or --osm_url=...
	// or OSM_PATH=... OSM_URL=... environmental argument
	// or osm_path=... osm_url=... in a config file
	.addShapefileSource("EPSG:3857", BasemapProfile.LAKE_CENTERLINE_SOURCE,
	sourcesDir.resolve("lake_centerline.shp.zip"),
	"https://github.com/lukasmartinelli/osm-lakelines/releases/download/v0.9/lake_centerline.shp.zip")
	.addShapefileSource(BasemapProfile.WATER_POLYGON_SOURCE,
	sourcesDir.resolve("water-polygons-split-3857.zip"),
	"https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip")
	.addNaturalEarthSource(BasemapProfile.NATURAL_EARTH_SOURCE,
	sourcesDir.resolve("natural_earth_vector.sqlite.zip"),
	"https://naciscdn.org/naturalearth/packages/natural_earth_vector.sqlite.zip")
	.addOsmSource(BasemapProfile.OSM_SOURCE,
	sourcesDir.resolve(area.replaceAll("[^a-zA-Z]+", "_") + ".osm.pbf"),
	"planet".equalsIgnoreCase(area) ? ("aws:latest") : ("geofabrik:" + area))
	// override with --mbtiles=... argument or MBTILES=... env var or mbtiles=... in a config file
	.setOutput("mbtiles", dataDir.resolve("output.mbtiles"))
	.run();

[HarelM]

I'm looking into adding geojson support, I'll look at @erik's PR to better understand where I need to change the code.
As far as I can tell JTS only handles single geometry parsing here, so it can't be used to convert FeatureCollection and Feature, right?
https://github.com/locationtech/jts/blob/master/modules/io/common/src/main/java/org/locationtech/jts/io/geojson/GeoJsonReader.java
I found the following libraries:
https://github.com/Urban-Research-Lab/jts-geojson
https://github.com/geotools/geotools
https://github.com/opendatalab-de/geojson-jackson

Not sure which is the preferred one, geotools seems like the one with the highest stars...
Let me know if you are familiar with a dependency I can use to start hacking this reader into planetiler...

[wipfli]

cc @bdon since you have been looking into a geojson parser for the clipping branch https://github.com/protomaps/basemaps/pull/351/files#diff-f325f761549c50b4e2b6f1cb1e77dd8232d99a9f5d4abedc934cabdc3260a6aa