Planet generation experiences

[systemed]

I've successfully generated an .mbtiles from planet.osm.pbf. 🎉

Using current master plus #292. Planet renumbered first with osmium renumber. Command line:

tilemaker --input planet.osm.pbf --output planet.mbtiles --bbox -180,-60,180,75 \
    --compact --store /media/ssd/store

Execution time c. 37 hours (with the exception of deallocating, see below). This is a 16-threaded HP machine, 2x X5650, 144GB RAM.

Maximum memory usage c. 131GB, plus 267GB store. mbtiles filesize is 65GB.

What needs fixing?

• Deallocating is very slow, first in the .clear() method after reading the pbf, but more significantly at the end of running (where it takes hours). The .pbf is complete and usable but tilemaker is still nominally running after its "Filled the tileset with good things" message.
• Ijsselmeer and the Great Lakes are not showing.
  
• Roundabouts don't show at lower zoom levels, causing a gap in the road. I suspect that they are simplified to 2-point lines and ultimately disappear entirely. We probably need to ensure that closed linestrings are never simplified below 4 points.
  
• Boundary lines are often discontinuous. This is because we don't yet support type=boundary relations, so any way that is a member of a type=boundary relation but not otherwise tagged will get dropped.
  

I suspect we might be able to get the running time down further too and optimise tile sizes, but the above issues are the most important.

[kleunen]

Yes the issue with the ijsselmeer we had before. The default style only renders water from osm up to zoom level 8. But large lakes are visible after zoom level 8. Some even up to zoom level 0.

[kleunen]

The cleanup at the end is not even needed. The data structures inside the mmap file get deallocated and then when all this is done. The complete file gets deleted anyway ..

[systemed]

There's something very wrong at the end - the "Filled the tileset with good things" line is literally the last line of tilemaker.cpp, but it's been hung for 12 hours now since that was output.

Ijsselmeer and the Great Lakes are missing entirely - not even showing at z8+. I'll see if I can work out what's going on there (need to check it wasn't introduced by #292).

[kleunen]

"Filled the tileset with good things" is the last line in tilemaker.cpp, but after this all the destructors are run. I also ran a planet conversion, it did nog hang or take a lot of time to cleanup. I generated the planet by merging 4 extracts of the planet. So probably there is a deadlock now somewhere when cleaning up ?

I do have ijsselmeer in my extract:

[systemed]

That's interesting. I don't get Ijsselmeer at all when generating from netherlands-latest.osm.pbf, either with #292 or current master.

[kleunen]

I am not sure if I now used my adapted style for this.

[systemed]

I've gone through commit-by-commit, and the problem starts with #307 (a9a54f9). Previous versions of the code rendered Ijsselmeer fine. Not sure yet what it might be in #307 causing it.

Edit: within #307, fe5229e is fine so it's something after that. IJsselmeer is read correctly by Lua which writes it to the water layer.

[kleunen]

maybe something with 'Use OsmID to lookup generated geometry'.

that would be the most likely candidate to cause this issue.

[kleunen]

I think i actually see the issue with this commit

Try #317, i have not tested it yet

[systemed]

Changing that line to

osmID = (relationId & OSMID_MASK) | OSMID_RELATION;

does seem to work. We previously had --newWayId because way and relation IDs were sharing the same ID space, but if we're separating them bitwise anyway with OSMID_RELATION and OSMID_MASK, I don't think we need newWayId any more. Does that make sense or have I misunderstood?

[kleunen]

Yes, it does seem to be a collision between the OSMIDs. That is why i added the extra two bits and the | OSMID_RELATION. To make the osm id unique. So yes, that does make sense.

The & OSMID_MASK is just to make sure the id only takes 40 bits max.

[systemed]

Ok - addressed in #318.

[kuwapa]

I wanted to add my results regarding the boundary issue here learned from #322

Boundaries which are tagged as ways and not as relations are not being generated by tilemaker. For example these -
https://www.openstreetmap.org/way/707454735
https://www.openstreetmap.org/way/136107215

Also, disputed borders are not being generated. Like this one -
https://www.openstreetmap.org/way/133267807

[systemed]

Yep. The issue is that tilemaker doesn't currently have any support for type=boundary relations, so anything that's tagged simply as a boundary relation (without the ways being tagged as boundaries) won't come through. I know how I'm going to fix it - the solution will enable other relation types like routes to be handled too - but haven't yet had the time to do so.

[systemed]

Boundaries are addressed in #360.

[systemed]

With the same setup as above (--compact, store on SSD, -60° to 75°), plus #386 with a renumbered locations-on-ways planet, the peak drops to 118GB RAM. That's a 10% saving from the previous 131GB even though we're reading more data (boundary relations), and brings it down to within the capacity of a 128GB machine. Store size is 274GB.

It's still processing - will update with timings later.

[systemed]

Successfully completed for a 72GB mbtiles.

We have a performance regression in that it took 48hr09 (vs 37hr last time). The slowdown was very clearly in tile generation, and I think probably at higher zoom levels. Reviewing the commits since 13 Sep my best guess is that since #360, we're creating massive boundary (multi-)linestrings which have to be clipped for each output tile. In particular it might be https://github.com/systemed/tilemaker/blob/master/src/output_object.cpp#L112 where we are running geom::intersection over the whole MultiLinestring; we could filter individual segments much as we do for Linestring (above it in the code).

[systemed]

#387 should hopefully fix the regression - I'll try another planet run in a few days.

[KevenRing]

I recently converted a renumber'ed planet.osm.pbf using the latest master branch and stock OSM config and process files (it took about 26hrs on a 32 core system). Large lakes (eg Great Lakes) still don't show up until zoom level 6.

While reading the ocean shape file (obtained at https://osmdata.openstreetmap.de/download/water-polygons-split-4326.zip), there are several invalid shape entities reported as having invalid self-intersections, and others that have too few points. Is there a better/newer ocean shapefile set?

There seem to be "holes" in various places; these disappear at different zoom levels. The attached pictures are from the raw vector data using tileserver-gl around Sri Lanka. I would presume these occur because of the ocean shapefile errors mentioned above?

[timsluis]

Large lakes (eg Great Lakes) still don't show up until zoom level 6.

In the resources/process-openmaptiles.lua file there is a function somewhere around line 617 which is called when processing bodies of water. This function filters water from the output tiles based the size of the body of water. You'll also find that the biggest bodies of water are filtered at zoom level 6. You can change the zoom level for example to 3 which will render large lakes until zoom level 3.

Is there a better/newer ocean shapefile set?

The shape file is updated every day based on OSM data. So probably you can fix the polygons in OSM and your changes will reflect in the dataset the next day.

They also mention the following on the osmdata.openstreetmap.de website: The coastline in OpenStreetMap is often broken. The update process will try to repair it, but this does not always work. If the OSM data can't be repaired automatically, the data here will not be updated.

[duaneking]

Any updates? Asking out of curiosity.

[systemed]

@duaneking Updates about anything in particular? :)

[duaneking]

I'm very interested in this, but unfortunately I haven't been able to find working examples or correct documentation.

Tools like tilemill No longer seem to exist in a functional state; other tool sets just don't seem to work, either. Others will crash if you use a modern system, and can no longer be installed on even older systems due to dependencies. Every single stack overflow answer on this is wrong, Because they say to use tools that don't exist anymore. So many tutorials say to use tile Mill, but it wont even install or run in my tests due to glibc issues.

I can't find working open source examples of how to process the open street maps data from the planet parts I want to the theme and zoom levels I need (7 at most, probably more like 6) on a theme that is respectful of the low visioned, because none of the tools that people say to use will even install cleanly or work; the projects themselves will not compile, install, or run cleanly, and things like tilemill no longer exist so all the people saying to use that are wrong now, because tilemill wont even install.

Its extremely frustrating.

I thought maybe I could short cut the processing issue by finding an MBtiles file that fit my needs, but it's impossible to find a tile set of files that have been processed in dark mode with a theme targeted at the hard of sight or the visually disabled who are blinded by the brightness of the default theme; my use cases are for the disabled and partially sighted, and so I have theme requirements because the default QA tile set - besides being incomplete - it's physically painful to look at for many people due to physical visual disability,so it's just rendered wrong for my needs.

Would you be willing to point me in the right direction? I thought maybe this bug was tracking getting these systems fixed so that's why I started to watch it.

[typebrook]

@duaneking

it's impossible to find a tile set of files that have been processed in dark mode with a theme targeted at the hard of sight or the visually disabled who are blinded by the brightness of the default theme

Things you need to know:

1. MBTiles is just a sqlite database, which contains tiles. Tile format could either be image or Mapbox-Vector-Tile(MVT)¹

2. tilemaker only convert OSM file into MVT, not images (rendered map)

3. Mapbox-Vector-Tile is just a dataset, not a rendered map. To let end-user see the real map, you need a rendering solution in frontend. It could be:

   • Mapbox Style file²: Used for Maplibre³/Mapbox⁴⁵ native/web solutions
   • Tangram Scene file⁶: Used for Tangram native/web solutions

   Or any others...

4. Make a rendering solution for visually disabled is off-topic in this repo. You may try to make MVT by tilemaker, and try to build your own rendering solution by some web tools:

   • https://maputnik.github.io/
   • https://cloud.maptiler.com/maps/
   • https://studio.mapbox.com/
   • Or just hire me 😄

I can't find working open source examples of how to process the open street maps data from the planet parts

Then tilemaker is the best solution AFAIK! But remember, the result is just MVT, not a rendered map

Footnotes

1. https://docs.mapbox.com/help/glossary/mbtiles/ ↩

2. https://docs.mapbox.com/mapbox-gl-js/style-spec ↩

3. https://maplibre.org/ ↩

4. https://docs.mapbox.com/mapbox-gl-js/guides/ ↩

5. https://docs.mapbox.com/android/maps/guides/ ↩

6. https://tangrams.readthedocs.io/en/main/Overviews/Scene-File/ ↩