Update to OpenMapTiles V3_13 #46
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Adds a script to downloads multiple areas and compute their test parameters * added a large test that uses a combined 76MB file with equatorial-guinea, liechtenstein, district-of-columbia, greater-london * cache wikidata downloads
Somehow test fails on options in github actions, add debugging output for that cmd
…most false positive (openmaptiles#841) Thank you @frodrigo for PR.
* Trims SSD drives, flushes cache before each performance test. Unfortunately these are still incomplete -- need to use real hardware machines for all these to take effect. * A bit more output in PR updater
Avoid `http://http://...`
Missed another typo
* quickstart support for osmfr and bbike areas - Use the `--empty` flag to start with an empty database - Geofabrik as the default server - osmfr is used for hierarchical area names such as `europe/austria` - bbbike is used for Capitalized area names such as `Adelaide`
Code cleaning. The function `update_gbr_route_members()` use two sub queries, but only one is relay needed.
Generate multiple SQL files to be imported in parallel. The files will respect the cross-layer dependencies, so they can be all ingested at the same time.
* make sure all required dirs are initialized before running any of the docker commands. Otherwise docker will create them as root. * minor docs cleanup
preloaded postgis image can take longer than 60 seconds to start on slower machines. Increase the timeout to 3 minutes.
…ptiles#853) Replacing materialized view by a tables with update from trigger on change only. Start with the most simple cases. Just replicate the change on: * `osm_water_polygon` to `osm_water_lakeline`, * `osm_water_polygon` to `osm_water_point`. Use a view to factorize the `osm_water_lakeline` and `osm_water_point_view` definition and reuse it in the trigger. The update of `osm_important_waterway_linestring` is more complex, as it is a merge of `osm_waterway_linestring`. It not done in the same way. At the end of the transaction we remove impacted and recompute them. The goal is to update more quickly the content of derivated table by just updating the changing content. It replaces the update of materialized view because their need a full recompute (with lock issue). Note, an advanced version of differential update over materialized view as already implemented in the building cluster PR openmaptiles#725. It addresses openmaptiles#814 and a part of openmaptiles#809.
layer_building uses osm_all_buildings, which uses a non-parallel-safe as_numeric. Marking it as non-parallel safe for now until we switch to CleanNumeric later.
Add `getmvt` sql function as part of the build-sql make step. The function will be imported into PG during the `import-sql` step after all other SQL is done. This is a noop for the `generate-tiles` approach.
Adds tag leisure=swimming_pool as water polygon
Add support of tag landuse=basin as water polygon.
…iles#864) * Fix space at start in update_osm_route_member() * Update Canada network route name * Update Canadian route network
Uses the patch by @zstadler to filter out irrelevant Mapnik warnings. This will be ignored until tools v5.1 is used.
`nullif(as_numeric(height),-1)` is doing a double conversion -- null into -1 followed by -1 back into null. Remove that, and delete unused `as_numeric`
Add a new `show-metadata` target to show all settings in the .mbtiles file.
Places the output of `pg_dump --schema-only` into the saved build artifact as `schema.sql` Closes openmaptiles#877
Change the feature_id of objects in the tiles from arbitrary number to osm_id. From discussion here openmaptiles#725 (comment) Note the multiploygon building are load with negative relation osm_id. But the use in feature_id do an overflow (eg -7980888 -> 18446744073701570000). We can keep as is or not. No stats done for yet. Trying to let the CI do it.
Add more clear step into quickstart.md for using own OSM data fix openmaptiles#843
Code clean. Use the same naming convention for the SLQ update file as other layer.
Closes openmaptiles#271 This PR adds track and path rendering at lower zooms than currently provided, and also achieves near-parity with openstreetmap-carto on track and path rendering. A previously-abandoned attempt, with significant discussion, was openmaptiles#1169.
Add `province` (next to `state`) tag into `osm_state_point`
This PR performs the same fix on the park layer that openmaptiles#1268 applied to the waterway layer. It removes blank name, name_en, and name_de from the tiles.
This PR implements non-blocking updates in the park layer. The approach was to use the `REFRESH MATERIALIZED VIEW CONCURRENTLY` feature in postgres. In order to achieve this, a unique index was added on the z4 dissolved park area table. The `ST_Union` / `ST_Dump` sequence was changed to an explicit cluster DB scan (an equivalent operation) so that a unique osm ID could be generated from each dissolved polygon. Below is a screen shot from Idaho, USA showing that the dissolved z4 still works as expected. 
…maptiles#1295) I discovered this bug while investigating issues with the updates process related to openmaptiles#1190 openmaptiles#1292, and openmaptiles#814. The `transportation_name` layer produces slightly different `tags` hstore values in the `osm_transportation_name_linestring` table during the initial import versus when running an update. As currently written, the import code produces null-value keys in the `tags` column, while the update code suppresses them. This PR removes that difference and makes the import code use same method that is currently used in the update code. With a test case I've written, the import code produces a tags hstore that looks like this: `"name"=>"OpenMapTiles Secondary 2", "name:de"=>NULL, "name:en"=>NULL, "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"` ...while the update code produces a tags hstore that looks like this: `"name"=>"OpenMapTiles Secondary 2", "name_int"=>"OpenMapTiles Secondary 2", "name:latin"=>"OpenMapTiles Secondary 2"` Note the missing NULL values. This bug causes a small amount of space wastage after an update is run, because the update matching code detects the `tags` value as different, resulting in a duplicate copy of the tags value if that row is updated. This causes duplicate objects and breaks GROUP BY clauses that expect to group same-tagged features together. I've tested this by inspection of a generated mbtiles, database spot checks, and the unit test code included in this PR.
Fixes openmaptiles#1148 This PR adds expressway tagging to the `transportation` layer, by setting `expressway=1` for non-motorway roads tagged `expressway=yes`, and omitting the tag otherwise. Additionally, I've added a few unit tests to verify that the expressway tagging is being imported and updated into the intermediate tables. Here is an example of expressway tagging on US-1 in Rhode Island, USA: 
This PR adds labels to `aerialway` linestrings after zoom level 12. I roughly followed PR openmaptiles#1179 These are useful in ski resorts and should not inflate the size of the tiles by much as aerialways are rare:  Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
* Add info about using MID_ZOOM to optimize tile generation * Remove params for generation that are now automatically detected Co-authored-by: Tomas Pohanka <TomPohys@gmail.com>
Preparation for release 3.13
`make list` insn't a valid command. This updates the documentation with the proper commands.
Remove `STRICT` statement from `transportation_name_tags` function. Concurrently with remove `STRICT` from `get_basic_names` and `get_latin_name` in OMT-T FIX openmaptiles#1319
PR move `get_basic_names(tags, geometry)` function from `transportation_name_tags` function and adding it after `transportation_name_tags` resolve street abbrevation. PR runs through `test-sql` test without error. Adding to `osm_transportation_name_network` (table and for update) was done in consequence of `test-sql` errors ([update test 500](https://github.com/openmaptiles/openmaptiles/blob/a747d985508933c9619c9a765efc59d8c5c88085/tests/test-post-update.sql#L53)). Originally `get_basic_names` was just in `osm_transportation_name_linestring` table. FIX openmaptiles#1324 
Remove `FILTER_MAPNIK_OUTPUT` variables from docker-compose which stop raising warnings during start-up dockers. This functionality was removed in openmaptiles/openmaptiles-tools#349. The current version of the mapnik does not need `FILTER_MAPNIK_OUTPUT`.
…s#1334) This PR fixes a bug that causes that `track` lines disappear at z13. This bug was introduced in openmaptiles#1190, which adds rendering of paths and tracks at z12 and z13. Before this PR: z12: lines with `route_rank = 1` are added (no matter what `highway`). z13: lines with `route_rank BETWEEN 1 AND 2` and `highway = 'path'` are added. -> tracks with `route_rank=1` are added at z12 but not at z13. After this PR z12: lines with `route_rank = 1` and `highway IN ('path', 'track')` are added. z13: lines with `route_rank BETWEEN 1 AND 2` and `highway IN ('path', 'track')` are added . -> only tracks and paths are added at z12 and z13 (which was IMHO the goal of openmaptiles#1190) * Add only the most important paths and tracks (route_rank=1) to z12 and more important ones (route_rank between 1-2 or sac_scale or has name) to z13.
…emove all_tags mapping temporarily
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.