Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove support for old-style MPs #862

Merged
merged 4 commits into from
Dec 22, 2018

Conversation

lonvia
Copy link
Collaborator

@lonvia lonvia commented Aug 10, 2018

Also removes support for superseding ways from polygons. The lua bindings still expect that the supersede table is present but will simply drop it.

Still todo:

  • recheck that really all old-style MP code is gone
  • adapt example Lua style
  • fix regression tests

I've done some performance tests as this PR moves all way processing to the mono-threaded first computation stage. Test were done on a Hetzner EX41-SSD (i7-7700, 32GB RAM, SSD in Raid0) with command:

./osm2pgsql -d gis --drop -C 10 -s -S ../default.style --flat-nodes gis.flatnodes planet-180625.osm.pbf

The minimal cache is a bit of a worst case scenario as all node lookups go to the slightly slower flatnode file.

master:

Node stats: total(4581403065), max(5716309124) in 425s
Way stats: total(504094523), max(601431841) in 49995s
Relation stats: total(5899892), max(8406934) in 26409s

Going over pending ways...
        357968593 ways are pending

Using 8 helper-processes
Finished processing 357968593 ways in 28948 s

Osm2pgsql took 125689s overall

This PR:

Node stats: total(4581403065), max(5716309124) in 471s
Way stats: total(504094523), max(601431841) in 71760s
Relation stats: total(5899892), max(8406934) in 25839s

Going over pending ways...
        0 ways are pending

Using 8 helper-processes
Finished processing 0 ways in 0 s

Osm2pgsql took 116951s overall

So this looks fairly promising and it would be ok to go ahead with dropping old-style MPs without improving the first-stage processing first.

@lonvia lonvia force-pushed the drop-old-style-mps branch 2 times, most recently from c007c49 to 5ac4993 Compare August 18, 2018 21:54
@lonvia
Copy link
Collaborator Author

lonvia commented Aug 18, 2018

I had forgotten the part where tags are copied from ways to relations. That is gone now, too. I think that covers all old-style MP code.

@pnorman, it would help if you could look over the tagtransform code, if the handling is still right

@lonvia lonvia changed the title WIP: remove support for old-style MPs Remove support for old-style MPs Aug 18, 2018
@mmd-osm
Copy link
Contributor

mmd-osm commented Sep 1, 2018

In context of #865: does this mean that the concept of a "pending way" becomes obsolete, and could be removed as well?

Going over pending ways...
        0 ways are pending

@lonvia
Copy link
Collaborator Author

lonvia commented Sep 1, 2018

There is no more pending step during import but it is still needed for updates, where changes on nodes/way need to be propagated upwards to ways/relations.

@pnorman
Copy link
Collaborator

pnorman commented Sep 3, 2018

bumping this for myself - haven't gotten to look at it with holidays, forest fires, job hunting, and other stuff.

@bagage
Copy link
Contributor

bagage commented Sep 24, 2018

@pnorman Did you have the time to look at this? You seem really busy but it'd be awesome to have this I think (when I see how much ways need too be processed for France only... it's killing my server!).

Also removes support for superseding ways from polygons.
The lua bindings still expect that the supersede table
is present but will simply drop it.
Remove some of the tests that test old-style MP
behaviour.
The tag type=multipolygon already indicates an area, no
matter what the other tags say.

Fixes osm2pgsql-dev#866.
@lonvia
Copy link
Collaborator Author

lonvia commented Dec 12, 2018

This has been lying around for a while now. I have some more refactoring changes pending and like to get this in to avoid merge conflicts. If there are no further objections by the end of the week, I'll just get it merged.

@lonvia lonvia merged commit 706f3a0 into osm2pgsql-dev:master Dec 22, 2018
@lonvia lonvia deleted the drop-old-style-mps branch March 8, 2019 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants