Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow queries with filters to use fast-lane reducers #511

Merged
merged 2 commits into from
Aug 31, 2023

Conversation

tyrasd
Copy link
Member

@tyrasd tyrasd commented Jul 14, 2023

Currently, every query which is using a filter (either via an ohsome filter or a custom filter predicate) is using the slower backend routines, because the actual filtering is implemented using flatMap which has a larger overhead than the more streamlined routines which know that no flatMap operations have to be performed.

This has the small side-effect that the NullPointerException from #159 is back (yet, in a different place). But I'd say this is acceptable, as using null values for actual map results should be avoided anyway.

Todo

  • compare performance (should be most notable on a query which affects many entities, e.g. type:way and building=* in a country-sized bbox like Germany)
  • revert benchmarking-only code (7418f6d)

Checklist

@tyrasd tyrasd added the performance Performance optimizations, bottlenecks of the current pipeline, etc. label Jul 14, 2023
@tyrasd tyrasd force-pushed the perf-non-flatmapped-filters branch 2 times, most recently from 3e852d2 to aeae5ba Compare July 14, 2023 16:03
@tyrasd tyrasd marked this pull request as draft July 14, 2023 16:05
@tyrasd tyrasd force-pushed the perf-non-flatmapped-filters branch from aeae5ba to 39c341a Compare July 14, 2023 16:25
@tyrasd tyrasd force-pushed the perf-non-flatmapped-filters branch from 266be07 to 7418f6d Compare August 18, 2023 10:16
@tyrasd
Copy link
Member Author

tyrasd commented Aug 30, 2023

compare performance

some preliminary results using the ohsome API as a benchmark:

query local instance of master branch local instance of #511 branch
type:node and created_by=*, global monthly1 36s 27s
type:way and building=*, Germany bbox, monthly2 1min 22s 59s
type:way and building=*, global, yearly3 2m 49s 2min 27s

Footnotes

  1. https://api.ohsome.org/v1/elements/count?bboxes=-180,-90,180,90&filter=type%3Anode%20and%20created_by%3D*&format=json&time=2013-01-01%2F2023-01-01%2FP1M

  2. https://api.ohsome.org/v1/elements/count?bboxes=6.2,47,14.9,54.7&filter=type%3Away%20and%20building%3D*&format=json&time=2013-01-01%2F2023-01-01%2FP1M

  3. https://api.ohsome.org/v1/elements/count?bboxes=-180,-90,180,90&filter=type%3Away%20and%20building%3D*&format=json&time=2013-01-01%2F2023-01-01%2FP1Y

@tyrasd tyrasd force-pushed the perf-non-flatmapped-filters branch from 2a5398b to e0aa315 Compare August 31, 2023 11:30
@tyrasd tyrasd marked this pull request as ready for review August 31, 2023 11:31
Copy link
Member

@rtroilo rtroilo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed by peer-review.
Thank you

@tyrasd tyrasd merged commit bf9e5e5 into master Aug 31, 2023
1 check passed
@tyrasd tyrasd deleted the perf-non-flatmapped-filters branch August 31, 2023 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance optimizations, bottlenecks of the current pipeline, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants