Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[oshdb-filter] allow filtering by changeset id and (optionally) contributor id #380

Merged
merged 34 commits into from
Jun 12, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e5eac41
extract interfaces for OSMContribution & OSMEntitySnapshot
tyrasd May 14, 2021
e93f912
prepare methods for filtering snapshots/contributions directly
tyrasd May 14, 2021
71445db
prepare changeset/contibutor filters
tyrasd May 14, 2021
fa30c53
parse contributor and changeset filters
tyrasd May 14, 2021
05c5d1d
only check valid entity states for creations/deletions
tyrasd May 14, 2021
4ea9a24
directly use new "high level" filter methods in mapreducer
tyrasd May 14, 2021
263e3c6
add to changelog
tyrasd May 18, 2021
3c980f2
enhance unit tests
tyrasd May 18, 2021
7a3092e
drop unnecessary (sometimes wrongly copy-pasted) javadoc for simple g…
tyrasd May 18, 2021
9b0c426
add test that changeset/contributor filters fails on snapshot view qu…
tyrasd May 18, 2021
56ccb92
test that contributor/changeset filter let all entities through
tyrasd May 18, 2021
7b49ef7
add new filters to oshdb-filter documentation
tyrasd May 19, 2021
7caf5e0
use interface instead of impl. class; consistent naming of helpers
tyrasd May 20, 2021
10bec30
replace potentially slow default implementation of applyOSH
tyrasd May 20, 2021
df1fb11
f
tyrasd May 20, 2021
78dccee
implement applyOSH pre-filtering to changeset/contributor filters
tyrasd May 20, 2021
628a82a
implement changeset/contributor filtering by list & range
tyrasd May 20, 2021
23067fb
improve tests for contribution based filters
tyrasd May 21, 2021
7fbba7e
remove now redundant reimplementation of applyOSH
tyrasd May 25, 2021
e194364
add documentation for range/list versions of changeset/uid filters
tyrasd May 25, 2021
ed5da8f
update changelog
tyrasd May 27, 2021
31a6b41
simplify min/max swapping
tyrasd May 27, 2021
fbfc3ad
properly implement negateable filters
tyrasd May 28, 2021
b51d75f
make `byOSHEntity` filters a bit sharper
tyrasd Jun 1, 2021
9f8fecf
drop `Filter.by` helpers
tyrasd Jun 2, 2021
a34b973
remove unused variable and import
tyrasd Jun 4, 2021
674445e
also test referenced nodes of way-members of a relation
tyrasd Jun 4, 2021
d130193
explicitly test negated versions of geometry filters
tyrasd Jun 4, 2021
495ec0d
drop unnecessarily overridden method
tyrasd Jun 4, 2021
974620c
don't crash on user-supplied filter classes during optimization
tyrasd Jun 4, 2021
18a822a
add test for the default applyOSMNegated
tyrasd Jun 4, 2021
53eaccc
also test "fallback" implementation in negatable filter
tyrasd Jun 4, 2021
a5b3888
add test for (negated) applyOSMGeometry fallback
tyrasd Jun 4, 2021
35d199a
add additional negated test
tyrasd Jun 11, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ Changelog

> See the _upgrading from 0.6_ section below for instructions how to update your code according to these breaking changes.

### new features

* enhance functionality of oshdb-filter: add new `changeset: <id|(ids)>` and (optional) `contributor: <id|(ids)>` filters ([#380])

### performance improvements

* replace an unnecessarily used Map with a more lightweight implementation using a List. ([#352])
Expand Down Expand Up @@ -60,6 +64,7 @@ Changelog
[#369]: https://github.com/GIScience/oshdb/pull/369
[#374]: https://github.com/GIScience/oshdb/pull/374
[#375]: https://github.com/GIScience/oshdb/pull/375
[#384]: https://github.com/GIScience/oshdb/pull/380
[#384]: https://github.com/GIScience/oshdb/pull/384


Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
import java.util.OptionalLong;
import org.heigit.ohsome.oshdb.OSHDB;
import org.heigit.ohsome.oshdb.api.mapreducer.MapReducer;
import org.heigit.ohsome.oshdb.api.object.OSHDBMapReducible;
import org.heigit.ohsome.oshdb.util.exceptions.OSHDBTimeoutException;
import org.heigit.ohsome.oshdb.util.mappable.OSHDBMapReducible;

/**
* OSHDB database backend connector.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@
import org.heigit.ohsome.oshdb.api.mapreducer.backend.MapReducerIgniteAffinityCall;
import org.heigit.ohsome.oshdb.api.mapreducer.backend.MapReducerIgniteLocalPeek;
import org.heigit.ohsome.oshdb.api.mapreducer.backend.MapReducerIgniteScanQuery;
import org.heigit.ohsome.oshdb.api.object.OSHDBMapReducible;
import org.heigit.ohsome.oshdb.osm.OSMType;
import org.heigit.ohsome.oshdb.util.TableNames;
import org.heigit.ohsome.oshdb.util.exceptions.OSHDBTableNotFoundException;
import org.heigit.ohsome.oshdb.util.mappable.OSHDBMapReducible;

/**
* OSHDB database backend connector to a Ignite system.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@
import org.heigit.ohsome.oshdb.api.mapreducer.MapReducer;
import org.heigit.ohsome.oshdb.api.mapreducer.backend.MapReducerJdbcMultithread;
import org.heigit.ohsome.oshdb.api.mapreducer.backend.MapReducerJdbcSinglethread;
import org.heigit.ohsome.oshdb.api.object.OSHDBMapReducible;
import org.heigit.ohsome.oshdb.osm.OSMType;
import org.heigit.ohsome.oshdb.util.TableNames;
import org.heigit.ohsome.oshdb.util.exceptions.OSHDBTableNotFoundException;
import org.heigit.ohsome.oshdb.util.mappable.OSHDBMapReducible;

/**
* OSHDB database backend connector to a JDBC database file.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,16 @@
import java.util.stream.Stream;
import org.heigit.ohsome.oshdb.OSHDBBoundable;
import org.heigit.ohsome.oshdb.OSHDBBoundingBox;
import org.heigit.ohsome.oshdb.api.object.OSMContribution;
import org.heigit.ohsome.oshdb.api.object.OSMEntitySnapshot;
import org.heigit.ohsome.oshdb.api.object.OSMContributionImpl;
import org.heigit.ohsome.oshdb.api.object.OSMEntitySnapshotImpl;
import org.heigit.ohsome.oshdb.util.celliterator.ContributionType;
import org.heigit.ohsome.oshdb.util.celliterator.LazyEvaluatedObject;
import org.heigit.ohsome.oshdb.util.geometry.OSHDBGeometryBuilder;
import org.heigit.ohsome.oshdb.util.geometry.fip.FastBboxInPolygon;
import org.heigit.ohsome.oshdb.util.geometry.fip.FastBboxOutsidePolygon;
import org.heigit.ohsome.oshdb.util.geometry.fip.FastPolygonOperations;
import org.locationtech.jts.geom.Envelope;
import org.heigit.ohsome.oshdb.util.mappable.OSMContribution;
import org.heigit.ohsome.oshdb.util.mappable.OSMEntitySnapshot;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.GeometryFactory;
import org.locationtech.jts.geom.Polygonal;
Expand Down Expand Up @@ -102,7 +103,7 @@ public Map<U, OSMEntitySnapshot> splitOSMEntitySnapshot(OSMEntitySnapshot data)
}

// now we can check against the actual contribution geometry
Geometry snapshotGeometry = data.getGeometry();
var snapshotGeometry = data.getGeometry();
OSHDBBoundingBox snapshotBbox = OSHDBGeometryBuilder.boundingBoxOf(
snapshotGeometry.getEnvelopeInternal()
);
Expand All @@ -124,7 +125,7 @@ public Map<U, OSMEntitySnapshot> splitOSMEntitySnapshot(OSMEntitySnapshot data)
// not actually intersecting -> skip
return Stream.empty();
} else {
return Stream.of(new IndexData<>(index, new OSMEntitySnapshot(data,
return Stream.of(new IndexData<>(index, new OSMEntitySnapshotImpl(data,
new LazyEvaluatedObject<>(() ->
faultTolerantIntersection(snapshotGeometry, poop))
)));
Expand Down Expand Up @@ -166,8 +167,8 @@ public Map<U, OSMContribution> splitOSMContribution(OSMContribution data) {
}

// now we can check against the actual contribution geometry
Geometry contributionGeometryBefore = data.getGeometryBefore();
Geometry contributionGeometryAfter = data.getGeometryAfter();
var contributionGeometryBefore = data.getGeometryBefore();
var contributionGeometryAfter = data.getGeometryAfter();
OSHDBBoundingBox contributionGeometryBbox;
if (data.is(ContributionType.CREATION)) {
contributionGeometryBbox = OSHDBGeometryBuilder.boundingBoxOf(
Expand All @@ -178,7 +179,7 @@ public Map<U, OSMContribution> splitOSMContribution(OSMContribution data) {
contributionGeometryBefore.getEnvelopeInternal()
);
} else {
Envelope env = contributionGeometryBefore.getEnvelopeInternal();
var env = contributionGeometryBefore.getEnvelopeInternal();
env.expandToInclude(contributionGeometryAfter.getEnvelopeInternal());
contributionGeometryBbox = OSHDBGeometryBuilder.boundingBoxOf(env);
}
Expand All @@ -203,7 +204,7 @@ public Map<U, OSMContribution> splitOSMContribution(OSMContribution data) {
// not actually intersecting -> skip
return Stream.empty();
} else {
return Stream.of(new IndexData<>(index, new OSMContribution(data,
return Stream.of(new IndexData<>(index, new OSMContributionImpl(data,
new LazyEvaluatedObject<>(() ->
faultTolerantIntersection(contributionGeometryBefore, poop)),
new LazyEvaluatedObject<>(() ->
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,6 @@
import org.heigit.ohsome.oshdb.api.generic.OSHDBCombinedIndex;
import org.heigit.ohsome.oshdb.api.generic.WeightedValue;
import org.heigit.ohsome.oshdb.api.mapreducer.MapReducer.Grouping;
import org.heigit.ohsome.oshdb.api.object.OSHDBMapReducible;
import org.heigit.ohsome.oshdb.api.object.OSMContribution;
import org.heigit.ohsome.oshdb.api.object.OSMEntitySnapshot;
import org.heigit.ohsome.oshdb.filter.FilterExpression;
import org.heigit.ohsome.oshdb.osm.OSMType;
import org.heigit.ohsome.oshdb.util.exceptions.OSHDBInvalidTimestampException;
Expand All @@ -42,12 +39,14 @@
import org.heigit.ohsome.oshdb.util.function.SerializableFunction;
import org.heigit.ohsome.oshdb.util.function.SerializablePredicate;
import org.heigit.ohsome.oshdb.util.function.SerializableSupplier;
import org.heigit.ohsome.oshdb.util.mappable.OSHDBMapReducible;
import org.heigit.ohsome.oshdb.util.mappable.OSMContribution;
import org.heigit.ohsome.oshdb.util.mappable.OSMEntitySnapshot;
import org.heigit.ohsome.oshdb.util.tagtranslator.OSMTagInterface;
import org.jetbrains.annotations.Contract;
import org.locationtech.jts.geom.Geometry;
import org.locationtech.jts.geom.Polygonal;


/**
* A MapReducer with built-in aggregation by an arbitrary index.
*
Expand Down Expand Up @@ -220,17 +219,15 @@ MapAggregator<OSHDBCombinedIndex<U, V>, X> aggregateByGeometry(Map<V, P> geometr
);
} else {
MapAggregator<OSHDBCombinedIndex<U, V>, ? extends OSHDBMapReducible> ret;
if (this.mapReducer.forClass.equals(OSMContribution.class)) {
if (mapReducer.isOSMContributionViewQuery()) {
ret = this.flatMap(x -> gs.splitOSMContribution((OSMContribution) x).entrySet())
.aggregateBy(Entry::getKey, geometries.keySet()).map(Entry::getValue);
} else if (this.mapReducer.forClass.equals(OSMEntitySnapshot.class)) {
} else if (mapReducer.isOSMEntitySnapshotViewQuery()) {
ret = this.flatMap(x -> gs.splitOSMEntitySnapshot((OSMEntitySnapshot) x).entrySet())
.aggregateBy(Entry::getKey, geometries.keySet()).map(Entry::getValue);
} else {
throw new UnsupportedOperationException(
"aggregateByGeometry not implemented for objects of type: "
+ this.mapReducer.forClass.toString()
);
throw new UnsupportedOperationException(String.format(
MapReducer.UNIMPLEMENTED_DATA_VIEW, this.mapReducer.viewClass));
}
@SuppressWarnings("unchecked") // no mapper functions have been applied -> the type is still X
MapAggregator<OSHDBCombinedIndex<U, V>, X> result =
Expand Down
Loading