OSHDB helpers integration

CHANGELOG.md

@@ -18,6 +18,7 @@ Changelog
 
 * allow to flexibly combine (automatic) aggregation methods (like `aggregateByGeometry(…)` or `aggregateByTimestamp()`) with each other and with `filter` or `map`/`flatMap`, regardless of the order of the applied operations ([#451])
 * add new OSHDB filters: `perimeter`, `geometry.vertices`, `geometry.outers`, `geometry.inners`, `geometry.roundness` and `geometry.squareness` ([#436])
+* add OSHDB-helpers module providing two helpers (OSHDBDriver and OSHDBApplication) to simplify database setup ([#474])
 
 ### bugfixes
 

documentation/first-steps/README.md

@@ -41,6 +41,9 @@ Now, we're ready to go and use the OSHDB to query the OSM history data. If we're
 OSHDBDatabase oshdb = new OSHDBH2("path/to/extract.oshdb");
 ```
 
+| 🛈 | Alternatively, we provide simple [connection helper](../manual/helpers). |
+|----|----------------------------------|
+
 ## 4. Select OSHDB view
 
 The next step is to decide which kind of analysis we want to perform on the OSM history data. Two different analysis views are provided by the OSHDB:
@@ -148,4 +151,4 @@ The result from this query is visualized in the following graph:
 
 ## 12. Next steps
 
-That's it for our first-steps tutorial. Of course there are many more options and features to explore in the OSHDB. For example how the contribution [view](../manual/views.md) lets you analyze each modification to the OSM objects individually, more advanced [filtering](../manual/filters.md) options, or other concepts like the [`flatMap`](../manual/map-reduce.md#flatmap) function, custom [`aggregateBy`](../manual/aggregation.md) and [`reduce`](../manual/map-reduce.md#reduce) operations, etc.
+That's it for our first-steps tutorial. Of course there are many more options and features to explore in the OSHDB. For example how the contribution [view](../manual/views.md) lets you analyze each modification to the OSM objects individually, more advanced [filtering](../manual/filters.md) options, or other concepts like the [`flatMap`](../manual/map-reduce.md#flatmap) function, custom [`aggregateBy`](../manual/aggregation.md) and [`reduce`](../manual/map-reduce.md#reduce) operations, etc.

documentation/manual/README.md

@@ -26,6 +26,8 @@ Contents
   Explains the design of the OSHDB data model.
 * [Installation](installation.md) <br>
   A guide for how to install the OSHDB from sources.
+* [Helpers](helpers) <br>
+  Documentation for helpers that reduce complexity in connection setup for OSHDB.
 
 See also
 --------

documentation/manual/data-model.md

@@ -29,6 +29,6 @@ The global OSM history data set is divided into a set of partitions (grid cells)
 Keytables
 ---------
 
-In order to minimize memory needed to store the key and value strings of OSM tags, the OSHDB uses so called “keytables” that assign every string (e.g. the tag key `builing`) to a number. More often used strings are assigned to lower numbers compared to rarely used strings which are assigned to higher numbers.
+In order to minimize memory needed to store the key and value strings of OSM tags, the OSHDB uses so called “keytables” that assign every string (e.g. the tag key `building`) to a number. More often used strings are assigned to lower numbers compared to rarely used strings which are assigned to higher numbers.
 
 This allows the data stored in the OSH entities to be more compact compared to storing each complete string with each entity.

documentation/manual/helpers/OSHDBApplication.md

@@ -0,0 +1,98 @@
+# OSHDB Application Template
+## Installation
+
+Replace your OSHDB dependency with the following:
+
+```xml
+<dependency>
+   <groupId>org.heigit.ohsome</groupId>
+   <artifactId>oshdb-application-template</artifactId>
+   <version>0.7.2</version>
+</dependency>
+```
+
+## Usage
+
+Create a main Class extending on the `OSHDBApplication`:
+
+```java
+public class MyApplication extends OSHDBApplication {
+
+    public static void main(String[] args) {}
+
+    @Override
+    protected int run(OSHDBConnection oshdb) throws Exception {
+      return 0;
+    }
+}
+```
+
+Activate you application by executing it in the `main` method:
+
+```java
+    public static void main(String[] args) {
+        OSHDBApplication.run(MyApplication.class, args);
+    }
+```
+
+The `run` method now replaces your `main` method. You can add any functionality starting from there. The return value represents the status code. 0 signals a successful execution and should be the default.
+
+```java
+    @Override
+    protected int run(OSHDBConnection oshdb) throws Exception {
+        // your code goes here
+        return 0;
+    }
+```
+
+Other output (e.g. from the OSHDB query itself), needs to be handled within the method like writing it to a file, storing it in a database or printing it to the screen.
+
+To provide input to your application or query you can equally use the methods' body. Alternatively you can extend the predefined [picocli](https://picocli.info/)-based CLI with your own arguments
+
+```java
+    @CommandLine.Option(names = {"--myInput"}, description = "custom CLI input")
+    protected String input;
+
+    @Override
+    protected int run(OSHDBConnection oshdb) throws Exception {
+        System.out.println(this.input);
+```
+
+To run your application/query execute it in a command line. To configure the oshdb connection (see the [README.md](README.md)) you can either use the provided CLI options (`--oshdb`, `--prefix`, `--keytables`), provide a `.properties` file (`--props`), or both. Any given CLI option will overwrite the properties provided in the file (if both parameters are present).
+
+## Example
+
+```java
+package mypackage;
+
+import org.heigit.ohsome.oshdb.OSHDBBoundingBox;
+import picocli.CommandLine;
+
+public class MyApplication extends OSHDBApplication {
+
+    public static void main(String[] args) {
+        OSHDBApplication.run(MyApplication.class, args);
+    }
+
+    @CommandLine.Option(defaultValue="2018-05-01", names = {"--ts"}, description = "target timestamp, default=${DEFAULT-VALUE}")
+    protected String ts;
+
+    @Override
+    protected int run(OSHDBConnection oshdb) throws Exception {
+        OSHDBBoundingBox bbox = OSHDBBoundingBox.bboxWgs84Coordinates(8.651133, 49.387611, 8.6561, 49.390513);
+
+        Integer result = oshdb.getSnapshotView()
+                .areaOfInterest(bbox)
+                .filter("type:node")
+                .timestamps(this.ts)
+                .count();
+
+        System.out.println(result);
+
+        return 0;
+    }
+
+}
+```
+
+Run it e.g. with `mvn exec:java -Dexec.mainClass="mypackage.MyApplication" -Dexec.args="--oshdb h2:/path/to/file.oshdb.mv.db"`.

documentation/manual/helpers/OSHDBDriver.md

@@ -0,0 +1,79 @@
+# OSHDBDriver
+## Installation
+
+Replace your OSHDB dependency with the following:
+
+```xml
+<dependency>
+   <groupId>org.heigit.ohsome</groupId>
+   <artifactId>oshdb-database-driver</artifactId>
+   <version>0.7.2</version>
+</dependency>
+```
+
+## Usage
+
+To connect to the OSHDB using the OSHDBDriver you first have to define the configuration like:
+
+```java
+    Properties props = new Properties();
+    // For H2:
+    props.setProperty("oshdb", "h2:PATH_TO_H2");
+    // Or for Ignite:
+    props.setProperty("oshdb","ignite:PATH_TO_CFG");
+    props.setProperty("keytables","jdbc:postgresql://localhost/keytables-${prefix}?user=ohsome&password=secret");
+```
+
+Alternatively you can read-in a `….properties` file with this information e.g. like so:
+
+```java
+    Properties props = new Properties();
+    try(Reader reader = new FileReader("oshdb.properties")){
+          props.load(reader);
+    }
+```
+
+Then connect to OSHDB using the driver like:
+
+```java
+    OSHDBDriver.connect(props, (OSHDBConnection oshdb) -> {
+        // your oshdb code goes here
+        return 0;
+    });
+```
+
+## Example
+
+```java
+package mypackage;
+
+import java.util.Properties;
+import org.heigit.ohsome.oshdb.OSHDBBoundingBox;
+import org.heigit.ohsome.oshdb.helpers.OSHDBDriver;
+
+public class OSHDBDriverExample {
+
+    public static void main(String[] args) throws Exception {
+        Properties props = new Properties();
+        // For H2:
+        props.setProperty("oshdb", "h2:PATH_TO_H2");
+        // Or for Ignite:
+        props.setProperty("oshdb", "ignite:PATH_TO_CFG");
+        props.setProperty("keytables", "jdbc:postgresql://localhost/keytables-global?user=ohsome&password=secret");
+
+        OSHDBDriver.connect(props, oshdb -> {
+            OSHDBBoundingBox bbox = OSHDBBoundingBox.bboxWgs84Coordinates(8.651133, 49.387611, 8.6561, 49.390513);
+
+            Integer result = oshdb.getSnapshotView()
+                    .areaOfInterest(bbox)
+                    .filter("type:node")
+                    .timestamps("2018-05-01")
+                    .count();
+
+            System.out.println(result);
+            return 0;
+        });
+    }
+
+}
+```

documentation/manual/helpers/README.md

@@ -0,0 +1,25 @@
+# OSHDB Helpers
+
+Simple OSHDB connection helpers that automatically open an Ignite or H2 connection, depending on the input.
+
+Two functionalities are available:
+
+ - The [OSHDBApplication](OSHDBApplication.md) (recommended usage) provides a full [Spring boot](https://spring.io/projects/spring-boot)-like application including a CLI. "Just add" your OSHDB functionality to create a usage-ready application.
+ - The [OSHDBDriver](OSHDBDriver.md) provides a static method that exhibits an OSHDB connection to a respective `Consumer`. It leaves you with all setup work for your application and will only handle the OSHDB connection part.
+
+
+##  Configuration
+
+Both functionalities will need the following configuration options. The details how to specify them will be discussed in the respective subsection.
+
+ - `oshdb`
+    - for a connection to an H2-file this is the absolute path to the H2-file prefixed by `h2:` like `h2:/path/to/file.oshdb.mv.db`
+    - for a connection to an Ignite cluster this is the absolute path to an ignite-config.xml file prefixed by `ignite:` like `ignite:/path/to/file.xml`
+ - `prefix` (optional)
+    - a string prefixed to database objects to allow multiple data versions to co-exist in the backend. It is (only) necessary if you want to access a legacy OSHDB ignite cluster. You will be notified about this once you get access to an H2-file or the Ignite cluster.
+ - `keytables` (optional for H2)
+    - a JDBC string defining a connection to the [keytables](../data-model.md#keytables) linked to the targeted OSHDB. H2 files normally self contain the keytables where the tools are able to find them.
+ - `multithreading` (optional for H2)
+    - a boolean parameter for jdbc based connections (i.e. H2) if multithreading should be enabled during processing. 
+
+ Note that any `${some-property}` (e.g. `${prefix}`) within these property strings will be automatically replaced by the respective property value. So you can safely include these placeholders into your keytables URL (or any other property), if needed.

oshdb-helpers/oshdb-application-template/pom.xml

@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <parent>
+    <groupId>org.heigit.ohsome</groupId>
+    <artifactId>oshdb-helpers</artifactId>
+    <version>1.0.0-SNAPSHOT</version>
+  </parent>
+
+  <artifactId>oshdb-application-template</artifactId>
+  <name>OSHDB Application Template</name>
+  <description>TODO</description>
+
+  <properties>
+  </properties>
+
+  <dependencies>
+    <dependency>
+      <groupId>${project.groupId}</groupId>
+      <artifactId>oshdb-database-driver</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>info.picocli</groupId>
+      <artifactId>picocli</artifactId>
+      <version>4.6.3</version>
+    </dependency>
+  </dependencies>
+</project>

oshdb-helpers/oshdb-application-template/src/main/java/org/heigit/ohsome/oshdb/helpers/applicationtemplate/OSHDBApplication.java

@@ -0,0 +1,105 @@
+package org.heigit.ohsome.oshdb.helpers.applicationtemplate;
+
+import java.nio.file.Path;
+import java.util.Properties;
+import java.util.concurrent.Callable;
+import org.heigit.ohsome.oshdb.helpers.db.OSHDBConnection;
+import org.heigit.ohsome.oshdb.helpers.db.OSHDBDriver;
+import picocli.CommandLine;
+import picocli.CommandLine.ArgGroup;
+import picocli.CommandLine.Command;
+import picocli.CommandLine.Option;
+
+/**
+ * Class that can be used to bootstrap and launch a OSHDB application from a Java main method. By
+ * default will perform the following steps to bootstrap your application:
+ * <ul>
+ * <li>parse command line parameters</li>
+ * <li>connect to specified oshdb</li>
+ * </ul>
+ * In most circumstances the static run(Class, String[]) method can be called directly from your
+ * main method to bootstrap your application:
+ *
+ * <pre>
+ * {@code
+ * public class MyApplication extends OSHDBApplication {
+ *
+ *    public static void main(String[] args){
+ *      OSHDBApplication.run(MyApplication.class, args);
+ *    }
+ *
+ *    protected int run(OSHDBConnection oshdb){
+ *     oshdb.getSnapshotView()
+ *         .areaOfInterest((Geometry & Polygonal) areaOfInterest)
+ *         .timestamps(tstamps)
+ *         .osmTag(key)
+ *         ...
+ *     return 0; // exitCode
+ *    }
+ * }}
+ * </pre>
+ */
+@Command(mixinStandardHelpOptions = true, sortOptions = false)
+public abstract class OSHDBApplication implements Callable<Integer> {
+
+  @ArgGroup(exclusive = false, multiplicity = "1")
+  ConfigOrUrl configOrUrl;
+
+  static class ConfigOrUrl {
+    @Option(names = {"--props"}, description = ".properties file path")
+    protected Path config;
+
+    @Option(names = {"--oshdb"}, description = "oshdbUrl (ignite:...|h2:...)")
+    protected String oshdbUrl;
+  }
+
+  @Option(names = {"--keytables"}, description = "keytablesUrl jdbc:...")
+  protected String keytableUrl;
+
+  @Option(names = {"--prefix"}, description = "prefix to use for ignite")
+  protected String prefix;
+
+  @Option(names = {
+      "--multithreading"}, description = "for jdbc based connections", negatable = true)
+  protected Boolean multithreading = null;
+
+  protected Properties props;
+
+  @SuppressWarnings("java:S112")
+  protected abstract int run(OSHDBConnection oshdb) throws Exception;
+
+  /**
+   * Method to be called from the implemented application.
+   *
+   * @param clazz Class that will be started.
+   * @param args main args
+   * @throws Exception from application
+   */
+  @SuppressWarnings("java:S112")
+  public static void run(Class<? extends OSHDBApplication> clazz, String[] args)
+      throws Exception {
+    var app = clazz.getDeclaredConstructor().newInstance();
+    int exit = new CommandLine(app).execute(args);
+    System.exit(exit);
+  }
+
+  @Override
+  public Integer call() throws Exception {
+    props = new Properties();
+    PropsUtil.load(props, configOrUrl.config);
+    PropsUtil.set(props, OSHDBDriver.OSHDB_PROPERTY_NAME, configOrUrl.oshdbUrl);
+    PropsUtil.set(props, OSHDBDriver.KEYTABLES_PROPERTY_NAME, keytableUrl);
+    PropsUtil.set(props, OSHDBDriver.PREFIX_PROPERTY_NAME, prefix);
+    PropsUtil.set(props, OSHDBDriver.MULTITHREADING_PROPERTY_NAME, multithreading);
+    return OSHDBDriver.connect(props, this::setAndRun);
+  }
+
+  private int setAndRun(OSHDBConnection connection) throws Exception {
+    configOrUrl.oshdbUrl = PropsUtil.get(props, OSHDBDriver.OSHDB_PROPERTY_NAME).orElseThrow();
+    keytableUrl = PropsUtil.get(props, OSHDBDriver.KEYTABLES_PROPERTY_NAME).orElse(null);
+    prefix = PropsUtil.get(props, OSHDBDriver.PREFIX_PROPERTY_NAME).orElse("");
+    multithreading = Boolean.valueOf(
+        PropsUtil.get(props, OSHDBDriver.MULTITHREADING_PROPERTY_NAME).orElse("false"));
+    return run(connection);
+  }
+}