Skip to content

Commit

Permalink
Update docstrings, logging, and args name to not be MBTiles specific [o…
Browse files Browse the repository at this point in the history
  • Loading branch information
bdon committed Jan 13, 2023
1 parent 0e0685a commit 6d47077
Show file tree
Hide file tree
Showing 7 changed files with 30 additions and 31 deletions.
10 changes: 5 additions & 5 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Planetiler builds a map in 3 phases:
intermediate files on disk
2. [Sort Features](#2-sort-features) by tile ID
3. [Emit Vector Tiles](#3-emit-vector-tiles) by iterating through sorted features to group by tile ID, encoding, and
writing to the output MBTiles file
writing to the output tile archive

User-defined [profiles](#profiles) customize the behavior of each part of this pipeline.

Expand Down Expand Up @@ -96,15 +96,15 @@ of the intermediate features using a worker thread per core:

## 3) Emit Vector Tiles

[MbtilesWriter](planetiler-core/src/main/java/com/onthegomap/planetiler/mbtiles/MbtilesWriter.java) is the main driver.
[TileArchiveWriter](planetiler-core/src/main/java/com/onthegomap/planetiler/writer/TileArchiveWriter.java) is the main driver.
First, a single-threaded reader reads features from disk:

- [ExternalMergeSort](planetiler-core/src/main/java/com/onthegomap/planetiler/collection/ExternalMergeSort.java) emits
sorted features by doing a k-way merge using a priority queue of iterators reading from each sorted chunk
- [FeatureGroup](planetiler-core/src/main/java/com/onthegomap/planetiler/collection/FeatureGroup.java) collects
consecutive features in the same tile into a `TileFeatures` instance, dropping features in the same group over the
grouping limit to limit point label density
- Then [MbtilesWriter](planetiler-core/src/main/java/com/onthegomap/planetiler/mbtiles/MbtilesWriter.java) groups tiles
- Then [TileArchiveWriter](planetiler-core/src/main/java/com/onthegomap/planetiler/writer/TileArchiveWriter.java) groups tiles
into variable-sized batches for workers to process (complex tiles get their own batch to ensure workers stay busy
while the writer thread waits for finished tiles in order)

Expand All @@ -116,9 +116,9 @@ Then, process tile batches in worker threads (default 1 per core):
- gzip each encoded tile
- Pass the batch of encoded vector tiles to the writer thread

Finally, a single-threaded writer writes encoded vector tiles to the output MBTiles file:
Finally, a single-threaded writer writes encoded vector tiles to the output archive format:

- Create the largest prepared statement supported by SQLite (999 parameters)
- For MBTiles, create the largest prepared statement supported by SQLite (999 parameters)
- Iterate through finished vector tile batches until the prepared statement is full, flush to disk, then repeat
- Then flush any remaining tiles at the end

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
/**
* A collection of utilities for merging features with the same attributes in a rendered tile from
* {@link Profile#postProcessLayerFeatures(String, int, List)} immediately before a tile is written to the output
* mbtiles file.
* archive.
* <p>
* Unlike postgis-based solutions that have a full view of all features after they are loaded into the database, the
* planetiler engine only sees a single input feature at a time while processing source features, then only has
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -452,30 +452,29 @@ public Planetiler setProfile(Function<Planetiler, Profile> profileProvider) {
}

/**
* Sets the location of the output {@code .mbtiles} file to write rendered tiles to. Fails if the file already exists.
* Sets the location of the output archive to write rendered tiles to. Fails if the archive already exists.
* <p>
* To override the location of the file, set {@code argument=newpath.mbtiles} in the arguments.
* To override the location of the file, set {@code argument=newpath} in the arguments.
*
* @param argument the argument key to check for an override to {@code fallback}
* @param fallback the fallback value if {@code argument} is not set in arguments
* @return this runner instance for chaining
* @see MbtilesWriter
* @see TileArchiveWriter
*/
public Planetiler setOutput(String argument, Path fallback) {
this.output = arguments.file(argument, "mbtiles output file", fallback);
this.output = arguments.file(argument, "output tile archive", fallback);
return this;
}

/**
* Sets the location of the output {@code .mbtiles} file to write rendered tiles to. Overwrites file if it already
* exists.
* Sets the location of the output archive to write rendered tiles to. Overwrites file if it already exists.
* <p>
* To override the location of the file, set {@code argument=newpath.mbtiles} in the arguments.
* To override the location of the file, set {@code argument=newpath} in the arguments.
*
* @param argument the argument key to check for an override to {@code fallback}
* @param fallback the fallback value if {@code argument} is not set in arguments
* @return this runner instance for chaining
* @see MbtilesWriter
* @see TileArchiveWriter
*/
public Planetiler overwriteOutput(String argument, Path fallback) {
this.overwrite = true;
Expand All @@ -484,7 +483,7 @@ public Planetiler overwriteOutput(String argument, Path fallback) {

/**
* Reads all elements from all sourced that have been added, generates map features according to the profile, and
* writes the rendered tiles to the output mbtiles file.
* writes the rendered tiles to the output archive.
*
* @throws IllegalArgumentException if expected inputs have not been provided
* @throws Exception if an error occurs while processing
Expand Down Expand Up @@ -537,7 +536,7 @@ public void run() throws Exception {
}
}
LOGGER.info(" sort: Sort rendered features by tile ID");
LOGGER.info(" mbtiles: Encode each tile and write to {}", output);
LOGGER.info(" archive: Encode each tile and write to {}", output);
}

// in case any temp files are left from a previous run...
Expand Down Expand Up @@ -574,7 +573,7 @@ public void run() throws Exception {
stats.monitorFile("nodes", nodeDbPath);
stats.monitorFile("features", featureDbPath);
stats.monitorFile("multipolygons", multipolygonPath);
stats.monitorFile("mbtiles", output);
stats.monitorFile("archive", output);

for (Stage stage : stages) {
stage.task.run();
Expand Down Expand Up @@ -638,7 +637,7 @@ private void checkDiskSpace() {
readPhase.addDisk(featureDbPath, featureSize, "temporary feature storage");
writePhase.addDisk(featureDbPath, featureSize, "temporary feature storage");
// output only needed during write phase
writePhase.addDisk(output, outputSize, "mbtiles output");
writePhase.addDisk(output, outputSize, "archive output");
// if the user opts to remove an input source after reading to free up additional space for the output...
for (var input : inputPaths) {
if (input.freeAfterReading()) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
* (i.e. one per layer) and forwarding each element/event to the handlers that care about it.
*/
public interface Profile {
// TODO might want to break this apart into sub-interfaces that ForwardingProfile (and MbtilesMetadata) can use too
// TODO might want to break this apart into sub-interfaces that ForwardingProfile (and TileArchiveMetadata) can use too

/**
* Allows profile to extract any information it needs from a {@link OsmElement.Node} during the first pass through OSM
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@ private void writeTileFeatures(int zoom, long id, FeatureCollector.Feature featu
// Store lines with extra precision (2^scale) in intermediate feature storage so that
// rounding does not introduce artificial endpoint intersections and confuse line merge
// post-processing. Features need to be "unscaled" in FeatureGroup after line merging,
// and before emitting to output mbtiles.
// and before emitting to the output archive.
scale = Math.max(config.maxzoom(), 14) - zoom;
// need 14 bits to represent tile coordinates (4096 * 2 for buffer * 2 for zigzag encoding)
// so cap the scale factor to avoid overflowing 32-bit integer space
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ default Timers.Finishable startStage(String name, boolean log) {
/** Records that an input element was processed and emitted some output features in {@code layer}. */
void processedElement(String elemType, String layer);

/** Records that a tile has been written to the mbtiles output where compressed size is {@code bytes}. */
/** Records that a tile has been written to the archive output where compressed size is {@code bytes}. */
void wroteTile(int zoom, int bytes);

/** Returns the timers for all stages started with {@link #startStage(String)}. */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -264,23 +264,23 @@ void testMetadataButNoPoints() throws Exception {
void testOverrideMetadata() throws Exception {
var results = runWithReaderFeatures(
Map.of(
"mbtiles_name", "mbtiles_name",
"mbtiles_description", "mbtiles_description",
"mbtiles_attribution", "mbtiles_attribution",
"mbtiles_version", "mbtiles_version",
"mbtiles_type", "mbtiles_type"
"tileset_name", "override_name",
"tileset_description", "override_description",
"tileset_attribution", "override_attribution",
"tileset_version", "override_version",
"tileset_type", "override_type"
),
List.of(),
(sourceFeature, features) -> {
}
);
assertEquals(Map.of(), results.tiles);
assertSubmap(Map.of(
"name", "mbtiles_name",
"description", "mbtiles_description",
"attribution", "mbtiles_attribution",
"version", "mbtiles_version",
"type", "mbtiles_type"
"name", "override_name",
"description", "override_description",
"attribution", "override_attribution",
"version", "override_version",
"type", "override_type"
), results.metadata);
}

Expand Down

0 comments on commit 6d47077

Please sign in to comment.