Future of schema definitions

[msbarry]

Currently you need to change Java code and recompile the tool to make change the map schema that controls how input source features map to output vector tile features. How should these custom profiles be defined going forward?

Please upvote your preference, reply to an answer to discuss in more detail, or add another suggested answer for different ideas.

Thank you! 🙏

[msbarry]

Schemas should be driven by config (yaml/json) files. A similar example in the wild today looks like: https://docs.mapbox.com/mapbox-tiling-service/reference/

[msbarry]

Any config file format would be a clean-sheet design to control the existing behavior of the different components of the architecture although they likely would use embedded maplibre gl expressions to control more complex logic or attributes that change based on zoom level, since people using vector tiles are already likely to be familiar with that style expression syntax.

[nyurik]

I feel expressions don't scale as well as a scripting solution with the project growth, but I might be wrong. How difficult is it to implement a scripting solution? Also, I feel we are trying to decide without enough data, i.e. performance implications and the usage complexity with the simplest vs most difficult known cases.

[msbarry]

With yaml/json, the idea is if you need something custom we’d try to express that custom functionality as a reusable capability and compile in the code to implement it, but then to use it you just add config. For example: "merge nearby polygons" "connect linestrings".

A scripting solution would involve pulling in an interpreter for the language of choice, then hooking in the APIs and entrypoints to it - shouldn't be too difficult, but either route would definitely need some investigation on performance.

[bchapuis]

Personally, I really like json configuration files as they are easy to parse and their content is either correct or wrong. Scripting languages require to deal with a lot of corner cases, such as infinite loops, etc. I feel that a simple configuration format would allow us to focus more on the definition of the schema and the internal steps of the data transformation pipeline (e.g. geometry simplification, etc.) needed to generate vector tiles.

I also believe that this kind of format can be verbose and hard to scale (e.g. large mapbox stylesheets). However, I guess that the generation of the json config can still be done with an ad-hoc scripting language chosen by the users.

[msbarry]

Schemas should be driven by a scripting language (i.e. lua) A similar example in the wild today looks like: https://github.com/systemed/tilemaker/blob/master/resources/process-openmaptiles.lua

[msbarry]

Defining schemas in Java is fine (for speed and ide support)

[nyurik]

this approach might actually also be required, but rather than being "the only one", it will be more like a plugin module that declarative or scripting solution would call out to perform complex work.

[msbarry]

Yeah agreed... any functionality exposed through scripting language or config file would have to use APIs compiled into the tool, so with the current implementation you implement a profile in Java, then create a new MyMapSchema(), a scripting or config based language would delegate to what's defined in the file, something like new LuaSchema("./my_map_schema.lua") or new YamlSchema("./my_map_schema.yaml")

[bchapuis]

I guess that the configuration class may actually be (re)compiled and loaded at runtime by planetiler (e.g. with the compiler API and a custom class loader). I recently played with Amazon CDK, which uses familiar programming languages (instead of json/yaml) to configure cloud resources and got pretty convinced by this approach.

[wipfli]

I am playing around right now with Java and it is a lot of fun :)

[wipfli]

I am just hacking code into this file:

planetiler/planetiler-basemap/src/main/java/com/onthegomap/planetiler/basemap/layers/Power.java

Lines 21 to 29 in 37571d6

	public void processAllOsm(SourceFeature feature, FeatureCollector features) {
	// basic implementation, more ideas here: https://github.com/ache051/openmaptiles/tree/power_facilities/layers/power
	if (feature.canBeLine() && feature.hasTag("power", "line")) {
	features.line(LAYER_NAME)
	.setBufferPixels(BUFFER_SIZE)
	.setMinZoom(14)
	.setAttr("class", "line");
	}
	}

And vscode gives me all the nice auto completion and hover function info and so on. Doing customization in code seems like a cool option...

[nyurik]

General discussion. Please post anything not specific to any of the above as a reply to this topic, unless you want to propose another big topic.

[ZeLonewolf]

I don't have a strong opinion here, but I would encourage the decision to be made on the basis of what would be the most effective implementation that maximizes both flexibility and performance. On that basis, doing them in Java is probably going to be the most performant, and I note that a Java solution can still be backed by configuration files. So perhaps there is a Java-based schema with a defined set of interfaces that pulls from a set of yaml/json files for its behavior.

I would not be a fan of the lua-based approach because I personally find the language hard to understand, but I would happily bite the bullet and learn it if it's truly the best option. Since lua is used by osm-carto, there's at least some level of community understanding of how to develop in it that can be leveraged.

[msbarry]

One thorny issue to handle is state and joins. A few examples from openmaptiles that required Java layers in planetiler to store state:

• reconstructing country boundary polygons to determine which side of each boundary way is on the inside or outside
• mountain_peaks that fall inside US polygon from natural earth get the customary_ft tag
• places points in openstreetmap used their corresponding place points in natural earth for name, rank, and determining min zoom
• transportation junction nodes get their class from the class of the way they are a part of
• motorways inside the great britain polygon from natural earth determine if they are a motorway based on a regex on name instead of relation membership
• ways in general often need information about the relations they are a part of (road networks, etc.)
• some times all ways in a relation need to be processed together (i.e. multipolygons, and low-zoom rivers to ensure they are long enough)
• we determine what water points in OSM are "important" by if they have a corresponding match from natural earth by name / name prefix

a few of these should be extracted as high level functionality that schema writers don't even need to think about (polygon border left/right, accessing information about relations a way is in, multipolygons) but the others seem to be legit special-case joins between data sources...

[ZeLonewolf]

I personally thing the customary_ft processing is weird and would happily turn it off to save on compute time. :-D But if we have that processing today, a declarative configuration could just point to those specific routines.

[wipfli]

When I started exploring OSM data, I found https://overpass-turbo.eu/ helpful. For example, to get data on power lines, this query can be used in overpass turbo:

/*
This has been generated by the overpass-turbo wizard.
The original search was:
“power = line”
*/
[out:json][timeout:25];
// gather results
(
  // query part for: “power=line”
  node["power"="line"]({{bbox}});
  way["power"="line"]({{bbox}});
  relation["power"="line"]({{bbox}});
);
// print results
out body;
>;
out skel qt;

which shows this map:

The overpass query language is maybe not suited for planetiler, but from a functional point of view, it did what I wanted: select some OSM features and get the vector data.

[nyurik]

After some thinking, I would like to propose an in-between approach. Declarative configurations are much easier for users and simpler to implement. Scripting is something most users want once they hit declarative limitations. So, let's have it both ways (kinda):

• only support scripts
• simple example script will just returns a static configuration - i.e. a big json/yaml-like structure that describes what is needed, without any functions or any other magic
• eventually we can extend this to allow individual branches of that config to include functions, complex expressions, etc.

Performance-wise this will be negligible -- the get_script() will only be called once to get the profile data. If we use some well known language like JavaScript or Lua (not my fav, but common), we can have multiple implementations of planetiler use it without changes.

// default example - all planetiler runs must use a file like this to process data
getConfig() {
  return {
    layers: [
      {
        name: "water",
        // no idea how to configure layers yet
        // it will contain fields and their tag mappings, data sources, etc
      }
    ]
  };
}

Complex example is similar, only this time users could add extra functions and lambda expressions:

// Simple case of this could still be declarative for performance reasons, TBD
getPoiClass(tags) {
  if (tags.subclass === 'station' && tags.mapping_key === 'aerialway') return 'aerialway';
  ...
}

getConfig() {
...
    layers: [
      {
        name: 'poi',
        fields: [
          name: 'class',
          dynamic: getPoiClass,  // a reference to the function above
        ]
...
}

[nyurik]

@msbarry agree - there would need to be a sort of "prefiltering" - some way to quickly get rid of irrelevant features before the phase 2 -- in-depth evaluation/conversion. Using the maplibre styling lang would ensure that everything used in phase 2 was correctly prefiltered in phase 1. But I am not sure such "consistency testing" of the user config is worth implementing a much more complicated system. Instead, we could encourage proper scripting behavior, e.g. for some cases of conversion, could use this approach.

const subclassMap = {
  accessories: 'shop',
  antiques: 'shop',
  townhall: 'town_hall',
  courthouse: 'town_hall',
  ...
}
convertSubclass(tags) {
  return subclassMap[tags.subclass];
}
const tagHandlers = {
  subclass: convertSubclass,
  ...
}
getPrefilterTags() {
  // This ensures the list of pre-filter tags is always the same as the one needed
  return Object.keys(tagHandlers);
}
getFeatureTags(tags) {
  // This is called later when actually forming the output tags
  for (const tag of Object.keys(tags)) {
    const handler = tagHandlers[tag];
    if (handler) result = {...result, handler(tags)};
  }
  return result;
}

P.S. I suspect it will be a tiny fraction of time to implement a script engine support vs a full parser for maplibre evaluation. Once done, we can always add the maplibre styling DSL too - as it won't conflict with the existing system, but rather add yet another way to declare things.

[flother]

I like the idea of declarative config with a fallback to scripting when things get too complex.

On a general point it's worth remembering that we shouldn't talk purely in terms of tags and tagging because Planetiler should eventually support more than just OpenStreetMap PBFs as data sources (see #46). If GeoJSON, FlatGeobuf, and Geopackage are supported input formats then any schema definition should talk about columnar data more generally rather than just OSM tags. (In fact, values in GeoJSON feature properties can be JSON objects and arrays, not just strings/numbers/booleans.)

On the topic of the Maplibre Style Spec, I find writing the style JSON (by hand) to be the least pleasant part of making a vector map. It's possible that it's due to the lack of examples in the documentation, but expressions are fiddly and I find the syntax hard to remember. I like the idea of config-based layers but I'm not sure the style spec is the best source for inspiration. (I can't think of any better suggestions though!)

[msbarry]

Good points, I agree on the style spec - for onthegomap I construct the JSON with javascript and have a whole library of functions built up to map the low level expressions to high level building blocks I need. Another example in the wild for expressions embedded in yaml would be github action expressions ?

[ZeLonewolf]

Just remember that style JSON is still JSON, and thus can also be constructed as actual JavaScript objects which are well suited for dealing with repetition. We do that fairly extensively in OpenStreetMap Americana to deal with the repetitive nature of style development, for example in dealing with roads and layers.

[bchapuis]

It is worth mentionning that java now has nice scripting capabilities comming from the GraalVM project (javascript, python, R, etc.). In my understanding, these scripts also get JIT compiled. However, I'm not sure if this already works on OpenJDK.

[klokan]

Notes from today's @openmaptiles meeting with @msbarry @nyurik @ZeLonewolf and others:

A hybrid of YAML config + Java/Rust/JavaScript code which is layer specific may be the right answer here - allowing the official OpenMapTiles compatibility. Currently planetiler is already generating the code from some of the YAML files from OMT.

A very basic use-case if you want to just generate vector tiles from a shapefile can be done only with a single YAML - with a code similar to https://github.com/klokantech/vector-tiles-sample/blob/master/countries.tm2source/data.yml

More complex schema can be done with preprocessor - as OMT is done with "generate-*" commands. On OpenMapTiles it would mean port the code from SQL to JAVA or another picked language - and the layer structure could be preserved with additional benefits and maintainability. Example: https://github.com/openmaptiles/openmaptiles/tree/master/layers/aeroway

[bdon]

Tangentially related to planetiler - I've made a quick experiment for tileset definitions in the Starlark language, the benefit being that it's a constrained subset of Python that should be familiar to developers.

The definition file would look something like this: https://github.com/bdon/protomaps-experiments/blob/master/cartographic-scripting/tileset_definition.star

I've also attempted an exhaustive list of existing textual formats that attempt this task, both declarative and imperative - GL expressions, JSON, YAML, and more: https://github.com/bdon/protomaps-experiments/tree/master/cartographic-scripting#other-designs