Support .mvt vector tile format

[thomasneirynck]

Mapbox vector tile format (https://github.com/mapbox/vector-tile-spec/) is a common industry spec to deliver geospatial data to clients.

It is particularly useful to web-clients:

• the geometries within the tiles are scale-dependent
  • features with a small footprint that would be invisible at zoomed-out scales are excluded from the tiles
  • geometries are simplified to match the desired scale-level (ie. complex lines are simplified so there are no overlapping vertices in view-coordinates)
• uses efficient file-format (protobuf) (as opposed to text-based json)

This cuts down on the size of the response that is sent over the wire. Since the data is tiled, the client (e.g. browser) can also cache requests client-side and re-use earlier results as users pan and zoom.

It would be useful if Elasticsearch could return the result of a _search (either with individual documents or an aggregation) as a vector tile.

Kibana Maps is working on introducing .mvt support for 3rd party services and ES-data (elastic/kibana#58519). In an initial phase, the tile-generation from ES-data would happen in the Kibana-server. Over time, it would be great to push down more of this lower-level functionality into ES itself.

cc @giladgal @kmartastic @nknize

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Geo)

[thomasneirynck]

cc @nknize

[florinvirdol]

Hi again, @thomasneirynck !
@nickpeihl advised me to put this question here from 58519.

When you guys have some time, I have some questions, please.

As a bit of context, for my web app, on the Angular frontend, on top of Google Maps APIs, i'm using Uber's deck.gl library for better rendering geo-spatial data (GeoJson layers and MVT layers).
For the deck.gl MVT layers I was using PostgreSQL PostGIS as Vector tile server.

Now, I migrated away from PostGIS and I'm storing the geo-spatial data (collection of shapes and points) in ElasticSearch for faster indexing and querying.

I was wondering if with this initiative or in the near future (in a roadmap), will I able to use ElasticSearch or Kibana as a Vector tile server in order to get geo-spatial data (normal for shapes and clustered for points) only for the current map viewport (based on zoom and pan / drag on the map) and in the Mapbox .mvt / pbf formats?

I saw you guys did a wonderful job of implementing something similar into Kibana Maps (serving as Vector Tiles geo-spatial data from ElasticSearch: normal shapes and clustered points).
Until any ElasticSearch implementation, do you expose and can I use any APIs from Kibana?

I'm looking forward to your response!
Thank you very much!

[thomasneirynck]

hi @florinvirdol ,

we are currently investigating if/how/when to further build out the mvt support in the ELK stack.

imho you're right that there's basically two tracks to think about:

• Elasticsearch supports mvt format natively, through a dedicated endpoint. This is basically this ticket.
• Kibana is a middle-ware to publish mvt services, e.g. internally using this ES-mvt generation

No decisions around timing have been made around this point.

Note that the Kibana mvt server functionality is currently in beta (ie. elastic/kibana#58519). The .mvt generation happens on the Kibana backend, and for now it only supports the Kibana Maps application. There is still edge-cases to work through. You can read more about those existing gaps here elastic/kibana#79868.

[giladgal]

Hi @florinvirdol ,
To add to what @thomasneirynck wrote, we are currently working on a prototype to do what you described within Elasticsearch. We will update as we progress. It’s definitely something we are examining for our future roadmap.

[monfera]

Great idea :elasticheart: looking fwd to this

[thomasneirynck]

Closed with #73872

Kibana Maps equivalent closed as well: elastic/kibana#109954