[Maps] Add mvt format for ES-doc sources #74319

thomasneirynck · 2020-08-04T23:21:31Z

WIP

Partially addresses #58519

Summary

This adds an .mvt vector tile option for Elasticsearch geo_shape documents.

This allows display of a large amount of documents. The display of the geometries is optimized for scale. When zoomed in, the full geometry is returned, when zoomed out, a simplified geometry is shown.

`

Introduce a new scaling-type "mvt tiles"

Extend ESSearchSource to implement

This is different from the original POC (#57248), which introduced a new source type altogether. There are some advantages to just extending the existing ESSearchSource.

less code overall
toggling scaling-type can re-use the same infrastructure which was in place for blended-layers, without having to accommodate "source-type-swapping"

Add new `getTile` route-backend

The URL encodes all relevant state. By doing so, the existing logic in TiledVectorLayer that determines when and how to swap mb-sources remains functional.

Introduce the concept of a `too many features` bounds

When the number of features exceeds the ES-limit, the tile contains a bounds of where the features are.

This is somewhat a "one-off"-addition, but imho it is relevant for the .mvt scaling type. This bounds indicates the user where the data is, but does not show the actual geometries. This is a similar approaches exist in other toolkits. The goal is to give users an indication of where to zoom-in.

e.g. consider building footprints for an entire US-state. There are too many to display at once (in the millions), and even if Maps could display them all at once, at zoomed out scales they would be much smaller than a pixel to see.

This needs design work. The current hash-pattern is somewhat ugly.

Future work

For feature completeness, following functionality needs to be added: (excluded from this PR, to keep changes to a minimum).

add support for geo_point
add support for grid/cluster layers as tiles

Todo

unit tests for the new contracts and UX
API integration tests for the new route
functional tests for round-trip verification
ts type check
Language
Known bugs
- should only include necessary field-properties
- time based data? allow/disallow?

thomasneirynck · 2020-08-14T06:45:30Z

x-pack/plugins/maps/public/classes/layers/tiled_vector_layer/tiled_vector_layer.tsx

@@ -160,6 +160,11 @@ export class TiledVectorLayer extends VectorLayer {
      return false;
    }

+    if (!mbTileSource.tiles) {


This is because the layer-type and the underlying representation in mb changes when toggling between scaling-types. It needs to check it the mb-source is compatible with the layer-type.

thomasneirynck · 2020-08-14T06:56:46Z

x-pack/plugins/maps/public/classes/layers/vector_layer/vector_layer.js

@@ -836,6 +867,18 @@ export class VectorLayer extends AbstractLayer {
        type: 'geojson',
        data: EMPTY_FEATURE_COLLECTION,
      });
+    } else if (mbSource.type !== 'geojson') {


Needs to remove incompatible source-types and layers

thomasneirynck · 2020-08-14T07:01:10Z

x-pack/plugins/maps/public/connected_components/map/mb/view.js

@@ -143,6 +147,15 @@ export class MBMap extends React.Component {
        mbMap.addControl(new mapboxgl.NavigationControl({ showCompass: false }), 'top-left');
      }

+      const hatchImageBase64 =
+        'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAASwAAADhCAYAAAByfIirAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAAKYUAACmFAY9V/uEAAAAYdEVYdFNvZnR3YXJlAHBhaW50Lm5ldCA0LjAuNWWFMmUAAAlLSURBVHhe7dtRCuNmEoXREGb/S5h9ZHWZkhiBQXVjrB9cinw+aEM3nFwRCj9Zf1R/15+/6s9/tr+86c/689/6s5njD9vHsin2uvU/KsSyKXbOXobb39lzLJti1+3bEtz+/V0sm2LZ1GU7MlqxbIpl20ZGK5ZNsWzbyGjFsimWbRsZrVg2xbJtI6MVy6ZYtm1ktGLZFMu2jYxWLJti2baR0YplUywbmxhln223XzJP7LLPtvsv5K/AqQdm72+3o9reFbtip56Zvb897uqro+yzrS8rNrViX+9q/1h5e/obD8z+O6276mOvW0cVYtesu+pj16yjamLXrbs6x67bt93tgdlzLJv6KTsyWrFsimXbRkYrlk2xbNvIaMWyKZZtGxmtWDbFsm0joxXLpli2bWS0YtkUy7aNjFYsm2LZtpHRimVTLBubGGWfbbdfuU/sss+2+9sTV+DUA7P3t9tRHW/Wf2qnnpm9vz3u6quj7LOtLys2tWJf72r/8GZ9H7tm3VUfe906qhC7Zt1VH7tmHVUTu27d1Tl23b7tbg/MnmPZ1E/ZkdGKZVMs2zYyWrFsimXbRkYrlk2xbNvIaMWyKZZtGxmtWDbFsm0joxXLpli2bWS0YtkUy7aNjFYsm2LZ2MQo+2y7/cp9Ypd9tt3fnrgCpx6Yvb/djup4s/5TO/XM7P3tcVdfHWWfbX1ZsakV+3pX+4c36/vYNeuu+tjr1lGF2DXrrvrYNeuomth1667Osev2bXd7YPYcy6Z+yo6MViybYtm2kdGKZVMs2zYyWrFsimXbRkYrlk2xbNvIaMWyKZZtGxmtWDbFsm0joxXLpli2bWS0YtkUy8YmRtln2+1X7hO77LPt/vbEFTj1wOz97XZUx5v1n9qpZ2bvb4+7+uoo+2zry4pNrdjXu9o/vFnfx65Zd9XHXreOKsSuWXfVx65ZR9XErlt3dY5dt2+72wOz51g29VN2ZLRi2RTLto2MViybYtm2kdGKZVMs2zYyWrFsimXbRkYrlk2xbNvIaMWyKZZtGxmtWDbFsm0joxXLplg2NjHKPttuv3Kf2GWfbfe3J67AqQdm72+3ozrerP/UTj0ze3973NVXR9lnW19WbGrFvt7V/uHN+j52zbqrPva6dVQhds26qz52zTqqJnbduqtz7Lp9290emD3HsqmfsiOjFcumWLZtZLRi2RTLto2MViybYtm2kdGKZVMs2zYyWrFsimXbRkYrlk2xbNvIaMWyKZZtGxmtWDbFsrGJUfbZdvuV+8Qu+2y7vz1xBU49MHt/ux3V8Wb9p3bqmdn72+OuvjrKPtv6smJTK/b1rvYPb9b3sWvWXfWx162jCrFr1l31sWvWUTWx69ZdnWPX7dvu9sDsOZZN/ZQdGa1YNsWybSOjFcumWLZtZLRi2RTLto2MViybYtm2kdGKZVMs2zYyWrFsimXbRkYrlk2xbNvIaMWyKZaNTYyyz7bbr9wndtln2/3tiStw6oHZ+9vtqI436z+1U8/M3t8ed/XVUfbZ1pcVm1qxr3e1f3izvo9ds+6qj71uHVWIXbPuqo9ds46qiV237uocu27fdrcHZs+xbOqn7MhoxbIplm0bGa1YNsWybSOjFcumWLZtZLRi2RTLto2MViybYtm2kdGKZVMs2zYyWrFsimXbRkYrlk2xbGxilH223X7lPrHLPtvub09cgVMPzN7fbkd1vFn/qZ16Zvb+9rirr46yz7a+rNjUin29q/3Dm/V97Jp1V33sdeuoQuyadVd97Jp1VE3sunVX59h1+7a7PTB7jmVTP2VHRiuWTbFs28hoxbIplm0bGa1YNsWybSOjFcumWLZtZLRi2RTLto2MViybYtm2kdGKZVMs2zYyWrFsimVjE6Pss+32K/eJXfbZdn974gqcemD2/nY7quPN+k/t1DOz97fHXX11lH229WXFplbs613tH96s72PXrLvqY69bRxVi16y76mPXrKNqYtetuzrHrtu33e2B2XMsm/opOzJasWyKZdtGRiuWTbFs28hoxbIplm0bGa1YNsWybSOjFcumWLZtZLRi2RTLto2MViybYtm2kdGKZVMsG7sKt18ys2wXy6ZW7WV4vD3N/nMsm2I/tyOj7PtYNvXLdv/Y/rL947u2//g28jrK9rFsir1u/Y8KsWyKnbOX4fZ39hzLpth1+7YEt39/F8umWDZ12Y6MViybYtm2kdGKZVMs2zYyWrFsimXbRkYrlk2xbNvIaMWyKZZtGxmtWDbFsm0joxXLpli2bWS0YtkUy8YmRtln2+2XzBO77LPt/gv5K3Dqgdn72+2otnfFrtipZ2bvb4+7+uoo+2zry4pNrdjXu9o/Vt6e/sYDs/9O66762OvWUYXYNeuu+tg166ia2HXrrs6x6/Ztd3tg9hzLpn7KjoxWLJti2baR0YplUyzbNjJasWyKZdtGRiuWTbFs28hoxbIplm0bGa1YNsWybSOjFcumWLZtZLRi2RTLxiZG2Wfb7VfuE7vss+3+9sQVOPXA7P3tdlTHm/Wf2qlnZu9vj7v66ij7bOvLik2t2Ne72j+8Wd/Hrll31cdet44qxK5Zd9XHrllH1cSuW3d1jl23b7vbA7PnWDb1U3ZktGLZFMu2jYxWLJti2baR0YplUyzbNjJasWyKZdtGRiuWTbFs28hoxbIplm0bGa1YNsWybSOjFcumWDY2Mco+226/cp/YZZ9t97cnrsCpB2bvb7ejOt6s/9ROPTN7f3vc1VdH2WdbX1ZsasW+3tX+4c36PnbNuqs+9rp1VCF2zbqrPnbNOqomdt26q3Psun3b3R6YPceyqZ+yI6MVy6ZYtm1ktGLZFMu2jYxWLJti2baR0YplUyzbNjJasWyKZdtGRiuWTbFs28hoxbIplm0bGa1YNsWysYlR9tl2+5X7xC77bLu/PXEFTj0we3+7HdXxZv2nduqZ2fvb466+Oso+2/qyYlMr9vWu9g9v1vexa9Zd9bHXraMKsWvWXfWxa9ZRNbHr1l2dY9ft2+72wOw5lk39lB0ZrVg2xbJtI6MVy6ZYtm1ktGLZFMu2jYxWLJti2baR0YplUyzbNjJasWyKZdtGRiuWTbFs28hoxbIplo1NjLLPttuv3Cd22Wfb/e2JK3Dqgdn72+2ojjfrP7VTz8ze3x539dVR9tnWlxWbWrH/v6s//v4fiP+fU3FxzHsAAAAASUVORK5CYII=';


Need to explore better designs for this one.

typing

nreese · 2020-08-17T13:38:16Z

This is really neat.

Would it be possible to keep the dynamic icon from vector layer when using vector tiles from elasticsearch? Notice how the icon in the legend changes when switching to vector tiles.

x-pack/plugins/maps/public/classes/sources/es_search_source/es_search_source.js

x-pack/plugins/maps/public/classes/sources/es_search_source/scaling_form.tsx

x-pack/plugins/maps/public/classes/sources/es_search_source/update_source_editor.js

x-pack/plugins/maps/public/classes/sources/es_search_source/es_search_source.d.ts

x-pack/plugins/maps/server/mvt/mvt_routes.ts

feedback

elasticmachine · 2020-08-18T13:19:56Z

Pinging @elastic/kibana-gis (Team:Geo)

* refactor: 💡 move timeRange, filters and query to base embeddabl * refactor: 💡 use new base embeddable input in explore data * feat: 🎸 import types as types

…#74654)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* Server side changes - removed console_legacy plugin! - added new es_config endpoint that returns server side es config at the moment this is just the first value in hosts - Slight refactor to how routes are registered to bring them more in line with other ES UI plugins * Client side update - Updated the client to not get es host from injected metadata. Instead use the new endpoint created server side that returns this value - Added a small README.md regarding the hooks lib and need to refactor use of jQuery in console - Write code to init the es host value on the client once at start up in a non-blocking way. If this fails we just use the default value of http://localhost:9200 as this powers non-essential console functionality (i.e., copy as cURL). * fix type issue and jest tests * fix another type issue * simplify proxy assignment in proxy handler mock Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…tic#75536) * move csp usage collector from legacy kibana plugin to kibana_usage_collection * make scripts/telemetry_check happy. * remove assertion on legacy kibana plugin * remove test on legacy kibana plugin * update README

…c#75163) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

…circular deps graphs (elastic#75579)

add unit tests functional boiler plate fine tune remove cruft

elasticmachine · 2020-08-21T19:27:01Z

Pinging @elastic/ingest-management (Team:Ingest Management)

thomasneirynck · 2020-08-21T19:30:11Z

sorry all for the spam. I messed up my rebase. Closing this one and will re-open.

kibanamachine · 2020-08-21T19:35:48Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: b7504b3
Pipeline Steps (look for red circles / failed steps)
Interpreting CI Failures

Failed CI Steps

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
console	163	+4	159
esUiShared	227	+1	226
kibanaUtils	189	-2	191
visTypeMarkdown	258	+2	256
visTypeTimeseries	468	-1	469
total		+4

async chunks size

id	value	diff	baseline
console	1.1MB	+2.9KB	1.1MB
visTypeMarkdown	40.7KB	-1.0B	40.7KB
visTypeTimeseries	1.8MB	-386.0B	1.8MB
visualize	694.0KB	+123.0B	693.9KB
total		+2.6KB

page load bundle size

id	value	diff	baseline
console	30.4KB	-175.0B	30.6KB
embeddable	430.5KB	+632.0B	429.9KB
esUiShared	989.2KB	+1.1KB	988.1KB
expressions	346.6KB	-2.3KB	348.9KB
kibanaUtils	470.4KB	-2.5KB	472.8KB
savedObjects	237.4KB	+43.0B	237.3KB
visTypeMarkdown	556.3KB	+3.1KB	553.2KB
visualizations	407.4KB	-456.0B	407.8KB
total		-554.0B

History

💔 Build #69532 failed 4df20b4
💔 Build #69505 failed 6e2455f
💔 Build #69134 failed 919024a
💔 Build #69120 failed c73faef

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

thomasneirynck mentioned this pull request Aug 14, 2020

[Maps] [Meta] Add .mvt vector tile support #58519

Closed

16 tasks

mvt init

bd4f39b

thomasneirynck force-pushed the maps/es_mvt branch from 9b615c6 to bd4f39b Compare August 14, 2020 06:43

thomasneirynck commented Aug 14, 2020

View reviewed changes

thomasneirynck requested a review from a team August 14, 2020 07:17

thomasneirynck added release_note:enhancement v7.10.0 v8.0.0 [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation labels Aug 14, 2020

thomasneirynck added 4 commits August 14, 2020 03:19

cleanup

4da768c

typing

aae411f

typing

update snapshot

a3fbf97

Enable mvt only for geo_shape

828ce8a

kmartastic mentioned this pull request Aug 14, 2020

[Maps] Display millions of points on a map #75104

Closed

nreese reviewed Aug 17, 2020

View reviewed changes

thomasneirynck added 5 commits August 17, 2020 11:06

fix ts

02e7688

Fix snapshot

058c116

update functional tests

43efc7c

Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt

4c238c7

feedback

76bdfcc

feedback

thomasneirynck force-pushed the maps/es_mvt branch from bcce360 to 76bdfcc Compare August 17, 2020 22:16

thomasneirynck requested review from jsanz and kindsun August 17, 2020 22:17

thomasneirynck marked this pull request as ready for review August 18, 2020 13:19

thomasneirynck added 2 commits August 18, 2020 09:21

fix snapshot

2371e7b

disable client-side metadata retrieval

9769159

ppisljar and others added 9 commits August 21, 2020 15:26

adding markdown vis renderer (elastic#75532)

6e162ff

Embeddable input (elastic#73033)

42c2f8e

* refactor: 💡 move timeRange, filters and query to base embeddabl * refactor: 💡 use new base embeddable input in explore data * feat: 🎸 import types as types

[Lens] Use index pattern service instead saved object client (elastic…

aeb6573

…#74654)

[Uptime] Add delay in telemetry test (elastic#75162)

e91528e

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Data Telemetry] Add index pattern to identify "meow" attacks (elasti…

80ce878

…c#75163) Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

[Security Solution] modify circular deps checker to output images of …

2f1a03e

…circular deps graphs (elastic#75579)

update snap

b7504b3

add unit tests functional boiler plate fine tune remove cruft

thomasneirynck requested review from a team as code owners August 21, 2020 19:26

thomasneirynck requested a review from a team August 21, 2020 19:26

thomasneirynck requested review from a team as code owners August 21, 2020 19:26

thomasneirynck requested a review from a team August 21, 2020 19:26

thomasneirynck requested review from a team as code owners August 21, 2020 19:26

botelastic bot added Feature:Embedding Embedding content via iFrame Feature:ExpressionLanguage Interpreter expression language (aka canvas pipeline) Team:Fleet Team label for Observability Data Collection Fleet team labels Aug 21, 2020

thomasneirynck closed this Aug 21, 2020

thomasneirynck mentioned this pull request Aug 21, 2020

[Maps] Add mvt support for ES doc sources #75698

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Maps] Add mvt format for ES-doc sources #74319

[Maps] Add mvt format for ES-doc sources #74319

thomasneirynck commented Aug 4, 2020 •

edited

Loading

thomasneirynck Aug 14, 2020

thomasneirynck Aug 14, 2020

thomasneirynck Aug 14, 2020

nreese commented Aug 17, 2020

elasticmachine commented Aug 18, 2020

elasticmachine commented Aug 21, 2020

thomasneirynck commented Aug 21, 2020

kibanamachine commented Aug 21, 2020 •

edited

Loading

[Maps] Add mvt format for ES-doc sources #74319

[Maps] Add mvt format for ES-doc sources #74319

Conversation

thomasneirynck commented Aug 4, 2020 • edited Loading

Summary

Introduce a new scaling-type "mvt tiles"

Extend ESSearchSource to implement

Add new getTile route-backend

Introduce the concept of a too many features bounds

Future work

Todo

thomasneirynck Aug 14, 2020

Choose a reason for hiding this comment

thomasneirynck Aug 14, 2020

Choose a reason for hiding this comment

thomasneirynck Aug 14, 2020

Choose a reason for hiding this comment

nreese commented Aug 17, 2020

elasticmachine commented Aug 18, 2020

elasticmachine commented Aug 21, 2020

thomasneirynck commented Aug 21, 2020

kibanamachine commented Aug 21, 2020 • edited Loading

💔 Build Failed

Failed CI Steps

Build metrics

@kbn/optimizer bundle module count

async chunks size

page load bundle size

History

thomasneirynck commented Aug 4, 2020 •

edited

Loading

Add new `getTile` route-backend

Introduce the concept of a `too many features` bounds

kibanamachine commented Aug 21, 2020 •

edited

Loading