Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Maps] Auto generate legends and styles from mvt data #94811

Merged
merged 92 commits into from
Jul 8, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
d0c462b
init boilerplate
thomasneirynck Mar 16, 2021
e68ecf0
more boilerplate
thomasneirynck Mar 16, 2021
54b8429
boilerplate for doc_count
thomasneirynck Mar 16, 2021
c9e0d4f
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Apr 2, 2021
14c40b0
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Apr 5, 2021
a4d7370
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Apr 6, 2021
18d6ecd
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Apr 7, 2021
5f930ff
cleanup
thomasneirynck Apr 7, 2021
ee10a58
more boiler
thomasneirynck Apr 8, 2021
410276e
edits
thomasneirynck Apr 8, 2021
40ac552
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Apr 13, 2021
fe3bfbf
boiler for style
thomasneirynck Apr 14, 2021
40c0680
boiler for icon
thomasneirynck Apr 14, 2021
405a9df
minor fixes
thomasneirynck Apr 14, 2021
977b9e2
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Apr 21, 2021
bddeb59
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck May 26, 2021
f506db2
improve typing
thomasneirynck May 26, 2021
94f2a15
formatting snafu
thomasneirynck May 26, 2021
ae19c61
improve typing
thomasneirynck May 26, 2021
f87ce95
type fix
thomasneirynck May 26, 2021
35258a0
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck May 27, 2021
135bdb4
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 14, 2021
15cd4fd
type fixes
thomasneirynck Jun 14, 2021
2e5d037
more fixes
thomasneirynck Jun 14, 2021
65543b7
add incomplete results icon
thomasneirynck Jun 15, 2021
50c49e4
add no results icon
thomasneirynck Jun 15, 2021
8578ba2
extract commonalities
thomasneirynck Jun 15, 2021
664ab18
remove typo
thomasneirynck Jun 15, 2021
c429b09
typo
thomasneirynck Jun 15, 2021
4c98eb7
remove unused
thomasneirynck Jun 15, 2021
beaa7cd
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 16, 2021
b78fd7f
feedback
thomasneirynck Jun 16, 2021
dd2a830
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 23, 2021
f0594ff
feedback
thomasneirynck Jun 23, 2021
1bd388c
feedback
thomasneirynck Jun 23, 2021
0a838c0
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 23, 2021
58033f2
type fixes
thomasneirynck Jun 24, 2021
64702d2
fix test
thomasneirynck Jun 24, 2021
f1f3060
update snapshot
thomasneirynck Jun 24, 2021
b5cf12a
fix tests
thomasneirynck Jun 24, 2021
9a64f91
update functional test
thomasneirynck Jun 24, 2021
0609f57
rename
thomasneirynck Jun 24, 2021
3ea92fc
fix tests
thomasneirynck Jun 24, 2021
f660e0f
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 24, 2021
150c8dd
update test
thomasneirynck Jun 24, 2021
a4d10f7
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 24, 2021
80fa1a0
fix test
thomasneirynck Jun 24, 2021
6d1c76f
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 28, 2021
2ffb088
fix test
thomasneirynck Jun 28, 2021
6cc7b82
fix test
thomasneirynck Jun 28, 2021
90f00cf
feedback
thomasneirynck Jun 28, 2021
09d4fe7
fix typo
thomasneirynck Jun 28, 2021
99d3edd
add comment
thomasneirynck Jun 28, 2021
c7f463d
add unit test
thomasneirynck Jun 28, 2021
5999da5
add test
thomasneirynck Jun 28, 2021
2b9d545
fix impl/test
thomasneirynck Jun 28, 2021
72d8f1f
fix trimmed icon
thomasneirynck Jun 28, 2021
e3d4095
hide data-mapper ux for counts
thomasneirynck Jun 28, 2021
e695621
fieldMeta fix
thomasneirynck Jun 28, 2021
15730d6
do not allow for toggling get-from-data switch
thomasneirynck Jun 28, 2021
32ab71f
fix tests
thomasneirynck Jun 28, 2021
0e14370
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 28, 2021
4ebc17e
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 29, 2021
a258dfe
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 29, 2021
48ccd8b
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 29, 2021
10eb35f
fix edge cases
thomasneirynck Jun 29, 2021
bad6921
remove obsolete test
thomasneirynck Jun 29, 2021
6aa0f51
extract style-meta code to common
thomasneirynck Jun 30, 2021
b7dc391
reuse in tile-gen
thomasneirynck Jun 30, 2021
3246976
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jun 30, 2021
a047002
improve typing
thomasneirynck Jun 30, 2021
fb8b836
rewrite for clarity
thomasneirynck Jun 30, 2021
90377cc
fix delete
thomasneirynck Jul 1, 2021
8d25600
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jul 1, 2021
5872b08
type field-stylemeta
thomasneirynck Jul 1, 2021
1d3a276
add tilemetafeature type for clarity
thomasneirynck Jul 1, 2021
33fde42
tighten types
thomasneirynck Jul 1, 2021
cf77f83
align implementations
thomasneirynck Jul 1, 2021
ba79ed5
get categories from tilemeta
thomasneirynck Jul 1, 2021
4ad1ae7
add category-rollups
thomasneirynck Jul 1, 2021
3828671
move shape-counts
thomasneirynck Jul 1, 2021
f80aa2a
type fix
thomasneirynck Jul 1, 2021
35a5b60
add metadata to features
thomasneirynck Jul 1, 2021
df555e2
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jul 6, 2021
415959a
fix test
thomasneirynck Jul 6, 2021
bbd1403
fix test
thomasneirynck Jul 6, 2021
5f3ba8e
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jul 7, 2021
ff94926
feedback
thomasneirynck Jul 7, 2021
83a728b
type cleanup
thomasneirynck Jul 7, 2021
778775e
update snapshot
thomasneirynck Jul 7, 2021
ef41ff2
Merge branch 'master' of github.com:elastic/kibana into maps/metadata…
thomasneirynck Jul 7, 2021
c1bcdd1
feedback
thomasneirynck Jul 7, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions api_docs/maps.json
Original file line number Diff line number Diff line change
Expand Up @@ -3085,13 +3085,13 @@
},
{
"parentPluginId": "maps",
"id": "def-common.KBN_TOO_MANY_FEATURES_PROPERTY",
"id": "def-common.KBN_METADATA_FEATURE",
"type": "string",
"tags": [],
"label": "KBN_TOO_MANY_FEATURES_PROPERTY",
"label": "KBN_METADATA_FEATURE",
"description": [],
"signature": [
"\"__kbn_too_many_features__\""
"\"__kbn_metadata_feature__\""
],
"source": {
"path": "x-pack/plugins/maps/common/constants.ts",
Expand Down Expand Up @@ -3582,4 +3582,4 @@
}
]
}
}
}
5 changes: 4 additions & 1 deletion x-pack/plugins/maps/common/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,10 @@ export const MVT_GETGRIDTILE_API_PATH = 'mvt/getGridTile';
export const MVT_SOURCE_LAYER_NAME = 'source_layer';
// Identifies vector tile "too many features" feature.
// "too many features" feature is a box showing area that contains too many features for single ES search response
export const KBN_TOO_MANY_FEATURES_PROPERTY = '__kbn_too_many_features__';
export const KBN_METADATA_FEATURE = '__kbn_metadata_feature__';
export const KBN_FEATURE_COUNT = '__kbn_feature_count__';
export const KBN_IS_TILE_COMPLETE = '__kbn_is_tile_complete__';
export const KBN_VECTOR_SHAPE_TYPE_COUNTS = '__kbn_vector_shape_type_counts__';
export const KBN_TOO_MANY_FEATURES_IMAGE_ID = '__kbn_too_many_features_image_id__';
// Identifies centroid feature.
// Centroids are a single point for representing lines, multiLines, polygons, and multiPolygons
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,22 @@
/* eslint-disable @typescript-eslint/consistent-type-definitions */

import { Query } from 'src/plugins/data/public';
import { Feature } from 'geojson';
import {
FieldMeta,
HeatmapStyleDescriptor,
StyleDescriptor,
VectorStyleDescriptor,
} from './style_property_descriptor_types';
import { DataRequestDescriptor } from './data_request_descriptor_types';
import { AbstractSourceDescriptor, TermJoinSourceDescriptor } from './source_descriptor_types';
import { VectorShapeTypeCounts } from '../get_geometry_counts';
import {
KBN_FEATURE_COUNT,
KBN_IS_TILE_COMPLETE,
KBN_METADATA_FEATURE,
KBN_VECTOR_SHAPE_TYPE_COUNTS,
} from '../constants';

export type Attribution = {
label: string;
Expand All @@ -26,13 +35,24 @@ export type JoinDescriptor = {
right: TermJoinSourceDescriptor;
};

export type TileMetaFeature = Feature & {
properties: {
[KBN_METADATA_FEATURE]: true;
[KBN_IS_TILE_COMPLETE]: boolean;
[KBN_FEATURE_COUNT]: number;
[KBN_VECTOR_SHAPE_TYPE_COUNTS]: VectorShapeTypeCounts;
fieldMeta?: FieldMeta;
};
};

export type LayerDescriptor = {
__dataRequests?: DataRequestDescriptor[];
__isInErrorState?: boolean;
__isPreviewLayer?: boolean;
__errorMessage?: string;
__trackedLayerDescriptor?: LayerDescriptor;
__areTilesLoaded?: boolean;
__metaFromTiles?: TileMetaFeature[];
alpha?: number;
attribution?: Attribution;
id: string;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -225,14 +225,16 @@ export type GeometryTypes = {
isPolygonsOnly: boolean;
};

export type FieldMeta = {
[key: string]: {
range?: RangeFieldMeta;
categories?: CategoryFieldMeta;
};
};

export type StyleMetaDescriptor = {
geometryTypes?: GeometryTypes;
fieldMeta: {
[key: string]: {
range?: RangeFieldMeta;
categories?: CategoryFieldMeta;
};
};
fieldMeta: FieldMeta;
};

export type VectorStyleDescriptor = StyleDescriptor & {
Expand Down
4 changes: 2 additions & 2 deletions x-pack/plugins/maps/common/get_centroid_features.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ test('should not create centroid feature for point and multipoint', () => {
expect(centroidFeatures.length).toBe(0);
});

test('should not create centroid for too many features polygon', () => {
test('should not create centroid for the metadata polygon', () => {
const polygonFeature: Feature = {
type: 'Feature',
geometry: {
Expand All @@ -60,7 +60,7 @@ test('should not create centroid for too many features polygon', () => {
],
},
properties: {
__kbn_too_many_features__: true,
__kbn_metadata_feature__: true,
prop0: 'value0',
prop1: 0.0,
},
Expand Down
8 changes: 2 additions & 6 deletions x-pack/plugins/maps/common/get_centroid_features.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,15 @@ import turfArea from '@turf/area';
import turfCenterOfMass from '@turf/center-of-mass';
import turfLength from '@turf/length';
import { lineString, polygon } from '@turf/helpers';
import {
GEO_JSON_TYPE,
KBN_IS_CENTROID_FEATURE,
KBN_TOO_MANY_FEATURES_PROPERTY,
} from './constants';
import { GEO_JSON_TYPE, KBN_IS_CENTROID_FEATURE, KBN_METADATA_FEATURE } from './constants';

export function getCentroidFeatures(featureCollection: FeatureCollection): Feature[] {
const centroids = [];
for (let i = 0; i < featureCollection.features.length; i++) {
const feature = featureCollection.features[i];

// do not add centroid for kibana added features
if (feature.properties?.[KBN_TOO_MANY_FEATURES_PROPERTY]) {
thomasneirynck marked this conversation as resolved.
Show resolved Hide resolved
if (feature.properties?.[KBN_METADATA_FEATURE]) {
continue;
}

Expand Down
45 changes: 45 additions & 0 deletions x-pack/plugins/maps/common/get_geometry_counts.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { Feature } from 'geojson';
import { GEO_JSON_TYPE, VECTOR_SHAPE_TYPE } from './constants';

export interface VectorShapeTypeCounts {
[VECTOR_SHAPE_TYPE.POINT]: number;
[VECTOR_SHAPE_TYPE.LINE]: number;
[VECTOR_SHAPE_TYPE.POLYGON]: number;
}

export function countVectorShapeTypes(features: Feature[]): VectorShapeTypeCounts {
const vectorShapeTypeCounts: VectorShapeTypeCounts = {
[VECTOR_SHAPE_TYPE.POINT]: 0,
[VECTOR_SHAPE_TYPE.LINE]: 0,
[VECTOR_SHAPE_TYPE.POLYGON]: 0,
};

for (let i = 0; i < features.length; i++) {
const feature: Feature = features[i];
if (
feature.geometry.type === GEO_JSON_TYPE.POINT ||
feature.geometry.type === GEO_JSON_TYPE.MULTI_POINT
) {
vectorShapeTypeCounts[VECTOR_SHAPE_TYPE.POINT] += 1;
} else if (
feature.geometry.type === GEO_JSON_TYPE.LINE_STRING ||
feature.geometry.type === GEO_JSON_TYPE.MULTI_LINE_STRING
) {
vectorShapeTypeCounts[VECTOR_SHAPE_TYPE.LINE] += 1;
} else if (
feature.geometry.type === GEO_JSON_TYPE.POLYGON ||
feature.geometry.type === GEO_JSON_TYPE.MULTI_POLYGON
) {
vectorShapeTypeCounts[VECTOR_SHAPE_TYPE.POLYGON] += 1;
}
}

return vectorShapeTypeCounts;
}
46 changes: 46 additions & 0 deletions x-pack/plugins/maps/common/pluck_category_field_meta.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { Feature } from 'geojson';
import { CategoryFieldMeta } from './descriptor_types';

export function pluckCategoryFieldMeta(
features: Feature[],
name: string,
size: number
): CategoryFieldMeta | null {
const counts = new Map();
for (let i = 0; i < features.length; i++) {
const feature = features[i];
const term = feature.properties ? feature.properties[name] : undefined;
// properties object may be sparse, so need to check if the field is effectively present
if (typeof term !== undefined) {
if (counts.has(term)) {
counts.set(term, counts.get(term) + 1);
} else {
counts.set(term, 1);
}
}
}

return trimCategories(counts, size);
}

export function trimCategories(counts: Map<string, number>, size: number): CategoryFieldMeta {
const ordered = [];
for (const [key, value] of counts) {
ordered.push({ key, count: value });
}

ordered.sort((a, b) => {
return b.count - a.count;
});
const truncated = ordered.slice(0, size);
return {
categories: truncated,
} as CategoryFieldMeta;
}
34 changes: 34 additions & 0 deletions x-pack/plugins/maps/common/pluck_range_field_meta.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { Feature } from 'geojson';
import { RangeFieldMeta } from './descriptor_types';

export function pluckRangeFieldMeta(
features: Feature[],
name: string,
parseValue: (rawValue: unknown) => number
): RangeFieldMeta | null {
let min = Infinity;
let max = -Infinity;
for (let i = 0; i < features.length; i++) {
const feature = features[i];
const newValue = feature.properties ? parseValue(feature.properties[name]) : NaN;
if (!isNaN(newValue)) {
min = Math.min(min, newValue);
max = Math.max(max, newValue);
}
}

return min === Infinity || max === -Infinity
? null
: ({
min,
max,
delta: max - min,
} as RangeFieldMeta);
}
13 changes: 5 additions & 8 deletions x-pack/plugins/maps/public/actions/data_request_actions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@ import { FeatureCollection } from 'geojson';
import { MapStoreState } from '../reducers/store';
import {
KBN_IS_CENTROID_FEATURE,
LAYER_STYLE_TYPE,
LAYER_TYPE,
SOURCE_DATA_REQUEST_ID,
} from '../../common/constants';
Expand Down Expand Up @@ -49,7 +48,6 @@ import { IVectorLayer } from '../classes/layers/vector_layer';
import { DataMeta, MapExtent, MapFilters } from '../../common/descriptor_types';
import { DataRequestAbortError } from '../classes/util/data_request';
import { scaleBounds, turfBboxToBounds } from '../../common/elasticsearch_util';
import { IVectorStyle } from '../classes/styles/vector/vector_style';

const FIT_TO_BOUNDS_SCALE_FACTOR = 0.1;

Expand Down Expand Up @@ -95,14 +93,12 @@ export function updateStyleMeta(layerId: string | null) {
if (!layer) {
return;
}
const sourceDataRequest = layer.getSourceDataRequest();
const style = layer.getCurrentStyle();
if (!style || !sourceDataRequest || style.getType() !== LAYER_STYLE_TYPE.VECTOR) {

const styleMeta = await layer.getStyleMetaDescriptorFromLocalFeatures();
if (!styleMeta) {
return;
}
const styleMeta = await (style as IVectorStyle).pluckStyleMetaFromSourceDataRequest(
sourceDataRequest
);

dispatch({
type: SET_LAYER_STYLE_META,
layerId,
Expand Down Expand Up @@ -249,6 +245,7 @@ function endDataLoad(
dispatch(unregisterCancelCallback(requestToken));
const dataRequest = getDataRequestDescriptor(getState(), layerId, dataId);
if (dataRequest && dataRequest.dataRequestToken !== requestToken) {
// todo - investigate - this may arise with failing style meta request and should not throw in that case
throw new DataRequestAbortError();
}

Expand Down
21 changes: 21 additions & 0 deletions x-pack/plugins/maps/public/actions/layer_actions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ import {
JoinDescriptor,
LayerDescriptor,
StyleDescriptor,
TileMetaFeature,
} from '../../common/descriptor_types';
import { ILayer } from '../classes/layers/layer';
import { IVectorLayer } from '../classes/layers/vector_layer';
Expand Down Expand Up @@ -591,3 +592,23 @@ export function setAreTilesLoaded(layerId: string, areTilesLoaded: boolean) {
newValue: areTilesLoaded,
};
}

export function updateMetaFromTiles(layerId: string, mbMetaFeatures: TileMetaFeature[]) {
return async (
dispatch: ThunkDispatch<MapStoreState, void, AnyAction>,
getState: () => MapStoreState
) => {
const layer = getLayerById(layerId, getState());
if (!layer) {
return;
}

dispatch({
type: UPDATE_LAYER_PROP,
id: layerId,
propName: '__metaFromTiles',
newValue: mbMetaFeatures,
});
await dispatch(updateStyleMeta(layerId));
};
}
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ export class CountAggField implements IESAggField {
}

supportsAutoDomain(): boolean {
return this._canReadFromGeoJson ? true : this.supportsFieldMeta();
return true;
}

canReadFromGeoJson(): boolean {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -184,9 +184,9 @@ export class BlendedVectorLayer extends VectorLayer implements IVectorLayer {

private readonly _isClustered: boolean;
private readonly _clusterSource: ESGeoGridSource;
private readonly _clusterStyle: IVectorStyle;
private readonly _clusterStyle: VectorStyle;
private readonly _documentSource: ESSearchSource;
private readonly _documentStyle: IVectorStyle;
private readonly _documentStyle: VectorStyle;

constructor(options: BlendedVectorLayerArguments) {
super({
Expand All @@ -195,7 +195,7 @@ export class BlendedVectorLayer extends VectorLayer implements IVectorLayer {
});

this._documentSource = this._source as ESSearchSource; // VectorLayer constructor sets _source as document source
this._documentStyle = this._style as IVectorStyle; // VectorLayer constructor sets _style as document source
this._documentStyle = this._style; // VectorLayer constructor sets _style as document source

this._clusterSource = getClusterSource(this._documentSource, this._documentStyle);
const clusterStyleDescriptor = getClusterStyleDescriptor(
Expand Down Expand Up @@ -279,7 +279,7 @@ export class BlendedVectorLayer extends VectorLayer implements IVectorLayer {
return this._documentSource;
}

getCurrentStyle(): IVectorStyle {
getCurrentStyle(): VectorStyle {
return this._isClustered ? this._clusterStyle : this._documentStyle;
}

Expand Down
Loading