[index patterns] [skip-ci] Index pattern “pattern list” support #89330

stephmilovic · 2021-01-26T17:31:00Z

WIP

This is a Work In Progress PR. I'm sorry it is going to be quite large as this is a breaking change to Kibana index patterns.

Resolves the first part of #87851

Restructure index-pattern saved object to accommodate patternList
- title remains as a label for the index pattern
- title will be migrated to patternList from a string to a string array, separated by commas.
  - ex: { title: "filebeat-*,auditbeat-*" } becomes { title: "filebeat-*,auditbeat-*", patternList: ["filebeat-*","auditbeat-*"] }
- going forward, title will be a descriptive field
  - ex: { title: "Security Solution Indices", patternList: ["filebeat-*","auditbeat-*"] }
at runtime, detects index patterns in the patternList that do not match data and returns patternListActive

Plugin Changes (so far, WIP y'all)

Discover: updates sidebar index pattern selector UI
TSVB: update fetchFields function that checked title to check patternListActive instead

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

stephmilovic · 2021-01-26T20:17:23Z

src/plugins/data/server/index_patterns/routes.ts

+          (rif) => rif !== false
+        ) as FieldDescriptor[][];
+        const fields = !formatFields
+          ? await combineFields(responsesIndexFields)


@XavierM this is the documenter boolean. should i include this and the formatIndexFields?

stephmilovic · 2021-01-26T20:40:05Z

src/plugins/data/common/index_patterns/index_patterns/index_patterns.ts

-        intervalName,
-        fields,
-        sourceFilters,
+        allowNoIndex,


Apologies for the extra line changes. It is so much easier to see where updates have been made to objects/types when their properties are alphabetized, especially with bigger objects. I did this in a few places throughout this PR

stephmilovic · 2021-01-28T20:09:35Z

src/plugins/vis_type_timelion/server/series_functions/es/index.js

@@ -96,20 +96,15 @@ export default new Datasource('es', {
      kibana: true,
      fit: 'nearest',
    });
-
-    const findResp = await tlConfig.savedObjectsClient.find({


Since the patternListActive is generated at runtime, it is not stored on the index-pattern saved object. I changed this to fetch the index patterns from the indexPatternsService.getPatternCache(), which will give you a list of index patterns including the runtime patternListActive property

alexwizp · 2021-01-29T09:50:15Z

src/plugins/vis_type_timelion/server/routes/run.ts

@@ -79,6 +79,16 @@ export function runRoute(
      try {
        const uiSettings = await context.core.uiSettings.client.getAll();

+        const getIndexPatternsService = async () => {
+          // @ts-ignore need help resolving this one


to fix that, please update type in line 36 CoreSetup -> CoreSetup<TimelionPluginStartDeps>

alexwizp · 2021-01-29T10:01:00Z

src/plugins/vis_type_timelion/server/routes/run.ts

@@ -88,6 +98,7 @@ export function runRoute(
          allowedGraphiteUrls: configManager.getGraphiteUrls(),
          esShardTimeout: configManager.getEsShardTimeout(),
          savedObjectsClient: context.core.savedObjects.client,
+          getIndexPatternsService: async () => indexPatternsService,


nit: I see that code was copied from TSVB. But it that plugin we had a reason to return indexPatternsService in a Promise. For your case it doesn't have a lot of sense.

I suggest you to remove line 91: const indexPatternsService = await getIndexPatternsService();

And here write:

{ savedObjectsClient: context.core.savedObjects.client, getIndexPatternsService, }

alexwizp · 2021-01-29T10:10:48Z

src/plugins/vis_type_timeseries/server/lib/get_fields.ts

@@ -62,7 +62,7 @@ export async function getFields(
  if (!indexPatternString) {
    const defaultIndexPattern = await indexPatternsService.getDefault();

-    indexPatternString = defaultIndexPattern?.title ?? '';
+    indexPatternString = defaultIndexPattern?.patternListActive.join(',') ?? '';


I'm not a big fan to do patternListActive.join(',') on each places where we need to get string representation of index pattern. From my perspective it should be encapsulated as a method in IndexPattern service: e.g getIndexPatternString() or just a toString()

alexwizp · 2021-01-29T10:15:22Z

src/plugins/index_pattern_management/public/components/utils.ts

  );
-}
+  return patterns.sort((a, b) => {


nit: what about sortBy(patterns, 'sort') from lodash?

alexwizp · 2021-01-29T10:52:12Z

...nents/create_index_pattern_wizard/components/step_index_pattern/components/header/header.tsx

+      if (containsIllegalCharacters(searchValue, indexPatterns.ILLEGAL_CHARACTERS)) {
+        return false;
+      }
+      onQueryChanged([...selectedPatterns, searchValue]);


not sure that this code works correctly

if I input index-a,index-b,index-c. It should be as least splitted by comma and added as a three separate indexes.

Screen.Recording.2021-01-29.at.1.43.02.PM.mov

to discuss: some time ago we tried to solve the same problem on TSVB plugin. From my point of view here instead of EuiComboBox we should use EuiSuggest component. See #82696

alexwizp · 2021-01-29T12:07:28Z

src/plugins/vis_type_timelion/server/series_functions/es/index.js

-    const indexPatternSavedObject = findResp.saved_objects.find((savedObject) => {
-      return savedObject.attributes.title === config.index;
+    const indexPatternsService = await tlConfig.getIndexPatternsService();
+    const searchIndexPatterns = await indexPatternsService.find(config.index);


it's not working as expected. To reproduce that, let's create the following index:

and then let's try to find that index using indexPatternsService.find method. I used the following snippet:

console.log('f1', await indexPatterns.find('f1')); console.log('index-a,index-b,index-c,index-d', await indexPatterns.find('index-a,index-b,index-c,index-d'));

Result of that execution looks like:

it means that in that method we are looking by title not by patternList

alexwizp · 2021-01-29T12:15:37Z

x-pack/plugins/fleet/server/services/epm/kibana/index_pattern/install.ts

@@ -11,6 +11,7 @@ import { dataTypes, installationStatuses } from '../../../../../common/constants
 import { ArchivePackage, Installation, InstallSource, ValueOf } from '../../../../../common/types';
 import { RegistryPackage, DataType } from '../../../../types';
 import { getInstallation, getPackageFromSource, getPackageSavedObjects } from '../../packages/get';
+import { IndexPatternAttributes } from '../../../../../../../../src/plugins/data/common/index_patterns';


nit: could you please fix import. '../../../../../../../../src/plugins/data/server'

alexwizp · 2021-01-29T12:45:34Z

src/plugins/vis_type_timelion/server/series_functions/es/index.js

-      return savedObject.attributes.title === config.index;
+    const indexPatternsService = await tlConfig.getIndexPatternsService();
+    const searchIndexPatterns = await indexPatternsService.find(config.index);
+    const indexPatternSpec = searchIndexPatterns.find((indexPattern) => {


@timroes not sure that I fully understand how it should be. Before we could not create two indexes with the same title. Now, title it's just a name and for mapping ES indexes we use patternList property. It means that I can do something like:

Let's complicate the case a bit and make changes to the f2 index. You can set a custom field or add a new scripted field. It does not matter. For my example I added a scripted field

now let's go back to that code. So: searchIndexPatterns - it's an array which keeps 2 indexes (f1, f2). 'find' by array returns the first matched result. It's f1.

But what if I need to work with f2?

I really like that code and I think that it's a good enchantment for our users. But have no idea how TSVB, Timelion and Vega should work with that.

ryankeairns · 2021-02-02T14:53:36Z

This bit of UI is repeated in other places (e.g. Lens), so we ought to get their take once you're ready for wider feedback. Thanks for tackling this change.

stephmilovic · 2021-02-19T15:11:08Z

Closing until a few prerequisite updates are taken care of: #90076

stephmilovic added 30 commits January 11, 2021 08:48

logging

533f617

Merge branch 'master' into index-alias-collection

21e0138

getting fields like a siem

dd8814d

a little temp ui

9acf492

little ui

3b05751

cleaning

9df909c

more

c487780

broken but need to test pr

0328955

Merge branch 'master' into index-alias-collection

485789d

fix bug

a2de575

added those properties

ab13362

add validation for pattern lists

b844bf6

wip oh man

7b6b12d

Merge branch 'master' into index-alias-collection

415c9e5

back to working

17001f4

fix merge

b06bd6a

working on form

3a5f2d3

to hooks

bd54c68

hookin left and right

83bcade

my god it works

0265406

grammar fix

b345b63

revert to class on wizard

8b0d9c5

shorten import

3f9be16

fix merge conflicts

efb826a

Merge branch 'master' into index-alias-collection

1573ccc

tests

3c73237

type fixing and discover sidebar

b0c80c7

get rid of silly

c3c4371

hooked up search source

becae4c

discover ui

f465951

stephmilovic commented Jan 26, 2021

View reviewed changes

stephmilovic added 10 commits January 26, 2021 15:13

type and test fixing

f59eb5b

discover test fixing

564e82f

Fix stub imports

f5b8f00

more test fiixng

6ecfac5

comment out ts err

5f863a2

Fix mistake saving active list

f51ee62

small tsvb change

6ee27a6

avoid some unnecessary promises

c602f39

better notes

af21bde

time lion

82c93f6

stephmilovic commented Jan 28, 2021

View reviewed changes

stephmilovic added 3 commits January 28, 2021 13:32

make scripted fields work

0cd9cf0

change to find

da7a4b6

remove console log

d95fbb3

alexwizp reviewed Jan 29, 2021

View reviewed changes

stephmilovic added 2 commits January 29, 2021 13:25

fix doc function

a288e45

better name

950a8c0

stephmilovic mentioned this pull request Feb 2, 2021

[Meta] Index Patterns Everywhere #90076

Closed

1 task

stephmilovic closed this Feb 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[index patterns] [skip-ci] Index pattern “pattern list” support #89330

[index patterns] [skip-ci] Index pattern “pattern list” support #89330

stephmilovic commented Jan 26, 2021 •

edited by timroes

Loading

stephmilovic Jan 26, 2021 •

edited

Loading

stephmilovic Jan 26, 2021

stephmilovic Jan 28, 2021

alexwizp Jan 29, 2021

alexwizp Jan 29, 2021

alexwizp Jan 29, 2021

alexwizp Jan 29, 2021

alexwizp Jan 29, 2021

alexwizp Jan 29, 2021

alexwizp Jan 29, 2021

alexwizp Jan 29, 2021

ryankeairns commented Feb 2, 2021

stephmilovic commented Feb 19, 2021

[index patterns] [skip-ci] Index pattern “pattern list” support #89330

[index patterns] [skip-ci] Index pattern “pattern list” support #89330

Conversation

stephmilovic commented Jan 26, 2021 • edited by timroes Loading

WIP

Plugin Changes (so far, WIP y'all)

Checklist

For maintainers

stephmilovic Jan 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryankeairns commented Feb 2, 2021

stephmilovic commented Feb 19, 2021

stephmilovic commented Jan 26, 2021 •

edited by timroes

Loading

stephmilovic Jan 26, 2021 •

edited

Loading