[Index template] Fix editor should support mappings types #55804

sebelga · 2020-01-24T08:27:57Z

This PR adds support for mappings type in the Template UI (create/cone/edit).

Through the UI it is not possible to create a new template with a type, but it is possible that a template being edited has a custom type attached.

Fix #55645

Note for the reviewer

Activate the "Hide whitespace changes"

How to test

Create a template with a custom type in the dev console

PUT _template/my_index_template?include_type_name
{
  "index_patterns": [
    "my_index"
  ],
  "mappings": {
    "my_custom_type": {
      "properties": {
        "name": {
          "type": "keyword"
        }
      }
    }
  }
}

Then navigate to the Template UI and edit that template.
Got to the "Mappings" and add a field
Got to the review tab and make sure that the request still contains the custom type ("my_custom_type")

Save the template. No error should appear and you should be brought back to the template list
The mappings of the template should have the custom type

Indices created with the template

Create an index that matches the template

POST my_index/_doc
{
  "name": "a" 
}

Go to the Index Management --> Indices tab
Click on the "my_index" index
Navigate to the "Mappings" tab
The custom type should appear

Note for 7.5 backport

As the fix to support types requires the files from the lib folder of the mappings editor, we will need to manually cherry-pick on the backport PR the commits that don't modify those files in this PR, and then copy over the needed files manually in a separate commit.

Release note

We fixed a bug introduced in 7.4 when we shipped the index template UI. The template editor did not take into account custom type defined for the mappings when fetching and editing an index template. The mapping type ended up being converted to the default _doc losing the ability to query documents on the custom type.

elasticmachine · 2020-01-24T08:30:21Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

cjcenizal

Great work Seb! I found a bug, though. If you use the "Load JSON" modal to load in mappings that contain a type, the UI resets to the default state. This is particularly strange if you're looking at mappings that already contain a type. You could copy the mappings from the Review tab and paste it into the "Load JSON" modal and you'd expect it to show you the mappings you already have.

cjcenizal · 2020-01-24T20:52:23Z

...ns/index_management/public/app/components/mappings_editor/lib/extract_mappings_definition.ts

+  return typedMappings;
+};
+
+export const doesMappingsHasType = (mappings: GenericObject = {}): boolean =>


Nit: I think the name doMappingsHaveType or even isTypedMappings would be more grammatically correct.

For sure 😊

Custom can't start with an underscore so "_source" would never be a valid type

…ditor-should-support-mappings-types

sebelga · 2020-01-28T08:42:10Z

Thanks for the review @cjcenizal ! Great catch with the "Load JSON". I fixed it, can you have another look?

Custom type should be detected when loading the JSON and kept as is. If multi-type are detected, the unparsed (not validated) JSON is forwarded as the UI takes care of displaying a warning.

Custom type can be a mappings configuration

{
  "dynamic": { // Custom type is a mappings configuration
    "_source": {
      "enabled": true,
      "includes": [],
      "excludes": []
    },
    "_meta": {},
    "_routing": {
      "required": false
    },
    "dynamic": true,
    "numeric_detection": false,
    "date_detection": true,
    "properties": {
      "cool": {
        "type": "text"
      },
      "name": {
        "type": "keyword"
      }
    }
  }
}

but can't be "properties"

{
  "properties": {
    "_source": {
      "enabled": true,
      "includes": [],
      "excludes": []
    },
    "_meta": {},
    "_routing": {
      "required": false
    },
    "dynamic": true,
    "numeric_detection": false,
    "date_detection": true,
    "properties": {
      "cool": {
        "type": "text"
      },
      "name": {
        "type": "keyword"
      }
    }
  }
}

Validation lib

I added some comments in the code in the mappings_validator.ts to use another lib for validation instead of io-ts. It works OK in most cases but not to prevent unknown properties on objects. I've had great success with Avj, which works both on the server and in the browser.

cjcenizal

I tested the following conditions:

Removing the custom type from an index template
Adding a custom type to an index template
Loading multiple custom types
Loading JSON with all advanced options customized

Everything looks great!

I noticed that the text for the warning callout no longer makes sense, since custom types are supported:

How about we tweak the copy to say "Multiple mapping types detected" - "The mappings for this template uses multiple types, which are not supported."

cjcenizal · 2020-01-28T18:44:27Z

...nt/public/app/components/mappings_editor/components/load_mappings/load_mappings_provider.tsx

+    }
+
+    if (hasCustomType) {
+      customType = Object.keys(unparsed)[0];


Minor nit: In the case of isMultiTypeMappings === true, then customType loses meaning because it's not the only custom type. Maybe we should change the condition to hasCustomType && !isMultiTypeMappings so that customType is left undefined if there are multiple type mappings?

cjcenizal · 2020-01-28T18:46:48Z

...nt/public/app/components/mappings_editor/components/load_mappings/load_mappings_provider.tsx

+      }
+
+      // Custom type can't be "properties", ES will not treat it as such
+      // as it is reserved for fields definition


❤️ Great comments!

cjcenizal · 2020-01-28T18:49:32Z

...acy/plugins/index_management/public/app/components/mappings_editor/lib/mappings_validator.ts

+   * It is extremely verbose, and as we can see below, there isn't a way to make the validation fail on unknown properties.
+   * We need the hack below to return errors when unknown properties are detected.
+   *
+   * TODO: I propose to refactor and use Ajv (https://ajv.js.org/) that has a much simpler implementation for the validation we need to do.


Can we move this out of comments and into an issue? That will allow for team discussion and prevent this from being lost.

CC @jloleysens curious to hear your thoughts on this too since I believe you introduced io-ts.

cjcenizal · 2020-01-28T19:02:29Z

...acy/plugins/index_management/public/app/components/mappings_editor/lib/mappings_validator.ts

@@ -235,6 +261,15 @@ export const validateMappingsConfiguration = (
    });
  }

+  if (Boolean(unknownConfigurationParameters.length)) {


Minor nit: unknownConfigurationParameters.length > 0 evaluates to a boolean too, and I think also more clearly expresses our intention here (similar comment for the condition on line 268).

…ditor-should-support-mappings-types

sebelga · 2020-01-29T06:17:24Z

Thanks for the review @cjcenizal ! I addressed your suggestions on the PR. I will merge and backport to 7.6 when CI is green.

I created an issue to move to avj validator: #56272

)

…) (#56275)

…55831 * '7.x' of github.com:elastic/kibana: (78 commits) Re-enable watcher FireFox functional test (elastic#56112) (elastic#56294) [Metrics UI] Fixing title truncation in Metrics Explorer (elastic#55917) (elastic#56248) [APM] x-axis labels on Error occurrences chart are incorrect based on Kibana timezone (elastic#55686) (elastic#56288) Migrate saved_object_save_as_checkbox directive to timelion (elastic#56114) (elastic#56286) [APM] Treat error.exception.stacktrace.line as optional (elastic#55733) (elastic#55840) Remove alerts and actions from feature catalogue (elastic#56140) (elastic#56208) Migrate UI capabilities to use new platform APIs (elastic#56070) (elastic#56207) [ML] Add functional tests for analytics UI: creation addition and regression/outlier results (elastic#56059) (elastic#56191) [SIEM] Overview page feedback (elastic#56261) (elastic#56276) [NP Cleanup] Remove ui/public/inspector (elastic#55677) (elastic#56271) [Index template] Fix editor should support mappings types (elastic#55804) fixes map index message (elastic#56104) (elastic#56194) [SIEM] [TIMELINE] Only add endpoint logo when on event.module === endgame (elastic#56263) (elastic#56269) [SIEM] Fix filters on Hosts and Network page (elastic#56234) (elastic#56267) [SIEM] Adds ability to infer the newsfeed.enabled setting (elastic#56236) (elastic#56265) [SIEM][Detection Engine] critical blocker for updated rules (elastic#56259) [SIEM] Put the notice for rules in comment block (elastic#56123) (elastic#56246) [SIEM][Detection Engine] critical blocker, fixes ordering issue that causes rules to not run the first time (elastic#56256) [Reporting/NP] Migration of Reporting Security dependency (elastic#56046) (elastic#56198) [SIEM] Add link to endpoint app through reference.url (elastic#56211) (elastic#56250) ... # Conflicts: # x-pack/plugins/watcher/public/plugin.ts

…) (#56279)

kibanamachine · 2020-02-24T07:11:28Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 1cbaf4d

History

💚 Build #23235 succeeded 1cbaf4d
💚 Build #22958 succeeded 9f281c2
💔 Build #22933 failed df98c23
💔 Build #22311 failed 028bb28
💔 Build #22298 failed de48af4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

sebelga added 6 commits January 24, 2020 12:49

Add type metadata to extract mappings definition

0bae3df

Add "include_type_name" parameter to server API req

bfc5226

Return mappings inside type if one has been detected

892afa9

Add "include_type_name" req param to client requests

7d5def9

Move query param to "sendRequest" query object

92875e4

Forward "include_type_name" api call to ES request call

de48af4

sebelga requested a review from cjcenizal January 24, 2020 08:28

sebelga added bug Fixes for quality problems that affect the customer experience Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v7.6.0 labels Jan 24, 2020

sebelga added Feature:Index Management Index and index templates UI release_note:skip Skip the PR/issue when compiling release notes labels Jan 24, 2020

Fix API integration test

028bb28

cjcenizal suggested changes Jan 24, 2020

View reviewed changes

sebelga added 8 commits January 28, 2020 12:29

Refactor client integration tests: call registerTestBed() from setup

ba056e1

Add strictness to mappings validator to not allow unknown parameters

001442c

Add test to detect custom type with mappings configuration name

a97722a

Use "dynamic" in test type detection instead of "_source"

b9ad4ac

Custom can't start with an underscore so "_source" would never be a valid type

Detect mappings type in LoadMappingsProvider

1d2a508

Add client integration tests for LoadMappignsProvider

3493e7d

Merge remote-tracking branch 'upstream/7.x' into fix/Index-template-e…

0045d0d

…ditor-should-support-mappings-types

Refactor: change helper func to "doMappingsHaveType"

df98c23

Fix TS issues

9f281c2

cjcenizal approved these changes Jan 28, 2020

View reviewed changes

sebelga added 3 commits January 29, 2020 10:32

Merge remote-tracking branch 'upstream/7.x' into fix/Index-template-e…

0f5e083

…ditor-should-support-mappings-types

Address CR changes

8fa68b5

Remove comment about io-ts lib

1cbaf4d

sebelga merged commit 9aa4324 into elastic:7.x Jan 29, 2020

sebelga deleted the fix/Index-template-editor-should-support-mappings-types branch January 29, 2020 07:05

sebelga added a commit to sebelga/kibana that referenced this pull request Jan 29, 2020

[Index template] Fix editor should support mappings types (elastic#55804

86cc635

)

This was referenced Jan 29, 2020

[7.6] [Index template] Fix editor should support mappings types (#55804) #56275

Merged

[7.5] [Index template] Fix editor should support mappings types (#55804) #56279

Merged

[Mappings editor] Bring improvements from #55804 PR to master #56282

Merged

sebelga added a commit that referenced this pull request Jan 29, 2020

[7.6] [Index template] Fix editor should support mappings types (#55804…

a0529f7

…) (#56275)

cjcenizal mentioned this pull request Jan 29, 2020

Index Template editor should support mappings types #55645

Closed

sebelga added a commit that referenced this pull request Jan 30, 2020

[Mappings editor] Bring improvements from #55804 PR to master (#56282)

ae9cbfc

sebelga added a commit that referenced this pull request Jan 31, 2020

[7.5] [Index template] Fix editor should support mappings types (#55804…

8cd0a03

…) (#56279)

sebelga added release_note:fix and removed release_note:skip Skip the PR/issue when compiling release notes labels Feb 24, 2020

sebelga mentioned this pull request Feb 24, 2020

[Docs] [Bug fix] Add missing release note information for 7.6 #58329

Closed

alisonelizabeth mentioned this pull request Jul 10, 2020

[7.x] [Composable template] Demo and PR review fixes (#71065) #71331

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Index template] Fix editor should support mappings types #55804

[Index template] Fix editor should support mappings types #55804

sebelga commented Jan 24, 2020 •

edited

Loading

elasticmachine commented Jan 24, 2020

cjcenizal left a comment •

edited

Loading

cjcenizal Jan 24, 2020

sebelga Jan 27, 2020

sebelga commented Jan 28, 2020

cjcenizal left a comment

cjcenizal Jan 28, 2020

cjcenizal Jan 28, 2020

cjcenizal Jan 28, 2020

cjcenizal Jan 28, 2020

cjcenizal Jan 28, 2020

sebelga commented Jan 29, 2020

kibanamachine commented Feb 24, 2020

[Index template] Fix editor should support mappings types #55804

[Index template] Fix editor should support mappings types #55804

Conversation

sebelga commented Jan 24, 2020 • edited Loading

Note for the reviewer

How to test

Indices created with the template

Note for 7.5 backport

Release note

elasticmachine commented Jan 24, 2020

cjcenizal left a comment • edited Loading

Choose a reason for hiding this comment

cjcenizal Jan 24, 2020

Choose a reason for hiding this comment

sebelga Jan 27, 2020

Choose a reason for hiding this comment

sebelga commented Jan 28, 2020

Validation lib

cjcenizal left a comment

Choose a reason for hiding this comment

cjcenizal Jan 28, 2020

Choose a reason for hiding this comment

cjcenizal Jan 28, 2020

Choose a reason for hiding this comment

cjcenizal Jan 28, 2020

Choose a reason for hiding this comment

cjcenizal Jan 28, 2020

Choose a reason for hiding this comment

cjcenizal Jan 28, 2020

Choose a reason for hiding this comment

sebelga commented Jan 29, 2020

kibanamachine commented Feb 24, 2020

💔 Build Failed

History

sebelga commented Jan 24, 2020 •

edited

Loading

cjcenizal left a comment •

edited

Loading