[ML] Data Frames: Fix to avoid queries for indices with more than 1024 fields. #37183

walterra · 2019-05-27T13:07:49Z

Summary

7.0 introduced indices.query.bool.max_clause_count which defaults to 1024. This can break certain queries (e.g. simple_query) for indices which have more than 1024 fields (e.g. certain beats indices). The optional data frames query uses simple_query and could therefor break the source index preview as well the pivot preview and pivot job itself given these conditions.

This PR introduces the following fixes for 7.2:

Originally the default query (* used for simple_query) was always applied for source index previews and pivot previews. A new check isDefaultQuery() will now allow a) the source index preview to use a more efficient match_all query and b) avoid adding the query to the pivot config. This avoids triggering the max_clause_count when no optional query is set.
If an index has more than 1024 fields, the input form for an optional query will be hidden. A helper text explains the reasoning. This avoids triggering max_clause_count related errors from within the UI. A user can still copy a UI created config to the clipboard and add an optional query in Kibana dev console.
Additionally, this PR adds a fix to format date fields in the source index preview table using moment-timezone and formatHumanReadableDateTimeSeconds to display dates with the correct timezone.

The scope of this PR is to fix max_clause_count related issues with minimal code churn for 7.2. We will revisit this for 7.3 (e.g. possibly moving away from simple_query to KQL).

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

This was checked for cross-browser compatibility, including a check against IE11
Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
~~Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
This was checked for keyboard-only and screenreader accessibility

elasticmachine · 2019-05-27T13:07:52Z

Pinging @elastic/ml-ui

elasticmachine · 2019-05-27T14:08:03Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

…lt search.

elasticmachine · 2019-05-27T15:58:26Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

alvarezmelissa87

LGTM ⚡️

…4 fields. (elastic#37183) - 7.0 introduced indices.query.bool.max_clause_count which defaults to 1024. This can break certain queries (e.g. simple_query) for indices which have more than 1024 fields (e.g. certain beats indices). The optional data frames query uses simple_query and could therefor break the source index preview as well the pivot preview and pivot job itself given these conditions. - Originally the default query (* used for simple_query) was always applied for source index previews and pivot previews. A new check isDefaultQuery() will now allow a) the source index preview to use a more efficient match_all query and b) avoid adding the query to the pivot config. This avoids triggering the max_clause_count when no optional query is set. - If an index has more than 1024 fields, the input form for an optional query will be hidden. A helper text explains the reasoning. This avoids triggering max_clause_count related errors from within the UI. A user can still copy a UI created config to the clipboard and add an optional query in Kibana dev console. - Additionally, this PR adds a fix to format date fields in the source index preview table using moment-timezone and formatHumanReadableDateTimeSeconds to display dates with the correct timezone.

…4 fields. (#37183) (#37227) - 7.0 introduced indices.query.bool.max_clause_count which defaults to 1024. This can break certain queries (e.g. simple_query) for indices which have more than 1024 fields (e.g. certain beats indices). The optional data frames query uses simple_query and could therefor break the source index preview as well the pivot preview and pivot job itself given these conditions. - Originally the default query (* used for simple_query) was always applied for source index previews and pivot previews. A new check isDefaultQuery() will now allow a) the source index preview to use a more efficient match_all query and b) avoid adding the query to the pivot config. This avoids triggering the max_clause_count when no optional query is set. - If an index has more than 1024 fields, the input form for an optional query will be hidden. A helper text explains the reasoning. This avoids triggering max_clause_count related errors from within the UI. A user can still copy a UI created config to the clipboard and add an optional query in Kibana dev console. - Additionally, this PR adds a fix to format date fields in the source index preview table using moment-timezone and formatHumanReadableDateTimeSeconds to display dates with the correct timezone.

elasticmachine · 2019-05-28T16:02:54Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

…4 fields. (#37183) (#37226) - 7.0 introduced indices.query.bool.max_clause_count which defaults to 1024. This can break certain queries (e.g. simple_query) for indices which have more than 1024 fields (e.g. certain beats indices). The optional data frames query uses simple_query and could therefor break the source index preview as well the pivot preview and pivot job itself given these conditions. - Originally the default query (* used for simple_query) was always applied for source index previews and pivot previews. A new check isDefaultQuery() will now allow a) the source index preview to use a more efficient match_all query and b) avoid adding the query to the pivot config. This avoids triggering the max_clause_count when no optional query is set. - If an index has more than 1024 fields, the input form for an optional query will be hidden. A helper text explains the reasoning. This avoids triggering max_clause_count related errors from within the UI. A user can still copy a UI created config to the clipboard and add an optional query in Kibana dev console. - Additionally, this PR adds a fix to format date fields in the source index preview table using moment-timezone and formatHumanReadableDateTimeSeconds to display dates with the correct timezone.

…4 fields. (elastic#37183) - 7.0 introduced indices.query.bool.max_clause_count which defaults to 1024. This can break certain queries (e.g. simple_query) for indices which have more than 1024 fields (e.g. certain beats indices). The optional data frames query uses simple_query and could therefor break the source index preview as well the pivot preview and pivot job itself given these conditions. - Originally the default query (* used for simple_query) was always applied for source index previews and pivot previews. A new check isDefaultQuery() will now allow a) the source index preview to use a more efficient match_all query and b) avoid adding the query to the pivot config. This avoids triggering the max_clause_count when no optional query is set. - If an index has more than 1024 fields, the input form for an optional query will be hidden. A helper text explains the reasoning. This avoids triggering max_clause_count related errors from within the UI. A user can still copy a UI created config to the clipboard and add an optional query in Kibana dev console. - Additionally, this PR adds a fix to format date fields in the source index preview table using moment-timezone and formatHumanReadableDateTimeSeconds to display dates with the correct timezone.

[ML] Fix to avoid queries for indices with more than 1024 fields.

de7518e

walterra added bug Fixes for quality problems that affect the customer experience review :ml v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.2.0 v7.3.0 Feature:Transforms ML transforms labels May 27, 2019

walterra requested review from pheyos, alvarezmelissa87, peteharverson and jgowdyelastic May 27, 2019 13:07

walterra requested a review from a team as a code owner May 27, 2019 13:07

walterra self-assigned this May 27, 2019

walterra added 2 commits May 27, 2019 16:23

[ML] Fixes test. The pivot preview should no longer include the defau…

3e0233c

…lt search.

[ML] Fixes application of timezone to source index preview date fields.

2e8f181

alvarezmelissa87 approved these changes May 28, 2019

View reviewed changes

walterra merged commit 959416a into elastic:master May 28, 2019

walterra deleted the ml-data-frames-fix-max-fields-query-issue branch May 28, 2019 13:58

walterra mentioned this pull request May 28, 2019

[7.x] [ML] Data Frames: Fix to avoid queries for indices with more than 1024 fields. (#37183) #37226

Merged

walterra mentioned this pull request May 28, 2019

[7.2] [ML] Data Frames: Fix to avoid queries for indices with more than 1024 fields. (#37183) #37227

Merged

walterra mentioned this pull request Sep 28, 2020

KQL Search in the Transform UI doesn't support index patterns with > 1024 fields #64324

Closed

walterra mentioned this pull request Oct 22, 2020

[ML] Transforms: Remove index field limitation for custom query. #81467

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Data Frames: Fix to avoid queries for indices with more than 1024 fields. #37183

[ML] Data Frames: Fix to avoid queries for indices with more than 1024 fields. #37183

walterra commented May 27, 2019 •

edited

Loading

elasticmachine commented May 27, 2019

elasticmachine commented May 27, 2019

elasticmachine commented May 27, 2019

alvarezmelissa87 left a comment

elasticmachine commented May 28, 2019

[ML] Data Frames: Fix to avoid queries for indices with more than 1024 fields. #37183

[ML] Data Frames: Fix to avoid queries for indices with more than 1024 fields. #37183

Conversation

walterra commented May 27, 2019 • edited Loading

Summary

Checklist

elasticmachine commented May 27, 2019

elasticmachine commented May 27, 2019

💔 Build Failed

elasticmachine commented May 27, 2019

💚 Build Succeeded

alvarezmelissa87 left a comment

Choose a reason for hiding this comment

elasticmachine commented May 28, 2019

💚 Build Succeeded

walterra commented May 27, 2019 •

edited

Loading