[Index Management] Add data streams functionality to indices tab #67940

jloleysens · 2020-06-02T12:33:20Z

Summary

Migrate the URL to be the source of truth for the includeHiddenIndices toggle
Add a new url param dataStreams that filters out indices based on the data stream they are backing

How to test

Create a data stream in your ES cluster: PUT _data_stream/my-data-stream
Go to Index Management plugin in Kibana (requires x-pack). There should be no indices visible if this is a fresh cluster.
Press the include hidden toggle and all hidden indices should appear including my-data-stream-000001
You should be able to navigate to the url you are on with &filter=dataStream:my-data-stream added to the hash portion of your url. This filters for just the data stream backing index my-data-stream-000001. Something like: /app/kibana#/management/data/index_management/indices?includeHiddenIndices=true&filter=dataStream:my-data-stream

Notes to reviewer

A refactor was done to remove the middleware for syncing the URL to the store. State was moved down to the index table where it is derived from the location object we receive from withRouter.
The selector getFilteredIndices still does the work of filtering over all indices but now also depends on the component props, specifically the location.search value to determine what all it should be filtering. More information on this here.
We can do the same for filter as we did for includeHiddenIndices; move the source of truth to the URL. This will be a slightly bigger refactor because of how it is currently implemented and can be set by other plugins (like ILM). The filter will need to be converted to a string value and parsed into Query by us. I would consider this follow up work, but it would be nice to have the actual text in the filter sync up with the URL as the user types.

Screenshots

Filtering based on hidden

Filtering based on text filter and include hidden

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

- Added the data streams column - Moved state down to component (using withRouter) - Removed previous middleware for syncing url hash param - call history.push to update the url in the component

elasticmachine · 2020-06-02T12:33:21Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

jloleysens · 2020-06-02T17:49:16Z

@cjcenizal This is not 100% ready for review as I thought. I just realised I missed something in this implementation, will re-ping when it s ready.

jloleysens · 2020-06-02T18:42:57Z

@cjcenizal Ok, ready now!

sebelga · 2020-06-03T14:35:40Z

...gins/index_management/public/application/sections/home/index_list/index_table/index_table.js

-
-    if (filterFromURI) {
-      const decodedFilter = decodeURIComponent(filterFromURI);
+    this.interval = setInterval(


Just a note for our future selves: this is the code we want 4 (or more!) 👀 on when we get flakiness in our tests 😊

Hi! I am the future self 😊 : don't worry, it's okay, there is jest. useFakeTimers() that gives us control over this.

cjcenizal

Tested locally, code LGTM! Great work, JL! 👏 I found one bug that I think we should fix before merging, and one optimization I think we can address later.

Sort bug

Sorting for the data stream column is incorrect. We can fix this by adding a sorter to the sort_table.ts service.

dataStream: stringSort('dataStream'),

Filter optimization

Filter on the name is subject to unexpected behavior because it's possible that one data stream's name could contain another data stream's name. Here's a silly example but it demonstrates the result:

This is an edge case so I don't think we should spend too much time right now trying to get this right, but I imagine a solution would entail filtering on the indices' dataStream field.

...gins/index_management/public/application/sections/home/index_list/index_table/index_table.js

…ms-column * 'master' of github.com:elastic/kibana: (63 commits) remove scripts. prettire update has been done (elastic#68130) Closes elastic#68055 by detecting the local Kibana version and using that as (elastic#68198) [apm] docs: add deployment annotation example (elastic#67408) [ML] Extend population preview chart to show actual and typical value (elastic#67569) Refactor index management client integration tests for scalability (elastic#67917) Add generator function that creates multiple alerts (elastic#67713) chore(NA): remove config arg from os packages (elastic#67871) [Reporting] Move code out of Legacy (elastic#67904) [Metrics UI] Add overrides to Snapshot API to support alert previews (elastic#68125) [Security] [Cases] Manage timeline UI API (elastic#67719) [ENDPOINT][INGEST]Task/endpoint ingest update (elastic#67234) Fix code coverage for jest, upload merged reports (elastic#68149) Update documentation/examples of deprecated namespaceAgnostic field (elastic#68039) [DOCS] Updates Canvas docs with new menus (elastic#66061) chore(NA): avoids imports of server or public code into common (elastic#67231) [SIEM] Fix GetOneTimeline graphql type (elastic#68137) skip flaky suite (elastic#67838) [Uptime] Add loading message for monitor list no items (elastic#67378) [Ingest Manager] Update indexing strategy docs to use dataset.* (elastic#68068) [Ingest Manager] Fix datasource validation for streams without vars (elastic#67950) ... # Conflicts: # x-pack/plugins/index_management/__jest__/client_integration/helpers/index.ts # x-pack/plugins/index_management/__jest__/client_integration/home.test.ts # x-pack/plugins/index_management/__jest__/client_integration/home/index_templates_tab.helpers.ts

jloleysens · 2020-06-04T09:12:18Z

Thanks for the review @cjcenizal !

Sorting for the data stream column is incorrect.

I didn't realise we were specifying this imperatively! TIL :). I thought it better to present the user with data_stream as the key instead of dataStream. This will feel closer to the key conventions of ES.

Filter on the name is subject to unexpected behavior because it's possible that one data stream's name could contain another data stream's name

We should be able to address this by using the appropriate query syntax: data_stream=my-data-stream (this indicates a stricter check than :). (more info here)

The URL would become: /app/management/data/index_management/indices?filter=data_stream=my-data-stream&includeHiddenIndices=true.

This would match only data_stream fields that are strictly equal to the search term.

Since the changes I made are quite minor I'm going to merge once CI is green on this 👍

kibanamachine · 2020-06-04T10:57:41Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: fd1cdbf

History

💔 Build #51677 failed 7f7b959
💚 Build #51189 succeeded 20f9649
💚 Build #51177 succeeded 0f5f80a
💔 Build #51094 failed c40ade5
💔 Build #51066 failed a48df33

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…stic#67940) * First iteration of data streams in index management - Added the data streams column - Moved state down to component (using withRouter) - Removed previous middleware for syncing url hash param - call history.push to update the url in the component * Updated jest tests * refactor: includeHidden -> includeHiddenIndices * Fix types * Fix jest test and remove getting filter param from parent * Small refactor to read url params in render function * Clean up old data streams code * Fix sorting on data stream field in table * dataStream -> data_stream * qs > * as qs

* master: (26 commits) [Console]remove completion for type for filter queries and aggs (elastic#68103) [ML] Transforms: Filter aggregation support (elastic#67591) [ES UI Shared] Monaco XJSON (elastic#67485) [Index Management] Add data streams functionality to indices tab (elastic#67940) [Discover] Fix renaming of saved search not displayed in breadcrumb (elastic#67577) [SECURITY] Rename siem plugin to security_solution (elastic#67902) [Uptime] Fix Telemetry Api flaky test (elastic#67358) [Data plugin] Add configuration property to enable / disable autocomplete (elastic#67847) remove scripts. prettire update has been done (elastic#68130) Closes elastic#68055 by detecting the local Kibana version and using that as (elastic#68198) [apm] docs: add deployment annotation example (elastic#67408) [ML] Extend population preview chart to show actual and typical value (elastic#67569) Refactor index management client integration tests for scalability (elastic#67917) Add generator function that creates multiple alerts (elastic#67713) chore(NA): remove config arg from os packages (elastic#67871) [Reporting] Move code out of Legacy (elastic#67904) [Metrics UI] Add overrides to Snapshot API to support alert previews (elastic#68125) [Security] [Cases] Manage timeline UI API (elastic#67719) [ENDPOINT][INGEST]Task/endpoint ingest update (elastic#67234) Fix code coverage for jest, upload merged reports (elastic#68149) ...

) (#68233) * First iteration of data streams in index management - Added the data streams column - Moved state down to component (using withRouter) - Removed previous middleware for syncing url hash param - call history.push to update the url in the component * Updated jest tests * refactor: includeHidden -> includeHiddenIndices * Fix types * Fix jest test and remove getting filter param from parent * Small refactor to read url params in render function * Clean up old data streams code * Fix sorting on data stream field in table * dataStream -> data_stream * qs > * as qs

jloleysens added 3 commits June 2, 2020 13:31

First iteration of data streams in index management

1140757

- Added the data streams column - Moved state down to component (using withRouter) - Removed previous middleware for syncing url hash param - call history.push to update the url in the component

Updated jest tests

1de542a

refactor: includeHidden -> includeHiddenIndices

a48df33

jloleysens added Feature:Index Management Index and index templates UI v8.0.0 Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more release_note:skip Skip the PR/issue when compiling release notes v7.9.0 labels Jun 2, 2020

jloleysens requested a review from cjcenizal June 2, 2020 12:33

jloleysens requested a review from a team as a code owner June 2, 2020 12:33

jloleysens changed the title ~~Im/data streams column~~ [Index Management] Add data streams functionality to indices tab Jun 2, 2020

Fix types

c40ade5

jloleysens added 2 commits June 2, 2020 19:58

Fix jest test and remove getting filter param from parent

0f5f80a

Small refactor to read url params in render function

dc5470a

Clean up old data streams code

20f9649

sebelga reviewed Jun 3, 2020

View reviewed changes

cjcenizal approved these changes Jun 3, 2020

View reviewed changes

...gins/index_management/public/application/sections/home/index_list/index_table/index_table.js Outdated Show resolved Hide resolved

jloleysens added 3 commits June 4, 2020 10:33

Fix sorting on data stream field in table

764ab74

dataStream -> data_stream

ff668a7

qs > * as qs

fd1cdbf

jloleysens merged commit c97f5fb into elastic:master Jun 4, 2020

jloleysens deleted the im/data-streams-column branch June 4, 2020 11:37

jloleysens mentioned this pull request Jun 4, 2020

[7.x] [Index Management] Add data streams functionality to indices tab (#67940) #68233

Merged

alisonelizabeth mentioned this pull request Jul 7, 2020

[Index Management] Fix broken app links #71007

Merged

cjcenizal mentioned this pull request Nov 18, 2020

Support hidden data streams in Index Management #83709

Closed

kibanamachine mentioned this pull request Jan 25, 2022

[Index Management] Optimize ES response for indices list #123629

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Index Management] Add data streams functionality to indices tab #67940

[Index Management] Add data streams functionality to indices tab #67940

jloleysens commented Jun 2, 2020 •

edited

Loading

elasticmachine commented Jun 2, 2020

jloleysens commented Jun 2, 2020

jloleysens commented Jun 2, 2020

sebelga Jun 3, 2020

sebelga Sep 23, 2021

cjcenizal left a comment •

edited

Loading

jloleysens commented Jun 4, 2020

kibanamachine commented Jun 4, 2020

[Index Management] Add data streams functionality to indices tab #67940

[Index Management] Add data streams functionality to indices tab #67940

Conversation

jloleysens commented Jun 2, 2020 • edited Loading

Summary

How to test

Notes to reviewer

Screenshots

Checklist

For maintainers

elasticmachine commented Jun 2, 2020

jloleysens commented Jun 2, 2020

jloleysens commented Jun 2, 2020

sebelga Jun 3, 2020

Choose a reason for hiding this comment

sebelga Sep 23, 2021

Choose a reason for hiding this comment

cjcenizal left a comment • edited Loading

Choose a reason for hiding this comment

Sort bug

Filter optimization

jloleysens commented Jun 4, 2020

kibanamachine commented Jun 4, 2020

💚 Build Succeeded

History

jloleysens commented Jun 2, 2020 •

edited

Loading

cjcenizal left a comment •

edited

Loading