[APM] Remove ui filters #84526

sorenlouv · 2020-11-30T15:20:12Z

Related

TODO:

7.12

Remove UI filters from UI (Remove UI filters from UI #89793)

7.13

Remove esFilters from Elasticsearch queries (Remove kuery from uiFilters #91932)
Extract environment to separate API argument (Remove environment from uiFilters #89647)
Move /api/apm/ui_filters/environments to /api/apm/environments (Move ui_filters API endpoints #90900)
Extract kuery to separate API argument (Remove kuery from uiFilters #91932)
Move ui_filters API into UX directories (Move ui_filters API endpoints #90900)
Remove projections
Replace custom search bar with standard Kibana search bar ([APM] Replace custom kuery bar with shared SearchBar / TopNavMenu #82751)

UI filters were released 1.5 years ago in 7.4. Since then we haven't iterated on them due to concerns about their direction. Implementing them introduced a high degree of complexity and technical debt and in their current state this cost is not worth it.

UI Filters doesn't introduce any new capabilities that cannot be gained from the search bar. Additionally, ui filters is unique to APM and are as a concept inconsistent with the rest of Kibana.

To align with the rest of Kibana, and reduce complexity I suggest we remove the UI filters.

UI Filters are located on the left, under the heading "Filters"

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-11-30T15:20:14Z

Pinging @elastic/apm-ui (Team:apm)

dgieselaar · 2020-11-30T15:23:36Z

++, this will also allow us to aggregate on less fields when aggregation transaction metrics, which will make them more efficient.

alex-fedotyev · 2020-11-30T22:11:19Z

I think this would be a good first step in improving filtering experience.

Based on my interactions with customers so far, our customer try to filter a dataset using following steps:

Try to find attributes which have multiple values (for example, user finds that there are cloud availability zone and service version which have multiple values during selected time frame).
Identify whether any of those value stand out from the rest in a one or more measurements, typically number of calls vs. latency vs. error rates:
1. This could be done by either splitting by the attribute and showing top N values and their measurements (for example, compare service performance across all availability zones).
2. Another option could be to compare original dataset with a subset defined by a specific value (for example, compare latest canary deployment performance vs entire service).
In case one or more values are showing significant impact, apply the filter using that value to perform further troubleshooting.

I included this workflow to highlight that users will need filtering capabilities in the end.

Will we be able to introduce filters from Kibana into APM when we remove UI filters?

sorenlouv · 2020-11-30T22:44:51Z

Will we be able to introduce filters from Kibana into APM when we remove UI filters?

Yes, we can have a similar experience to how the rest of Kibana does this.

alex-fedotyev · 2020-11-30T23:51:38Z

I think this would be perfect.

@nehaduggal - how are you expecting transaction significant tags to work? Would you need filters for that?

nehaduggal · 2020-12-01T03:47:51Z

We don't need filters for significant terms. The plan is to surface the sig terms for transactions with high latency, errors or throughput regardless of transaction type. Taking away filters shouldn't affect that functionality in any way.

gimmic · 2020-12-01T15:44:59Z

Forgive my ignorance on this, but how will a user be able to quickly toggle a field query quickly between 'and', 'not', 'disabled'?

I understand that it can be changed manually in the query, but for high level dashboards UI filter elements are nice as toggles. Is this what is being referenced?
(It's possible I am misunderstanding the context of "remove ui filters")

sorenlouv · 2020-12-01T22:53:33Z

@iankoetter Thanks for chiming in. I've updated the issue with an screenshot to make it more clear.
This is how you could can achieve the same thing in the future:

Using a filter

Using kql

This should be more recognisable for people familiar with Kibana. Do you prefer the old ui filters approach?

formgeist · 2020-12-02T12:47:58Z

From a product design and UX point of view, removing the UI filters seems like a hasty decision based on performance issues, without much thought to what we're removing from the experience in APM.

The query bar filtering experience is not meant for the average user, because it requires them to go through a lot of steps for even the simplest filter and they need to know what they're looking for. The most important feature of the UI filters was the discoverability of those contextual values.

I understand that with the metadata and significant tags, we will eventually allow the user to instantly filter by those values and we would otherwise present contextual information such as service version and cloud information in the header.

Let's at least have a more formal plan for what we want our filtering experience to look like before we go and rip out the existing feature.

gimmic · 2020-12-02T16:03:46Z

Edit: I realize I have overlooked the primary use here, APM. If this is not a planned change for Kibana globally, then my input should be discarded!

So for junior analysts who may not be familiar with KQL syntax, or even know what fields are useful to filter against, how do we provide a frame of reference? Today this is one of the functions provided by UI filter elements on views. As a simple example we might have a saved discover search with that comes pre-enabled with some exclusion filters. The idea is to make it visible to the analyst their view is modified by those filters, but also give them the opportunity to quickly toggle it to an enabled or disabled state as well as inverse the filter(s). Those pre-canned UI elements are useful.

Example:

Removing filters also adds to the length of KQL queries in the search bar, which could also become an editing challenge. If I've got a long enough query to add horizontal scroll to KQL input, then when I want to "toggle" a field inclusion/exclusion, I'll have to find it in the text and manually change the text. Think of editing a very long URL in your browser's address bar..

With the filter options gone as an UI element, what's the alternative there? It seems like this would be removing knobs in favor of manual KQL typing which is more ephemeral in nature. Generally I am a fan of "less clicks", but not when removal of clicks adds complexity. Pivoting around in a dataset with a collection of "common knobs" makes much more sense than having to type those same knobs out over and over for each query/pivot.

The workflow generally goes as follows:
Is this a one time query filter? -> KQL
Is this a field/value we might often want to filter as an AND/NOT, but sometimes disable in general? -> UI filter element.

formgeist · 2020-12-03T10:05:55Z

@iankoetter Thanks for providing some more insights. Indeed, the UI filters removal is only related to the APM implementation. As discussed we're intending to make the UI filters experience consistent with what Kibana does today (e.g. what you see in Discover or Dashboards); that's allowing you to add saved searches, change how filters are applied (e.g. NOT), and more.

More specifically, I was referring to the user experience of adding a new filter in that KQL search bar. The user is going to get a full list of possible fields, whether they relate to APM/Observability or not, and have to remember which field applies to which part of the dataset. It's definitely something that can be improved upon.

formgeist · 2021-01-06T17:52:42Z

I've opened an enhancement issue to improve filtering capabilities in the UI possibly as part of the removal of the UI filters. #87537

formgeist · 2021-01-14T15:48:57Z

I've been discussing with the (RUM) UX app team about how this affects the UI filters in that application and I imagine we'd need to move over the UI filters code as they will continue to use them in lieu of not having a search bar and there's no replacement for the filtering experience at this time. Any thoughts @sqren ? cc @shahzad31

sorenlouv · 2021-01-14T17:01:38Z

I think it should be doable to only remove ui filters from the APM codebase.

sorenlouv · 2021-02-10T00:16:56Z

@smith Are you still working on this or should I move it back?

smith · 2021-02-10T01:32:50Z

@sqren definitely still working on it. See the linked issues in the description.

For a non-Java service, the previous link was like: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?rangeFrom=now-15m&rangeTo=now ``` which did not filter by the `service.node.name`. It now is: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?kuery=service.node.name:%226a7f116fe344aee7e92fceeb426cbfdf6a534a8e3ba6345c16a47793eba6daf5%22&rangeFrom=now-15m&rangeTo=now ```` Which links to the metrics page with the filter applied. The component is using a `MetricOverviewLink` which was using a `EuiLink` and passing throught the props, including `mergeQuery`, which includes the `kuery` parameter. Replace the `EuiLink` with an `APMLink` which does use the `mergeQuery` prop and does pass the parameters through correctly. Looks like this was changed to an `EuiLink` by a refactor in elastic#86986. Since we'll be making some further changes to how `kuery` is handled in elastic#84526, I'm just making the minimal change to fix this bug at this time.

For a non-Java service, the previous link was like: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?rangeFrom=now-15m&rangeTo=now ``` which did not filter by the `service.node.name`. It now is: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?kuery=service.node.name:%226a7f116fe344aee7e92fceeb426cbfdf6a534a8e3ba6345c16a47793eba6daf5%22&rangeFrom=now-15m&rangeTo=now ```` Which links to the metrics page with the filter applied. The component is using a `MetricOverviewLink` which was using a `EuiLink` and passing throught the props, including `mergeQuery`, which includes the `kuery` parameter. Replace the `EuiLink` with an `APMLink` which does use the `mergeQuery` prop and does pass the parameters through correctly. Looks like this was changed to an `EuiLink` by a refactor in #86986. Since we'll be making some further changes to how `kuery` is handled in #84526, I'm just making the minimal change to fix this bug at this time.

For a non-Java service, the previous link was like: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?rangeFrom=now-15m&rangeTo=now ``` which did not filter by the `service.node.name`. It now is: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?kuery=service.node.name:%226a7f116fe344aee7e92fceeb426cbfdf6a534a8e3ba6345c16a47793eba6daf5%22&rangeFrom=now-15m&rangeTo=now ```` Which links to the metrics page with the filter applied. The component is using a `MetricOverviewLink` which was using a `EuiLink` and passing throught the props, including `mergeQuery`, which includes the `kuery` parameter. Replace the `EuiLink` with an `APMLink` which does use the `mergeQuery` prop and does pass the parameters through correctly. Looks like this was changed to an `EuiLink` by a refactor in elastic#86986. Since we'll be making some further changes to how `kuery` is handled in elastic#84526, I'm just making the minimal change to fix this bug at this time.

…1842) For a non-Java service, the previous link was like: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?rangeFrom=now-15m&rangeTo=now ``` which did not filter by the `service.node.name`. It now is: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?kuery=service.node.name:%226a7f116fe344aee7e92fceeb426cbfdf6a534a8e3ba6345c16a47793eba6daf5%22&rangeFrom=now-15m&rangeTo=now ```` Which links to the metrics page with the filter applied. The component is using a `MetricOverviewLink` which was using a `EuiLink` and passing throught the props, including `mergeQuery`, which includes the `kuery` parameter. Replace the `EuiLink` with an `APMLink` which does use the `mergeQuery` prop and does pass the parameters through correctly. Looks like this was changed to an `EuiLink` by a refactor in #86986. Since we'll be making some further changes to how `kuery` is handled in #84526, I'm just making the minimal change to fix this bug at this time. Co-authored-by: Nathan L Smith <nathan.smith@elastic.co>

* Make kuery a standalone query parameter instead of part of uiFilters * Move getEsFilters helper to RUM * Move query utils from "common" to "server" (it uses esKuery from the data plugin, which is exported from server, not common.) * Move uiFiltersRt to RUM routes References #84526.

* Make kuery a standalone query parameter instead of part of uiFilters * Move getEsFilters helper to RUM * Move query utils from "common" to "server" (it uses esKuery from the data plugin, which is exported from server, not common.) * Move uiFiltersRt to RUM routes References elastic#84526. # Conflicts: # x-pack/plugins/apm/public/context/annotations/annotations_context.tsx

* Make kuery a standalone query parameter instead of part of uiFilters * Move getEsFilters helper to RUM * Move query utils from "common" to "server" (it uses esKuery from the data plugin, which is exported from server, not common.) * Move uiFiltersRt to RUM routes References #84526. # Conflicts: # x-pack/plugins/apm/public/context/annotations/annotations_context.tsx

For a non-Java service, the previous link was like: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?rangeFrom=now-15m&rangeTo=now ``` which did not filter by the `service.node.name`. It now is: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?kuery=service.node.name:%226a7f116fe344aee7e92fceeb426cbfdf6a534a8e3ba6345c16a47793eba6daf5%22&rangeFrom=now-15m&rangeTo=now ```` Which links to the metrics page with the filter applied. The component is using a `MetricOverviewLink` which was using a `EuiLink` and passing throught the props, including `mergeQuery`, which includes the `kuery` parameter. Replace the `EuiLink` with an `APMLink` which does use the `mergeQuery` prop and does pass the parameters through correctly. Looks like this was changed to an `EuiLink` by a refactor in elastic#86986. Since we'll be making some further changes to how `kuery` is handled in elastic#84526, I'm just making the minimal change to fix this bug at this time.

…3240) For a non-Java service, the previous link was like: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?rangeFrom=now-15m&rangeTo=now ``` which did not filter by the `service.node.name`. It now is: ``` http://localhost:5601/kbn/app/apm/services/opbeans-python/metrics?kuery=service.node.name:%226a7f116fe344aee7e92fceeb426cbfdf6a534a8e3ba6345c16a47793eba6daf5%22&rangeFrom=now-15m&rangeTo=now ```` Which links to the metrics page with the filter applied. The component is using a `MetricOverviewLink` which was using a `EuiLink` and passing throught the props, including `mergeQuery`, which includes the `kuery` parameter. Replace the `EuiLink` with an `APMLink` which does use the `mergeQuery` prop and does pass the parameters through correctly. Looks like this was changed to an `EuiLink` by a refactor in #86986. Since we'll be making some further changes to how `kuery` is handled in #84526, I'm just making the minimal change to fix this bug at this time.

smith · 2021-03-09T14:28:35Z

Closing this. Opened #94095 to remove projections. #82751 is open to move to the standard search bar.

sorenlouv added [zube]: Inbox discuss Team:APM All issues that need APM UI Team support labels Nov 30, 2020

sorenlouv mentioned this issue Dec 1, 2020

[APM] Meta issue: Performance as a feature #84661

Closed

7 tasks

sorenlouv added v7.12.0 [zube]: Backlog and removed [zube]: Inbox labels Dec 1, 2020

sorenlouv added [zube]: 7.12 and removed [zube]: Backlog labels Dec 18, 2020

cauemarcondes mentioned this issue Dec 29, 2020

[APM] Blank page when selecting a future time range #86964

Closed

formgeist mentioned this issue Jan 6, 2021

[APM] Improving the filtering experience for individual field values #87537

Open

sorenlouv mentioned this issue Jan 20, 2021

[APM] Discovery - need to re-evaluate implementation of filters and kuery bar across APM UI #72778

Closed

felixbarny mentioned this issue Jan 20, 2021

APM Central Management - Ability to apply configuration to a subset of agents with same service & environment elastic/apm#396

Open

smith added [zube]: In Progress and removed [zube]: 7.12 labels Jan 25, 2021

smith self-assigned this Jan 25, 2021

sorenlouv removed the discuss label Jan 26, 2021

smith mentioned this issue Feb 3, 2021

Remove UI filters from UI #89793

Merged

smith mentioned this issue Feb 10, 2021

Move ui_filters API endpoints #90900

Merged

smith mentioned this issue Feb 18, 2021

Pass service node name in query for instance table links #91796

Merged

This was referenced Feb 19, 2021

Remove kuery from uiFilters #91932

Merged

WIP: Remove projections #92088

Closed

[APM] Make environment and kuery parameters required #92286

Closed

smith closed this as completed Mar 9, 2021

zube bot added [zube]: Done and removed [zube]: In Progress labels Mar 9, 2021

graphaelli removed the [zube]: Done label Apr 21, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Remove ui filters #84526

[APM] Remove ui filters #84526

sorenlouv commented Nov 30, 2020 •

edited

Loading

elasticmachine commented Nov 30, 2020

dgieselaar commented Nov 30, 2020

alex-fedotyev commented Nov 30, 2020

sorenlouv commented Nov 30, 2020

alex-fedotyev commented Nov 30, 2020

nehaduggal commented Dec 1, 2020

gimmic commented Dec 1, 2020 •

edited

Loading

sorenlouv commented Dec 1, 2020

formgeist commented Dec 2, 2020

gimmic commented Dec 2, 2020 •

edited

Loading

formgeist commented Dec 3, 2020

formgeist commented Jan 6, 2021

formgeist commented Jan 14, 2021

sorenlouv commented Jan 14, 2021

sorenlouv commented Feb 10, 2021

smith commented Feb 10, 2021

smith commented Mar 9, 2021

[APM] Remove ui filters #84526

[APM] Remove ui filters #84526

Comments

sorenlouv commented Nov 30, 2020 • edited Loading

elasticmachine commented Nov 30, 2020

dgieselaar commented Nov 30, 2020

alex-fedotyev commented Nov 30, 2020

sorenlouv commented Nov 30, 2020

alex-fedotyev commented Nov 30, 2020

nehaduggal commented Dec 1, 2020

gimmic commented Dec 1, 2020 • edited Loading

sorenlouv commented Dec 1, 2020

Using a filter

Using kql

formgeist commented Dec 2, 2020

gimmic commented Dec 2, 2020 • edited Loading

formgeist commented Dec 3, 2020

formgeist commented Jan 6, 2021

formgeist commented Jan 14, 2021

sorenlouv commented Jan 14, 2021

sorenlouv commented Feb 10, 2021

smith commented Feb 10, 2021

smith commented Mar 9, 2021

sorenlouv commented Nov 30, 2020 •

edited

Loading

gimmic commented Dec 1, 2020 •

edited

Loading

gimmic commented Dec 2, 2020 •

edited

Loading