[Discover][ES|QL] Performance troubles when there are lots of fields

[kertal]

This appears to be a UI performance problem in the Lens Embeddable, surfacing when using ES|QL in Discover. When adding the visualization on a Dashboard, it's fast and snappy. But when using a many fields index pattern in Discover with ES|QL, I've waited about a minute to have the histogram rendered, and the browser was blocked then.

How to reproduce, first you need to add the many_fields functional test data to Elasticsearch

1. node scripts/es_archiver --kibana-url=http://elastic:changeme@localhost:5601 --es-url=http://elastic:changeme@localhost:9200 load test/functional/fixtures/es_archiver/many_fields

2. Navigate to Discover, switch to ES|QL mode and enter FROM indices-stats | LIMIT 10

In my local instance, while documents returned fast, the histogram was stuck in loading state, for >60s. It was not waiting for any request to be returned, there were some calculations goint on :

In the chrome performance problem it looked like this

The browser is busy with Javascript calculations, sady, it's not clear what the browser is doing

Trying the performance tools in Firefox, gave me a better hint:

So it seems to be about src/plugins/chart_expressions/expression_xy/public/helpers/data_layers.tsx, which contains many iteration on all columns a.k.a files. in the many_fields data we hava about 10k fields. This is definitely an edge case, but also a smaller amount of fields should benefit from less iterations in this part of the code. I've added console.count for iteration counting, and when there are 10k fields, it appears that there are > 140k iterations.

Here's a draft PR that can be used for iterations:
#199589

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[elasticmachine]

Pinging @elastic/kibana-esql (Team:ESQL)

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[jughosta]

Worth noting that it's not a usual histogram case. It's when Discover passes already fetched data to Lens via table prop.

This is why it's slow only on Discover page (where Lens gets data from props) and fast on Dashboard page (where Lens fetches data directly).

[stratoula]

The problem is in the histogram implementation. The logic changed from my initial implementation. What is happening here is that you run the suggestions api twice:

1. Get all suggestions
2. Get histogram suggestion

In case of histogram the first should not run at all, is useless (is not performant to run an api when you are not going to use it).

What is happening is:

• all suggestions run and the suggestions array is filled with suggestions
• then as we are on the histogram case, it tries to create a histogram but it fails (correctly in this case)

https://github.com/elastic/kibana/blob/main/src/plugins/unified_histogram/public/services/lens_vis_service.ts#L260

so now it goes to this if https://github.com/elastic/kibana/blob/main/src/plugins/unified_histogram/public/services/lens_vis_service.ts#L316

and adds the bad suggestion (which is a DSL one)

You should change the logic.

The histogram mode should be calculated first. In case it returns nothing, then we dont show the histogram

In case of stats mode then we ask for all suggestions. The suggestions api SHOULD not run for DSL mode

[jughosta]

@stratoula What do you mean by DSL mode?

[stratoula]

The dataview mode, in the dataview mode we should not request for suggestions. We have the state hardcoded and we pass it to Lens embeddable

[stratoula]

My suggestion is

• For DSL (dataview):
  • use the hardcoded state for the histogram
• for ES|QL
  • if there are no transformational commands (and the other checks we are doing there) try to use the suggestions api to return the histogram suggestion. If the api returns nothing there is a good reason for this, do not show nothing
  • If there are transformational commands run the suggestions api and display the first suggestion from the suggestions api

We should not mix the transformational commands mode with the non-transformational one otherwise we are creating bugs like this one. I recently fixed another one related #195863

[jughosta]

@stratoula Thanks for expanding on it!

For DSL (dataview):
use the hardcoded state for the histogram

The Lens suggestion API is not called for the data view mode

kibana/src/plugins/unified_histogram/public/services/lens_vis_service.ts

Line 636 in ccbcab9

: [];

What makes you think that it creates the reported issue?

for ES|QL
if there are no transformational commands (and the other checks we are doing there) try to use the suggestions api to return the histogram suggestion. If the api returns nothing there is a good reason for this, do not show nothing

Yes, in this case the current implementation would call the suggestions API twice (without histogram-related params and then with them). Should we now make only the call for suggestions with histogram-related params? What if it returns nothing? Sorry, I don't understand " If the api returns nothing there is a good reason for this, do not show nothing".

If there are transformational commands run the suggestions api and display the first suggestion from the suggestions api

That should be already the case as getHistogramSuggestionForESQL exits earlier with undefined for transformational commands.

[stratoula]

Should we now make only the call for suggestions with histogram-related params? What if it returns nothing?

Yes this was the initial's implementation logic. If there are transformational commands then all good. You run the suggestions api, I think this works ok. The problem is on the histogram mode. Histogram mode === Non transformational commands (no stats, no keep ...). In that case and if there is a date field it should check for https://github.com/elastic/kibana/blob/main/src/plugins/unified_histogram/public/services/lens_vis_service.ts#L485 I assume that one of this checks do not pass so it returns undefined. (is it timebased for example? It doesnt seem so) As it should, it will not return a histogram suggestion. No histogram suggestion means no chart. You should not return (nor run) the suggestions api and get other suggestions. This was never a requirement, I consider it a bug. I hope it is clear now.

The Lens suggestion API is not called for the data view mode

I don't say it creates the reported issue. The reported issue is being created by the problem I explain above (run the suggestions for the histogram twice). I just want to highlight that in case of DSL we should not run the suggestions api. If this is the case already then all good. 👍 But still the suggestion that it suggests is bad as it behaves as it is on the transformational mode while it should not (in that case and only we send the table to the Lens embeddable).

[jughosta]

@stratoula

While I agree that we could have 1 lens suggestion API call for non-transformational ES|QL query and time-based data instead of current 2 calls, it's not the root issue for the reported case.

In fact, this data set is not time-based, so UnifiedHistogram will call Lens suggestion API only once. The API call itself answers very fast with the following 3 suggestions. Then UnifiedHistogram picks the first suggestion and passes it to Lens for rendering. And the rendering of this suggestion (which was enriched with table prop for layers) is what takes the most time. Can we ask the Visualizations team to debug the rendering and verify the list of returned suggestions?

[
  {
    "title": "_all.primaries.bulk.avg_size_in_bytes & _all.primaries.bulk.avg_time_in_millis & _all.primaries.bulk.total_operations of _all.primaries.bulk.avg_time.keyword",
    "score": 1,
    "hide": true,
    "incomplete": false,
    "visualizationId": "lnsXY",
    "visualizationState": {
      "legend": {
        "isVisible": true,
        "position": "right"
      },
      "valueLabels": "hide",
      "fittingFunction": "Linear",
      "axisTitlesVisibilitySettings": {
        "x": true,
        "yLeft": true,
        "yRight": true
      },
      "tickLabelsVisibilitySettings": {
        "x": true,
        "yLeft": true,
        "yRight": true
      },
      "labelsOrientation": {
        "x": 0,
        "yLeft": 0,
        "yRight": 0
      },
      "gridlinesVisibilitySettings": {
        "x": true,
        "yLeft": true,
        "yRight": true
      },
      "preferredSeriesType": "bar_stacked",
      "layers": [
        {
          "layerId": "4b8a4616-ad4f-443e-8c9c-4825c2160591",
          "seriesType": "bar_stacked",
          "xAccessor": "_all.primaries.bulk.avg_time.keyword",
          "splitAccessor": "_all.primaries.bulk.avg_time",
          "accessors": [
            "_all.primaries.bulk.avg_size_in_bytes",
            "_all.primaries.bulk.avg_time_in_millis",
            "_all.primaries.bulk.total_operations"
          ],
          "layerType": "data",
          "colorMapping": {
            "assignments": [],
            "specialAssignments": [
              {
                "rule": {
                  "type": "other"
                },
                "color": {
                  "type": "loop"
                },
                "touched": false
              }
            ],
            "paletteId": "eui_amsterdam_color_blind",
            "colorMode": {
              "type": "categorical"
            }
          }
        }
      ]
    },
    "keptLayerIds": [
      "4b8a4616-ad4f-443e-8c9c-4825c2160591"
    ],
    "datasourceState": {
      "layers": {
        "4b8a4616-ad4f-443e-8c9c-4825c2160591": {
          "index": "b419020427934cdebe4007469063cea902db3cecc4cb9d413468cde97780b5a4",
          "query": {
            "esql": "FROM indices-stats | LIMIT 10"
          },
          "columns": [
            {
              "columnId": "_all.primaries.bulk.avg_size_in_bytes",
              "fieldName": "_all.primaries.bulk.avg_size_in_bytes",
              "meta": {
                "type": "number",
                "esType": "long"
              },
              "inMetricDimension": true
            },
            {
              "columnId": "_all.primaries.bulk.avg_time",
              "fieldName": "_all.primaries.bulk.avg_time",
              "meta": {
                "type": "string",
                "esType": "text"
              },
              "inMetricDimension": true
            },
            {
              "columnId": "_all.primaries.bulk.avg_time.keyword",
              "fieldName": "_all.primaries.bulk.avg_time.keyword",
              "meta": {
                "type": "string",
                "esType": "keyword"
              },
              "inMetricDimension": true
            },
            {
              "columnId": "_all.primaries.bulk.avg_time_in_millis",
              "fieldName": "_all.primaries.bulk.avg_time_in_millis",
              "meta": {
                "type": "number",
                "esType": "long"
              },
              "inMetricDimension": true
            },
            {
              "columnId": "_all.primaries.bulk.total_operations",
              "fieldName": "_all.primaries.bulk.total_operations",
              "meta": {
                "type": "number",
                "esType": "long"
              },
              "inMetricDimension": true
            }
          ]
        }
      },
      "indexPatternRefs": [
        {
          "id": "b419020427934cdebe4007469063cea902db3cecc4cb9d413468cde97780b5a4",
          "title": "indices-stats"
        }
      ]
    },
    "datasourceId": "textBased",
    "columns": 5,
    "changeType": "initial"
  },
  {
    "title": "_all.primaries.bulk.avg_size_in_bytes",
    "score": 0.51,
    "hide": false,
    "visualizationId": "lnsMetric",
    "visualizationState": {
      "layerId": "4b8a4616-ad4f-443e-8c9c-4825c2160591",
      "layerType": "data",
      "metricAccessor": "_all.primaries.bulk.avg_size_in_bytes"
    },
    "keptLayerIds": [
      "4b8a4616-ad4f-443e-8c9c-4825c2160591"
    ],
    "datasourceState": {
      "layers": {
        "4b8a4616-ad4f-443e-8c9c-4825c2160591": {
          "index": "b419020427934cdebe4007469063cea902db3cecc4cb9d413468cde97780b5a4",
          "query": {
            "esql": "FROM indices-stats | LIMIT 10"
          },
          "columns": [
            {
              "columnId": "_all.primaries.bulk.avg_size_in_bytes",
              "fieldName": "_all.primaries.bulk.avg_size_in_bytes",
              "meta": {
                "type": "number",
                "esType": "long"
              },
              "inMetricDimension": true
            }
          ]
        }
      },
      "indexPatternRefs": [
        {
          "id": "b419020427934cdebe4007469063cea902db3cecc4cb9d413468cde97780b5a4",
          "title": "indices-stats"
        }
      ]
    },
    "datasourceId": "textBased",
    "columns": 1,
    "changeType": "reduced"
  },
  {
    "title": "Tag cloud",
    "score": 0.4,
    "visualizationId": "lnsTagcloud",
    "visualizationState": {
      "layerId": "4b8a4616-ad4f-443e-8c9c-4825c2160591",
      "tagAccessor": "_all.primaries.bulk.avg_time",
      "valueAccessor": "_all.primaries.bulk.avg_size_in_bytes",
      "maxFontSize": 72,
      "minFontSize": 18,
      "orientation": "single",
      "showLabel": true,
      "colorMapping": {
        "assignments": [],
        "specialAssignments": [
          {
            "rule": {
              "type": "other"
            },
            "color": {
              "type": "loop"
            },
            "touched": false
          }
        ],
        "paletteId": "eui_amsterdam_color_blind",
        "colorMode": {
          "type": "categorical"
        }
      }
    },
    "keptLayerIds": [
      "4b8a4616-ad4f-443e-8c9c-4825c2160591"
    ],
    "datasourceState": {
      "layers": {
        "4b8a4616-ad4f-443e-8c9c-4825c2160591": {
          "index": "b419020427934cdebe4007469063cea902db3cecc4cb9d413468cde97780b5a4",
          "query": {
            "esql": "FROM indices-stats | LIMIT 10"
          },
          "columns": [
            {
              "columnId": "_all.primaries.bulk.avg_size_in_bytes",
              "fieldName": "_all.primaries.bulk.avg_size_in_bytes",
              "meta": {
                "type": "number",
                "esType": "long"
              },
              "inMetricDimension": true
            },
            {
              "columnId": "_all.primaries.bulk.avg_time",
              "fieldName": "_all.primaries.bulk.avg_time",
              "meta": {
                "type": "string",
                "esType": "text"
              },
              "inMetricDimension": true
            }
          ]
        }
      },
      "indexPatternRefs": [
        {
          "id": "b419020427934cdebe4007469063cea902db3cecc4cb9d413468cde97780b5a4",
          "title": "indices-stats"
        }
      ]
    },
    "datasourceId": "textBased",
    "columns": 2,
    "changeType": "reduced"
  }
]

[stratoula]

@jughosta we dont want to suggest anything different in case is not time based for non transformational commands mode. The logic is can I render a histogram? No? Then I dont want to render a chart

In fact, this data set is not time-based, so UnifiedHistogram will call Lens suggestion API only once.

This is the problem exactly, we don't want to suggest anything else here. This was always the requirement as I describe above.

[stratoula]

Ad to explain further. The table props was introduced by Peter for Discover transformational commands mode. It should not be passed for non transformational commands. We did this because on stats mode we dont want to run the api twice. It was never designed to work for histogram mode. (if anyone wants to pass the table in Lens should ensure that only the columns / rows that take part in the viz are passed, something not happening here)

The lens api works exactly as designed.

[kertal]

@jughosta we dont want to suggest anything different in case is not time based for non transformational commands mode. The logic is can I render a histogram? No? Then I dont want to render a chart

So I guess in this case, not rendering a chart, it should be caught higher up in the rendering tree, e.g. here we don't check for the case we are discussing in the function: checkChartAvailability

kibana/src/plugins/unified_histogram/public/chart/check_chart_availability.ts

Line 13 in b628770

export function checkChartAvailability({

kibana/src/plugins/unified_histogram/public/layout/layout.tsx

Line 312 in dfe00f2

const isChartAvailable = checkChartAvailability({ chart, dataView, isPlainRecord });

Modified it quickly in my exploration PR to take this into consideration

export function checkChartAvailability({
  chart,
  dataView,
  isPlainRecord,
  query,
}: {
  chart?: UnifiedHistogramChartContext;
  dataView: DataView;
  isPlainRecord?: boolean;
  query: Query | AggregateQuery;
}): boolean {
  if (query && isOfAggregateQueryType(query)) {
    const isOnHistogramMode = shouldDisplayHistogram(query);
    if (!isOnHistogramMode || !dataView.isTimeBased()) {
      return false;
    }
  }

  return Boolean(
    chart &&
      dataView.id &&
      dataView.type !== DataViewType.ROLLUP &&
      (isPlainRecord || (!isPlainRecord && dataView.isTimeBased()))
  );
}

Performance issue is gone, however the result is a bit ... white ... but I bet this is fixable

You can check here:
Preview
Code

[jughosta]

@stratoula Got it, thanks!

Going to work on a new implementation for the lens_vis_service.

[stratoula]

Thanx Jul! Let me know if I can help anywhere