[Search] Better communicate timeout settings

[Dosant]

This came up from a customer who noticed unexpected long-running async searches in their cluster and tried to mitigate the issue by configuring various documented timeout options, but none seem to help.

These are the documented options that historically worked, but with the introduction of async search by default and search session they don't anymore:

kibana.yml:

https://www.elastic.co/guide/en/kibana/current/settings.html

• elasticsearch.requestTimeout - Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer. Default: 30000. This option doesn't work as expected with async search, as the request run beyond a single request
• elasticsearch.shardTimeout - Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. Default: 30000. This option isn't used in async search code path.

advanced settings:

• search:timeout - Change the maximum timeout for a search session or set it to 0 to disable the timeout and allow queries to run to completion. - This seems to be only a client-side option and it would cancel the request only if the user stays on a page` The description also is outdated since we introduced search sessions (7.12). Seems like we also plan to deprecate it: Deprecate search:timeout advanced setting #94912

search sessions settings

Currently in reality what affects the timeout of most of the searches is search session settings:

https://www.elastic.co/guide/en/kibana/7.x/search-session-settings-kb.html

• xpack.data_enhanced. search.sessions.notTouchedInProgressTimeout - How long a search session can run after a user navigates away without saving a session. The default is 1m.
• xpack.data_enhanced. search.sessions.defaultExpiration - How long search session results are stored before they are deleted. Extending a search session resets the expiration by the same value. The default is 7d.

I think from Kibana user perspective it could seem that they don't need to look into those settings unless they are using search sessions (saving them), but in reality, it is not currently the case.

I think we need to somehow communicate better what timeout options do. Remove search:timeout to avoid further confusion #94912. And maybe somehow restructure the settings to have central important "async search timeout" on a top-level and not inside search sessions 🤷‍♂️

cc @lukasolson @lizozom @elastic-jb

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

Closing this because it's not planned to be resolved in the foreseeable future. It will be tracked in our Icebox and will be re-opened if our priorities change. Feel free to re-open if you think it should be melted sooner.