[data.search] Cross-cluster search error handling

[lukasolson]

Related: #94965

Currently, it is possible with different configurations of CCS to use the search service to query ES using features that are not supported in a remote cluster on a different version.

Elasticsearch is currently working on a compatibility layer for CCS that provides more useful error messages when features aren't available in remote clusters. We need to make sure these error messages are surfaced in our search services, and that we provide helpers to handle these types of errors for consumers of search APIs.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[stacey-gammon]

ES merged elastic/elasticsearch#71674 so I think this should be ready to be investigated.

[stacey-gammon]

I think the Kibana App team (cc @timroes ) is going to be blocked on this issue for verifying #100876. With that in mind, is it possible to get this merged with some buffer room to 7.14 FF? Say, at minimum a couple weeks prior?

[lukasolson]

Yeah, I can start taking a look at this either tomorrow or early next week.

[stacey-gammon]

@lukasolson - do you have any update on this?

[lukasolson]

@stacey-gammon We are currently exposing the error messages from ES in the rawResponse portion returned by search services. Correct me if I’m wrong, but I believe elastic/elasticsearch#70784 is needed before we can properly return errors in all CCS situations (like Discover not showing all properties in _source). Does that sound right?

[stacey-gammon]

Could you leverage this test module to test the errors and warnings? Sounds like error messages are good to go for consumers of search services to handle, but the warnings in the header are probably not.

@timroes - since the error messages are returned inside rawResponse, using the test module linked above, sounds like the Discover team can work on ensuring those errors are shown to the user (if they aren't already).

Then we'll just need to figure out how to return the warnings from the header. Perhaps an update to the SearchStrategy return type.

[timroes]

@stacey-gammon Could you maybe clarify which exact errors we should test. I couldn't find (neither in the linked parent issue), the exact errors that could happen anyhow that we should test the UIs for? And for me it's not clear what exact errors CCS would reveal that need to be tested.

[stacey-gammon]

I don't think we need to test for a specific error situation, just that any error or warning returned from the search API is caught and shown to the user. This is why I think it will be easiest to use the test module Jim merged to simulate an error and a warning failure and see what happens. It may be more difficult than I think though because you need to hack the search request to include that error_query part.

I don't think we can actually test the actual CCS silent data loss errors because I don't think ES is actually returning them yet. @jimczi - does that sound accurate?

I need to adjust the parent issue. When I wrote it, I thought ES was already reporting errors when a consumer tried to use the new fields parameter in a search request when a CCS environment was detected.

[stacey-gammon]

Synced with Tim and Jim this am a bit. Shard errors are already apparently handled generically by data search services. I believe the only thing left is to ensure warnings in the response header are likewise handled generically and shown to the user.