[Search Sessions] cleanup error when navigating away from a loading dashboard without saving the session

[flash1293]

Kibana version: 7.14

Describe the bug: When navigating away from a dashboard which is currently loading without saving the session first, the document in .async_search gets deleted, but the search session saved object is still around - it will enter error state after a while because the current state can't be polled, then get cleaned up. While doing the cleanup, Kibana attemps to delete the async search which will create an error in the Kibana log

Steps to reproduce:

1. Go to long loading dashboard (e.g. by using the shard delay agg)
2. Navigate away before loading finished
3. Wait for a few mins
4. Error in the server logs like this:

server    log   [16:31:52.721] [error][dataEnhanced][data_enhanced][plugins] search_sessions_cleanup Error while deleting async_search FkNHNVRkQWotVGlTenNoLTItZFZZdUEcTk5ub3g1elZTQkdPZUd5QW8xeHJxZzo5MzU4OA==: resource_not_found_exception: [resource_not_found_exception] Reason: FkNHNVRkQWotVGlTenNoLTItZFZZdUEcTk5ub3g1elZTQkdPZUd5QW8xeHJxZzo5MzU4OA==

Expected behavior:
No error

Any additional context:
As mentioned above, the async search managed by Elasticsearch and the session saved object managed by Kibana get out of sync. In this scenario the saved object should be deleted as well.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[flash1293]

Same thing happens when a session gets "stopped" - the async searches are cleaned up, but the session object sticks around.

[Dosant]

Related (possible duplicate): #98698

[Dosant]

I tried to reproduce, while I see the underlying problem, I wasn't able to see a log in kibana server.
Looks like this log is already muted:

kibana/x-pack/plugins/data_enhanced/server/search/session/check_non_persiseted_sessions.ts

Lines 90 to 94 in 32e6b1e

	if (e.message !== 'resource_not_found_exception') {
	logger.error(
	`${SEARCH_SESSIONS_CLEANUP_TASK_TYPE} Error while deleting async_search ${searchInfo.id}: ${e.message}`
	);
	}

It also should be muted in 7.14, but then I wonder @flash1293, what version have you tested?

Regarding the issue:
I think longer-term we should find a way to avoid getting in an out-of-sync state. This could possibly be addressed with some variation of #99658, which has some unsolved problems with dashboard<->lens session sharing atm.

But for now, I suggest a small improvement, where we stop trying to delete a search if we know it has already errored out: #105434

[Dosant]

We made an improvement to this by #105434. Also no error log should happen anymore.
Created longer-term ticket where we need to create some structure around canceling searches and changing session state: #105441