Error while deleting async_search resource_not_found_exception

[LeeDr]

Kibana version: 7.13.0-SNAPSHOT

Elasticsearch version: 7.13.0-SNAPSHOT

Server OS version: Ubuntu 16.04

Browser version: Chrome

Browser OS version: Windows 10

Original install method (e.g. download page, yum, from source, etc.): tar.gz file

Describe the bug: I'm seeing errors in the Kibana log like;

{"type":"log","@timestamp":"2021-04-28T22:33:27+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":3618,
"message":"Error while deleting async_search FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA==: resource_not_found_exception"}

Steps to reproduce:

1. I installed Elasticsearch and Kibana 7.13.0-SNAPSHOT builds Security is not enabled
2. ran a set of tests
3. checked the log for errors

Expected behavior: no errors in the log

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

root@packer-virtualbox-iso-1594564190:~/kibana/logs# grep FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA kibana.stdout | sed 's|$|\n|g'
{"type":"log","@timestamp":"2021-04-28T22:28:21+00:00","tags":["debug","plugins","dataEnhanced","data_enhanced"],"pid":3618,"message":"trackId | 2adb5dcb-1132-45ce-9aa3-199e139db743 | FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA=="}

{"type":"log","@timestamp":"2021-04-28T22:28:22+00:00","tags":["debug","plugins","dataEnhanced","data_enhanced"],"pid":3618,"message":"cancel FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA=="}

{"type":"log","@timestamp":"2021-04-28T22:28:22+00:00","tags":["debug","elasticsearch","query","data"],"pid":3618,"message":"200\nDELETE /_async_search/FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA%3D%3D"}

{"type":"log","@timestamp":"2021-04-28T22:28:22+00:00","tags":["debug","http","server","response"],"pid":3618,"client":{"ip":"10.0.2.2"},"http":{"request":{"method":"DELETE","mime_type":"application/json","referrer":"http://localhost:5601/app/dashboards","headers":{"host":"localhost:5601","connection":"keep-alive","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"","sec-ch-ua-mobile":"?0","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36","kbn-version":"7.13.0-SNAPSHOT","content-type":"application/json","accept":"*/*","origin":"http://localhost:5601","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"http://localhost:5601/app/dashboards","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"}},"response":{"body":{},"status_code":200,"headers":{"x-content-type-options":"nosniff","referrer-policy":"no-referrer-when-downgrade","kbn-name":"packer-virtualbox-iso-1594564190","kbn-license-sig":"1b6597e3f7f9d5e4e5a4ac27816bfe67cab4365a5d275754bf71aa9c19f02383","cache-control":"private, no-cache, no-store, must-revalidate","content-length":0},"responseTime":36}},"url":{"path":"/internal/search/ese/FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA==","query":""},"user_agent":{"original":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"},"message":"DELETE /internal/search/ese/FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA== 200 36ms"}

{"type":"log","@timestamp":"2021-04-28T22:28:23+00:00","tags":["debug","elasticsearch","query","data"],"pid":3618,"message":"201\nPUT /.kibana_7.13.0/_create/search-session%3A2adb5dcb-1132-45ce-9aa3-199e139db743?refresh=wait_for&require_alias=true\n{\"search-session\":{\"sessionId\":\"2adb5dcb-1132-45ce-9aa3-199e139db743\",\"status\":\"in_progress\",\"expires\":\"2021-05-05T22:28:22.691Z\",\"created\":\"2021-04-28T22:28:22.691Z\",\"touched\":\"2021-04-28T22:28:22.691Z\",\"idMapping\":{\"cff2e1bbc3234310e7309cb15d5cc6e0ade0dfbd0456bcc6078e77d93b52d4de\":{\"id\":\"FjhXbDNiVFpDVHVxZmZvSzVJb25UWXcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzMyMw==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"bf80a3a82525459a094f29dcc2ba1c58b2b4c709957b4cee2869362fc7fd6250\":{\"id\":\"FkNQcm1BckhQUm9TY0tmcFVZUUdyelEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzMzOA==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"2dd3801ccb1c5e9022d132e545d3874b44f626b5fc1fcd7b4ed9a5b5dd968ca5\":{\"id\":\"FkQtSTBNcS1IU2xhNVlmZGgwcGNkNlEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzMyOQ==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"3fa33e5cc6b81aa23051ce446a507d8863925ea69dc7bfd6a1a0e89d903a7c89\":{\"id\":\"FkNXRm1jZW05UXJXbmtkOFJwRXpYTWcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzMzNQ==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"ce89b4e90e98c729e5b352d3bd0ce9967054fbbacd1bc0069fd7cbd1e40941a3\":{\"id\":\"Flp0MWw3YzJYVDI2Qld2cWZrLUxROGcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzM1Mg==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"a7389b9f4dc56d070544d064dbe8271c24ae9a4b540b7b337223ca88f28ee87b\":{\"id\":\"FnUxZmJ0dEVCUXFHLW41S25SNnZZZHcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQwOA==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"60f9a3752d7a663de14f1505c631cb8b003a86e3393d8527fcccfb7ea8d5edd3\":{\"id\":\"FkNCX1Exei1pU3FpOUxhczNSbE5BQmcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQxNA==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"9309583ca8854035c1e99e57b5b7648ad5b41646c5c80ba907add25b2297c95a\":{\"id\":\"FnA0S3JnWTFHU2lLVlhCVHNkQ3ppbGcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzOQ==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"3b9c311aa4e8a31a1ddb1e8e44218e33ae51c77071a8e71f7be31e8bd493fc72\":{\"id\":\"FnVJUVBFd2pqU3VPWm9JWXplMjVVMXcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ1Nw==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"40a8dc7cba957317cefbd118061fd518dacc19e705000f5a6ecc6b70cbd26d5e\":{\"id\":\"FnJHLVpUelAtUTEtdEhKbzdJNmhiUGcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ1NA==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"84b0abb1e7307c5fbaa499936e371faf3c2e49cd8373798ca677c5cd03cbc702\":{\"id\":\"Fm5tTjRmMUxmUjIyNWZsR0EwNFFud0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ3OA==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"3936dbbf3566870f8e2a3abee95af390b85313089aaa3ac47fca1bebbba5d7bd\":{\"id\":\"FjdSZzV3LWFZVFJTeWxBQy04LVhHQncccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ5OQ==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"e36d68a73d4d8c720d617c5d4458073e04bbf036cb791c778e685bc3d74ef790\":{\"id\":\"FmR5T3RaOERIVGtteURpMWNWdndSWUEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ5Mw==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"266d4557d7c5a2bb7861e168756fd2c37521046edf7f10852b1b922f5cb378a0\":{\"id\":\"FjA4MFB6eEs2UVVtdnhDVkxhQlBxbVEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ4NA==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"69ab635f15a8f77e90f073a48e131675b6d5224261caf34821cdda902c00c997\":{\"id\":\"FnlnaTNxV29PVFp1a1ZFX3l6V0RfbUEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzUzMg==\",\"strategy\":\"ese\",\"status\":\"in_progress\"},\"ce3fa208dcd193f51aff997b400f07b6686b4be043e424133da1f25ae57b235c\":{\"id\":\"FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA==\",\"strategy\":\"ese\",\"status\":\"in_progress\"}},\"persisted\":false},\"type\":\"search-session\",\"references\":[],\"migrationVersion\":{\"search-session\":\"7.13.0\"},\"coreMigrationVersion\":\"7.13.0\",\"updated_at\":\"2021-04-28T22:28:22.691Z\"}"}

{"type":"log","@timestamp":"2021-04-28T22:28:26+00:00","tags":["debug","elasticsearch","query","data"],"pid":3618,"message":"404\nGET /_async_search/status/FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA%3D%3D [resource_not_found_exception]: FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA=="}

{"type":"log","@timestamp":"2021-04-28T22:28:26+00:00","tags":["debug","plugins","dataEnhanced","data_enhanced"],"pid":3618,"message":"search FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA== | status changed to error"}

{"type":"log","@timestamp":"2021-04-28T22:28:26+00:00","tags":["debug","elasticsearch","query","data"],"pid":3618,"message":"200\nPOST /_bulk?refresh=wait_for&_source_includes=originId&require_alias=true\n{\"update\":{\"_id\":\"search-session:2adb5dcb-1132-45ce-9aa3-199e139db743\",\"_index\":\".kibana_7.13.0\",\"if_seq_no\":396,\"if_primary_term\":1}}\n{\"doc\":{\"search-session\":{\"sessionId\":\"2adb5dcb-1132-45ce-9aa3-199e139db743\",\"status\":\"error\",\"expires\":\"2021-05-05T22:28:22.691Z\",\"created\":\"2021-04-28T22:28:22.691Z\",\"touched\":\"2021-04-28T22:28:26.335Z\",\"idMapping\":{\"cff2e1bbc3234310e7309cb15d5cc6e0ade0dfbd0456bcc6078e77d93b52d4de\":{\"id\":\"FjhXbDNiVFpDVHVxZmZvSzVJb25UWXcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzMyMw==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"bf80a3a82525459a094f29dcc2ba1c58b2b4c709957b4cee2869362fc7fd6250\":{\"id\":\"FkNQcm1BckhQUm9TY0tmcFVZUUdyelEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzMzOA==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"2dd3801ccb1c5e9022d132e545d3874b44f626b5fc1fcd7b4ed9a5b5dd968ca5\":{\"id\":\"FkQtSTBNcS1IU2xhNVlmZGgwcGNkNlEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzMyOQ==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"3fa33e5cc6b81aa23051ce446a507d8863925ea69dc7bfd6a1a0e89d903a7c89\":{\"id\":\"FkNXRm1jZW05UXJXbmtkOFJwRXpYTWcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzMzNQ==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"ce89b4e90e98c729e5b352d3bd0ce9967054fbbacd1bc0069fd7cbd1e40941a3\":{\"id\":\"Flp0MWw3YzJYVDI2Qld2cWZrLUxROGcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzM1Mg==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"a7389b9f4dc56d070544d064dbe8271c24ae9a4b540b7b337223ca88f28ee87b\":{\"id\":\"FnUxZmJ0dEVCUXFHLW41S25SNnZZZHcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQwOA==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"60f9a3752d7a663de14f1505c631cb8b003a86e3393d8527fcccfb7ea8d5edd3\":{\"id\":\"FkNCX1Exei1pU3FpOUxhczNSbE5BQmcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQxNA==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"9309583ca8854035c1e99e57b5b7648ad5b41646c5c80ba907add25b2297c95a\":{\"id\":\"FnA0S3JnWTFHU2lLVlhCVHNkQ3ppbGcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzOQ==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"3b9c311aa4e8a31a1ddb1e8e44218e33ae51c77071a8e71f7be31e8bd493fc72\":{\"id\":\"FnVJUVBFd2pqU3VPWm9JWXplMjVVMXcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ1Nw==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"40a8dc7cba957317cefbd118061fd518dacc19e705000f5a6ecc6b70cbd26d5e\":{\"id\":\"FnJHLVpUelAtUTEtdEhKbzdJNmhiUGcccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ1NA==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"84b0abb1e7307c5fbaa499936e371faf3c2e49cd8373798ca677c5cd03cbc702\":{\"id\":\"Fm5tTjRmMUxmUjIyNWZsR0EwNFFud0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ3OA==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"3936dbbf3566870f8e2a3abee95af390b85313089aaa3ac47fca1bebbba5d7bd\":{\"id\":\"FjdSZzV3LWFZVFJTeWxBQy04LVhHQncccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ5OQ==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"e36d68a73d4d8c720d617c5d4458073e04bbf036cb791c778e685bc3d74ef790\":{\"id\":\"FmR5T3RaOERIVGtteURpMWNWdndSWUEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ5Mw==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"266d4557d7c5a2bb7861e168756fd2c37521046edf7f10852b1b922f5cb378a0\":{\"id\":\"FjA4MFB6eEs2UVVtdnhDVkxhQlBxbVEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQ4NA==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"69ab635f15a8f77e90f073a48e131675b6d5224261caf34821cdda902c00c997\":{\"id\":\"FnlnaTNxV29PVFp1a1ZFX3l6V0RfbUEccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzUzMg==\",\"strategy\":\"ese\",\"status\":\"complete\"},\"ce3fa208dcd193f51aff997b400f07b6686b4be043e424133da1f25ae57b235c\":{\"id\":\"FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA==\",\"strategy\":\"ese\",\"status\":\"error\",\"error\":\"Search status threw an error resource_not_found_exception (404) status\"}},\"persisted\":false},\"updated_at\":\"2021-04-28T22:28:26.336Z\",\"references\":[]}}\n"}

{"type":"log","@timestamp":"2021-04-28T22:33:27+00:00","tags":["debug","elasticsearch","query","data"],"pid":3618,"message":"404\nDELETE /_async_search/FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA%3D%3D [resource_not_found_exception]: FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA=="}

{"type":"log","@timestamp":"2021-04-28T22:33:27+00:00","tags":["error","plugins","dataEnhanced","data_enhanced"],"pid":3618,"message":"Error while deleting async_search FjdxU29TUUMyUlU2dk01UVBMU1VPR0EccmVDZEFjRlFTUHVGSzJuNmkwNTBSdzozMzQzNA==: resource_not_found_exception"}

Any additional context:

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[lukasolson]

Okay, I can reliably reproduce this. Here are the steps:

1. Load a (long-running) dashboard
2. Click "Save session" in the search session indicator popover
3. Wait a few seconds so the search session monitoring task kicks in at least once
4. Click "Stop session" and watch the logs

What's happening is, as soon as I click "Stop session", we are cancelling all in-progress search requests (DELETE _async_search/{id}). Then, after the session becomes stale (either notTouchedTimeout or notTouchedInProgressTimeout), the task will delete the saved object and again cancel all in-progress search requests. This time, it will error because they have already been deleted previously.

We should probably just swallow resource_not_found_exception errors, since they are expected, and still log any other errors we encounter trying to cancel the search requests.

[LeeDr]

I probably don't understand all the details here, but it seems when we cancelling all in-progress search requests (DELETE _async_search/{id}) we should also delete the session saved object? That way it doesn't get stale and doesn't try to get deleted again.

[Dosant]

This is fixed with #104309