[ML] Scrolling datafeeds leave open scroll contexts on error

[dimitris-athanasiou]

Datafeeds that use search/scroll should be clearing the scroll contexts they use after they are done. While they do so under normal circumstances, if there is an error thrown during the search request it appears they leave the scroll contexts open. Given we set the timeout to be 30m, when an index pattern is used that matches many shards, this could mean quite a few of scroll contexts left open for a while. In combination with the change of the default value for the cluster setting search.max_open_scroll_context from unlimited to 500, this could lead to the cluster rejecting other search requests while those scroll contexts wait to be cleared.

Steps to reproduce

There are many ways to reproduce this. Here is one.

1. Setup a data index foo-1 with a time field (mapped as date) and some docs.

e.g.

POST foo-1/_doc
{
  "time": "2019-01-01T00:00:00Z",
  "some_value": 42.0      
}

2. Setup another data index foo-2 without that time field.

e.g.

POST foo-2/_doc
{
  "some_value": 42.0      
}

3. Create a simple count job with a datafeed using index pattern foo-*
4. Open job and start datafeed

Observed behaviour

Thee datafeed should have an error notification that the search failed because not all indices have a time field to sort on. This is as expected.

Now do:

GET _nodes/stats

and look for search.open_contexts. The number there should be 0 (assuming nothing else is using the cluster`. However, it is not as the datafeed did not clear the scroll context.

[elasticmachine]

Pinging @elastic/ml-core