Scroll API fails to apply Rescore query

[bwsawyer]

Elasticsearch version (bin/elasticsearch --version):
Elasticsearch 6.3.0

Plugins installed: []
None

JVM version (java -version):
java version "1.8.0_171"

OS version (uname -a if on a Unix-like system):
Darwin mbp-bsawy-5018 17.6.0 Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:
Performing a query using the Scroll API does not apply the 'rescore' portion of the query.

Steps to reproduce:

1. Create test index with one document:

PUT index

POST index/_doc
{
  "field": "value"
}

2. Run query with a rescore and see that the score returned for the hit is as expected (eg. _score=101)

GET index/_search
{
  "query": {
    "match_all": {}
  },
  "rescore": {
    "query": {
      "rescore_query": {
        "function_score": {
          "weight": 100
        }
      }
    }
  }
}

3. Run same query but with a scroll and that the score seems to indicate that the rescore was not applied (eg. _score=1)

GET index/_search?scroll=1m
{
  "query": {
    "match_all": {}
  },
  "rescore": {
    "query": {
      "rescore_query": {
        "function_score": {
          "weight": 100
        }
      }
    }
  }
}

This appears to affect all 6.* versions of Elasticsearch but is not present in 5.*

[elasticmachine]

Pinging @elastic/es-search-aggs

[jimczi]

@bwsawyer I marked this issue as a bug because it works on 5.x but I wonder why you'd use a rescorer in a scroll context ? Is there something you cannot achieve with the main query ? A scroll needs to score all documents which defeats the purpose of the rescorer. Maybe we should simply fail or document the fact that scroll and rescore don't play well together ?

[bwsawyer]

In our use case we use the main query as a blocking step to retrieve good candidates for a more computationally intensive rescore function (in a custom plugin). Running the function over all the documents in the index isn't feasible, nor can we accomplish our blocking query as a pre-filter.

Part of our application needs to retrieve all documents that score above a certain threshold after applying that rescore function. Sometimes the number of documents above the threshold is large enough that I assume scrolling would be more efficient than paging, at least as I understand it. I admit I have not done any benchmarking to verify this assumption.

[jimczi]

Scrolling is faster than simple pagination but the issue I have with rescored scroll is that it breaks the _score sort. In your example the main query is a match_all so each page is rescored up to window_size but documents with greater score can be returned in subsequent pages. For this reason I don't think rescored scroll should be possible or at least we should disable sort in this case.

[jimczi]

We've decided to forbid scroll with rescores, it used to work in 5x but should be considered as a bug since it breaks the sort. Though we should fail the creation of the scroll instead of silently ignoring the scorers as we do today which is why I am marking this issue as an adoptme.