Point in time fails for deleted indices and fails differently in comparison to scroll

[hendrikmuhs]

Using a point in time reader I can create a consistent state to query for data with multiple searches. The data in the index might change, documents might be updated, added or deleted but pit will return the state of the pit creation.

Investigating #81252 which is about a problem of a user of pit: transform, I found some inconsistencies in the behavior of pit. This is a collection of the issues found. I decided to keep them in 1 issue, however this is a story and the problems are complex and not easy to solve.

pit and deleted indices

However if pit is used together with a set of indices specified by a wildcard, e.g. metricbeat-* the results with respect to indices might be a bit surprising. Added indices are not part of the created pit, however deleted indices result in an index not found exception.

This is no surprise from the technical side, however for a user we document:

Elasticsearch pit (point in time) is a lightweight view into the state of the data as it existed when initiated.

Sub-Issue: It would be good to document the limitation regarding deleted indices.

A user might expect that we softly delete the index, meaning it stays available for the point in time reader until the destruction of the context.

pit returns a 404

Elasticsearch default behavior is to return results even if the result set is incomplete. This is the nature of a search engine, where we care more about recall than about correctness. With allow_partial_search_results you can choose to fail instead or you check for shard failures in the response.

Sub-Issue: In case of a deleted index a pit search fails completely if an index got deleted. This is in contrast to the default elasticsearch behavior.

Note:

• with pit I can still specify allow_partial_search_results, if true a shard failure does not cause a fail, a deleted index does
• a scroll returns results, the missing index causes a shard failure:

        "shard" : -1,
       "index" : null,
       "reason" : {
         "type" : "search_context_missing_exception",
         "reason" : "No search context found for id [110100]"
       }
     }

• the scroll error message does not contain the name of the deleted index
• it's not possible to combine allow_partial_search_results with scroll, it seems to be ignored, however I haven't tried the cluster setting

Therefore scroll and pit behave differently for the same scenario.

pit should allow a filter query

The scenario behind a wildcard search or data stream is to easily query time series data. In #81252 we do not even care about the index that gets deleted, because the data is out of scope. The pit searches use a range query. Therefore the pit context invalidates due to data in an index that we are not going to retrieve.

Sub-Issue: If I could specify a filter query at pit creation, I could reduce the pit context and the search would not stumble upon the deleted index.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[dnhatn]

@hendrikmuhs Thanks for writing a detailed report. We will prioritize addressing these points.

[hendrikmuhs]

Thanks @dnhatn!

One more thing I realized and that might help me handling the high-level issue:

Once the pit invalidates due to a deleted index, it throws a index not found exception. In wonder if that's the right exception type. Index not found should be thrown at pit creation, but when pit invalidates during usage this exception type might not be the right one. As explained above, scroll uses a search context missing exception. It seems to me a better fit.

This distinction is important for me, because I need to differentiate between a broken user request and a pit invalidation.

(Nit picking: At rest level index not found translates to 404, search context missing to 404, too. A 410 (GONE) seems to be a better fit for search context missing?)

[benwtrent]

PIT allows for partial results now. I don't know what else is required here, closing