[Transform] Transform fails with task encountered irrecoverable failure: org.elasticsearch.index.IndexNotFoundException

[hendrikmuhs]

Affected version: 7.15-
Fixed with: 7.16.1

Transform can fail with an IndexNotFoundException if you use transform on a data stream, index pattern or alias that covers multiple indices and you use index lifecycle management to delete old indices. In the case a transform updates documents while an index gets deleted by index lifecycle management the transform might fail.

This error is a regression introduced in 7.15, details below. Transform can be used together with index lifecycle management on source indices.

Mitigation

Until a fix is available you have 3 options:

• stay on an older version or don't upgrade to a version >= 7.15 until a fix is available
• don't use ILM to delete old data or ensure transform isn't running when ILM deletes an index
• watch and restart transform after a failure:

POST _transform/{job_id}/_stop?force=true&wait_for_completion=true
POST _transform/{job_id}/_start

You need force=true for jobs that are in failed state.
wait_for_completion=true waits until the transform has really stopped, otherwise it might not be stopped if you call _start right after - within nanoseconds - in a script.
You can use a _stats call to find out if a transform is in failed state.

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[hendrikmuhs]

Bug

The usage of a point in time reader has been introduced for 7.15 in #74984. This is a performance optimization. The point in time reader is a search context that gets 1st created and than reused. The error happens when an index gets deleted between the point in time reader calls. Transform treats this as permanent error and sets the transform to failed.

Before using a point in time reader transform executed ordinary search requests where the indices got resolved per call. It seems however that an index that gets deleted in between is somehow handled internally.

A point in time reader is meant to be a

lightweight view into the state of the data as it existed when initiated

This works fine if single documents are updated/deleted or added, a new index can be created as well, but not deleting an index.

In the transform use case this is a benign problem, most likely the transform does not care about the the index being deleted, because the deleted index likely does not contain data that transform queries for. For example for date_histogram groupings transform only updates buckets that have changed. The deleted index likely contains data from buckets that are not of interest and therefore the index gets pruned away early (can_match).

[hendrikmuhs]

Solution

If point in time search is used, handle index not found exceptions and retry without a pit. On the next iteration a new pit context gets created.

In case further issues with pit are found, pit search can be disabled.

[hendrikmuhs]

Note: in addition to the issue with pit, transform does some index resolving themselves in 7.16. This suffers from the same underlying issue. Until a better solution has been found, this should be disabled/reverted, too.

I am not quite sure how search handles this case. If an index gets deleted after index names are resolved, but before shard search requests are send, how does it handle this? It seems it is using an IndexEventListener, which removes deleted indexes from search contexts. However it's not clear to me, how this differs between explicit index names and resolved via wildcard index names. The former should create a failure, the later can be ignored.