Remove retention leases when unfollowing #39088
Conversation
This commit attempts to remove the retention leases on the leader shards when unfollowing an index. This is best effort, since the leader might not be available.
jasontedor
added
>enhancement
WIP
v7.0.0
:Distributed Indexing/Distributed
|
Pinging @elastic/es-distributed |
| remoteClient, | ||
| ActionListener.wrap( | ||
| groupListener::onResponse, | ||
| // TODO: should this should exponentially backoff, or should we simply never retry? |
There was a problem hiding this comment.
@martijnvg @dnhatn I opened this PR as WIP for discussion on this point. I am curious to hear your thoughts. Note that failure is a definite possibility since a common use case for the unfollow API will be when the leader is unavailable.
There was a problem hiding this comment.
I'm leaning towards not retrying. However I think that we should add a field in the response indication whether retention lease removal was successful. (retention_lease_removal boolean field?)
There was a problem hiding this comment.
(my motivation for not retrying is, that because it is a common scenario and therefore retrying would be futile to begin with).
There was a problem hiding this comment.
Yes, that's my inclination as well.
@dnhatn Is this okay with you too?
There was a problem hiding this comment.
I am + 1 to "not retry here" because a retention lease will be automatically expired. We need to make sure ILM works well in this scenario.
…follow * elastic/master: (37 commits) Enable test logging for TransformIntegrationTests#testSearchTransform. stronger wording for ilm+rollover in docs (elastic#39159) Mute SingleNodeTests (elastic#39156) AwaitsFix XPackUsageIT#testXPackCcrUsage. Resolve concurrency with watcher trigger service (elastic#39092) Fix median calculation in MedianAbsoluteDeviationAggregatorTests (elastic#38979) [DOCS] Edits the remote clusters documentation (elastic#38996) add version 6.6.2 Revert "Mute failing test 20_mix_typless_typefull (elastic#38781)" (elastic#38912) Rebuild remote connections on profile changes (elastic#37678) Document 'max_size' parameter as shard size for rollover (elastic#38750) Add some missing toString() implementations (elastic#39124) Migrate Streamable to Writeable for cluster block package (elastic#37391) fix RethrottleTests retry (elastic#38978) Disable date parsing test in non english locale (elastic#39052) Remove BCryptTests (elastic#39098) [ML] Stop the ML memory tracker before closing node (elastic#39111) Allow retention lease operations under blocks (elastic#39089) ML refactor DatafeedsConfig(Update) so defaults are not populated in queries or aggs (elastic#38822) Fix retention leases sync on recovery test ...
| retentionLeasesRemoved = in.readBoolean(); | ||
| // noinspection StatementWithEmptyBody | ||
| if (retentionLeasesRemoved) { | ||
|
|
There was a problem hiding this comment.
Did you try to have a symmetric flow with writeTo?
There was a problem hiding this comment.
Yes, but now I have removed this code. 😇
|
@gwbrown @dnhatn @martijnvg I have removed the WIP label, this is ready for review. @gwbrown @martijnvg I decided to not implement the case of not being able to remove the shards the way that you have proposed. I went a different way for two reasons:
Please me know your thoughts and if this will suffice for ILM. |
|
One last note, I think the integration tests I added cover all the branches. |
|
@jasontedor That should work too; the ilm unfollow step can check if the |
This commit attempts to remove the retention leases on the leader shards when unfollowing an index. This is best effort, since the leader might not be available.
This commit attempts to remove the retention leases on the leader shards when unfollowing an index. This is best effort, since the leader might not be available.
This commit attempts to remove the retention leases on the leader shards when unfollowing an index. This is best effort, since the leader might not be available.
This commit attempts to remove the retention leases on the leader shards when unfollowing an index. This is best effort, since the leader might not be available.
Closes #34648
Relates #37165