[CCR] Refactore auto follow coordinator #35895
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* AutofollowCoordinator now fetches cluster states per remote cluster. * AutofollowCoordinator makes use of the `wait_for_metadata_version` feature in cluster state API. * Removed the poll_interval setting, because it is no longer needed. Waiting is done via cluster state api for changes in remote clusters and auto follow pattern changes in the local cluster are immediately processed. * Improved auto follow stats to keep track of last time the remote clusters were checked for changes per remote cluster. Relates to elastic#33007
martijnvg
added
>enhancement
v7.0.0
:Distributed Indexing/CCR
|
Pinging @elastic/es-distributed |
martijnvg
commented
Nov 26, 2018
| @@ -33,6 +33,9 @@ | |||
| private static final ParseField RECENT_AUTO_FOLLOW_ERRORS = new ParseField("recent_auto_follow_errors"); | |||
| private static final ParseField LEADER_INDEX = new ParseField("leader_index"); | |||
| private static final ParseField AUTO_FOLLOW_EXCEPTION = new ParseField("auto_follow_exception"); | |||
| private static final ParseField TRACKING_REMOTE_CLUSTERS = new ParseField("tracking_remote_clusters"); | |||
| private static final ParseField CLUSTER_NAME = new ParseField("cluster_name"); | |||
| private static final ParseField TIME_SINCE_LAST_AUTO_FOLLOW_STARTED_MILLIS = new ParseField("time_since_last_auto_follow_started_millis"); | |||
There was a problem hiding this comment.
Maybe time_since_last_remote_cluster_state_api_call is a better name here.
|
run the gradle build tests 2 |
jasontedor
requested changes
Nov 29, 2018
There was a problem hiding this comment.
Heya @martijnvg I think this PR can be broken down into smaller PRs. I would suggest the following changes:
- remove the poll_interval setting, hard coded the poll_interval to the current default (this will show up a separate change in the change log, that's a good thing)
- add the use of the wait_for_metdata_version parameter on the cluster state API, replacing the hard coded poll_interval
- add the last requested time to the stats API
|
Closing this PR. The changes in this PR will be added in several other PRs. |
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Nov 29, 2018
…te cluster and replaced poll interval setting with a hardcoded poll interval. The hard coded interval will be removed in a follow up change to make use of cluster state API's wait_for_metatdata_version. Originates from elastic#35895 Relates to elastic#33007
martijnvg
added a commit
that referenced
this pull request
Dec 5, 2018
…ster (#36031) and replaced poll interval setting with a hardcoded poll interval. The hard coded interval will be removed in a follow up change to make use of cluster state API's wait_for_metatdata_version. Before the auto following was bootstrapped from thread pool scheduler, but now auto followers for new remote clusters are bootstrapped when a new cluster state is published. Originates from #35895 Relates to #33007
martijnvg
added a commit
that referenced
this pull request
Dec 5, 2018
…ster (#36031) and replaced poll interval setting with a hardcoded poll interval. The hard coded interval will be removed in a follow up change to make use of cluster state API's wait_for_metatdata_version. Before the auto following was bootstrapped from thread pool scheduler, but now auto followers for new remote clusters are bootstrapped when a new cluster state is published. Originates from #35895 Relates to #33007
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Dec 5, 2018
Changed AutofollowCoordinator makes use of the wait_for_metadata_version feature in cluster state API and removed hard coded poll interval. Originates from elastic#35895 Relates to elastic#33007
martijnvg
added a commit
that referenced
this pull request
Dec 12, 2018
martijnvg
added a commit
that referenced
this pull request
Dec 12, 2018
martijnvg
added a commit
to martijnvg/elasticsearch
that referenced
this pull request
Dec 12, 2018
For each remote cluster the auto follow coordinator, starts an auto follower that checks the remote cluster state and determines whether an index needs to be auto followed. The time since last auto follow is reported per remote cluster and gives insight whether the auto follow process is alive. Relates to elastic#33007 Originates from elastic#35895
martijnvg
added a commit
that referenced
this pull request
Dec 17, 2018
) For each remote cluster the auto follow coordinator, starts an auto follower that checks the remote cluster state and determines whether an index needs to be auto followed. The time since last auto follow is reported per remote cluster and gives insight whether the auto follow process is alive. Relates to #33007 Originates from #35895
martijnvg
added a commit
that referenced
this pull request
Dec 17, 2018
) For each remote cluster the auto follow coordinator, starts an auto follower that checks the remote cluster state and determines whether an index needs to be auto followed. The time since last auto follow is reported per remote cluster and gives insight whether the auto follow process is alive. Relates to #33007 Originates from #35895
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
wait_for_metadata_versionfeature in cluster state API.
Waiting is done via cluster state api for changes in remote clusters and
auto follow pattern changes in the local cluster are immediately processed.
were checked for changes per remote cluster.
Relates to #33007