Cross cluster failover #3310
Cross cluster failover #3310
Conversation
- Created new CircuitBreakerCommandExecutor to leverage new retry and circuit breaker capability for failover - Created new MultiClusterJedislientConfig to encapsulate resilience4j configurations - Created new MultiClusterPooledConnectionProvider to encapsulate multi-cluster management and operational capabilities - Created new JedisValidationException - Added new constructor to UnifiedJedis - Added resilience4j to pom.xml
- Changed CircuitBreakerCommandExecutor to be more thread safe by passing cluster by reference instead of multiple lookups - Exposed MultiClusterPooledConnectionProvider.Cluster as public so it can be accessed and passed by reference within CircuitBreakerCommandExecutor - Made some javadocs updates for easier readability - Removed debug from happy path so it has parity with other executors. It would likely be too busy on the logs in a production system anyway
- Moved log responsibility into provider for consistency - Added logging for manual failback/failover with consistent wording to the automated failover - Provided a better log for when the prioritized list is exhausted
- Added more safety for orchestration within mutation-operations on the activeMultiClusterIndex to avoid edge cases. In practice this will never likely come up but better to be extra careful as to avoid a deadlock or inaccurate transitions
|
@uglide I may take some time (days) to do proper review. BTW, are we going to do this without tests? @yangbodong22011 @dengliming @mina-asham Please participate if you could. Thanks! |
Codecov ReportPatch coverage:
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. Additional details and impacted files@@ Coverage Diff @@
## master #3310 +/- ##
============================================
- Coverage 68.09% 68.04% -0.06%
- Complexity 4498 4541 +43
============================================
Files 271 275 +4
Lines 14352 14597 +245
Branches 960 978 +18
============================================
+ Hits 9773 9932 +159
- Misses 4180 4252 +72
- Partials 399 413 +14
☔ View full report in Codecov by Sentry. |
|
Is there an issue discussion that already exists before your PR that gave me the background for this PR. |
@yangbodong22011 Sadly, no. I'm writing from my own: This PR is to communicate with multiple Redis servers which are set in multiple geo-locations and are eventually consistent. The aim is to choose the fastest and most reliable server for the (java) application. |
…lable - Moved increment below a validation so subsequent calls to lookup the cluster connection do not throw a nullpointer exception - Replaced custom connection close logic with try-with-resources statements
…possible - Added logic to fallback method that handles subsequent calls after all failover attempts have been exhausted and only a manual failback can resume operations - Added new flag to indicate that all attempts to failover have been exhausted - Changed comments to clarify that an endpoint can belong to a cluster but also a database so it is more OSS friendly - Added logic to the manual failback method to allow an existing cluster to reattempt to connect to its current cluster/database in case its the only option that became available
- Users can now configure their custom logic to persist the activeMultiClusterIndex or custom logging after a successful cluster failover via a functional interface
|
@aterleto @uglide Please resolve the conflicts with the master branch. I have created aterleto#2 for your convenience. |
Co-authored-by: Kyle Banker <banker@users.noreply.github.com>
Co-authored-by: Kyle Banker <banker@users.noreply.github.com>
Add failover docs
…er-failover-2 Conflicts: README.md pom.xml
src/main/java/redis/clients/jedis/executors/CircuitBreakerCommandExecutor.java
Outdated
Show resolved
Hide resolved
src/test/java/redis/clients/jedis/providers/MultiClusterPooledConnectionProviderTest.java
Outdated
Show resolved
Hide resolved
src/main/java/redis/clients/jedis/providers/MultiClusterPooledConnectionProvider.java
Outdated
Show resolved
Hide resolved
src/main/java/redis/clients/jedis/providers/MultiClusterPooledConnectionProvider.java
Show resolved
Hide resolved
Simplify the README's failover docs
src/main/java/redis/clients/jedis/providers/MultiClusterPooledConnectionProvider.java
Show resolved
Hide resolved
src/main/java/redis/clients/jedis/MultiClusterJedisClientConfig.java
Outdated
Show resolved
Hide resolved
src/main/java/redis/clients/jedis/MultiClusterJedisClientConfig.java
Outdated
Show resolved
Hide resolved
Completed as requested |
No description provided.