Refactor degrade hierarchy with new circuit breaker mechanism and improve strategy #1490

sczyh30 · 2020-05-18T15:12:13Z

Describe what this PR does / why we need it

Refactor legacy degrade hierarchy with new circuit breaker mechanism and improve strategy.

Does this pull request fix one issue?

Resolves #1421, #1032, #951, #154, #308, #56

Describe how you did it

Add CircuitBreaker abstraction (with half-open state) and add circuit breaker state change event observer support. Now Sentinel follows the canonical circuit breaker pattern (with some improvements).
Make statistics of each rule dependent (to support arbitrary statistic interval).
Improve RTT-based circuit breaking strategy (avg RT → slow request ratio).
Add simple "probe" mechanism (aka. half-open).
Refactor mechanism of metric recording and state change handling for circuit breakers: record RT and error when requests have completed (i.e. onExit, based on Refactor the mechanism of recording error (on completed) #1420).

Describe how to verify it

Run the test cases and demo.

Special notes for reviews

This PR contains internal breaking changes (and some behavioral changes for RT-based circuit breaking).

codecov-commenter · 2020-05-19T07:07:57Z

Codecov Report

Merging #1490 into master will increase coverage by 1.11%.
The diff coverage is 75.41%.

@@             Coverage Diff              @@
##             master    #1490      +/-   ##
============================================
+ Coverage     43.96%   45.08%   +1.11%     
- Complexity     1719     1786      +67     
============================================
  Files           376      382       +6     
  Lines         10660    10854     +194     
  Branches       1418     1443      +25     
============================================
+ Hits           4687     4893     +206     
+ Misses         5408     5373      -35     
- Partials        565      588      +23

Impacted Files	Coverage Δ	Complexity Δ
...degrade/circuitbreaker/CircuitBreakerStrategy.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...ntinel/slots/block/degrade/DegradeRuleManager.java	`46.08% <55.88%> (+24.21%)`	`16.00 <15.00> (+7.00)`
...degrade/circuitbreaker/AbstractCircuitBreaker.java	`71.42% <71.42%> (ø)`	`19.00 <19.00> (?)`
.../csp/sentinel/slots/block/degrade/DegradeSlot.java	`83.87% <81.48%> (-16.13%)`	`10.00 <9.00> (+7.00)`	⬇️
.../csp/sentinel/slots/block/degrade/DegradeRule.java	`86.84% <87.50%> (+15.04%)`	`19.00 <7.00> (-5.00)`	⬆️
...ade/circuitbreaker/ResponseTimeCircuitBreaker.java	`88.88% <88.88%> (ø)`	`11.00 <11.00> (?)`
...egrade/circuitbreaker/ExceptionCircuitBreaker.java	`89.47% <89.47%> (ø)`	`10.00 <10.00> (?)`
.../degrade/circuitbreaker/EventObserverRegistry.java	`90.90% <90.90%> (ø)`	`5.00 <5.00> (?)`
...s/block/degrade/circuitbreaker/CircuitBreaker.java	`100.00% <100.00%> (ø)`	`0.00 <0.00> (?)`
...tinel/slots/block/flow/param/ParamFlowChecker.java	`53.02% <0.00%> (-2.69%)`	`28.00% <0.00%> (-1.00%)`
... and 17 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46076b3...4c043ed. Read the comment docs.

cdfive · 2020-05-20T09:40:32Z

...inel-core/src/main/java/com/alibaba/csp/sentinel/slots/block/degrade/DegradeRuleManager.java

            for (DegradeRule rule : list) {
                if (!isValidRule(rule)) {
-                    RecordLog.warn(
-                        "[DegradeRuleManager] Ignoring invalid degrade rule when loading new rules: " + rule);
+                    RecordLog.warn("[DegradeRuleManager] Ignoring invalid rule when loading new rules: " + rule);
                    continue;
                }

                if (StringUtil.isBlank(rule.getLimitApp())) {
                    rule.setLimitApp(RuleConstant.LIMIT_APP_DEFAULT);
                }
-
-                String identity = rule.getResource();
-                Set<DegradeRule> ruleSet = newRuleMap.get(identity);
-                if (ruleSet == null) {
-                    ruleSet = new HashSet<>();
-                    newRuleMap.put(identity, ruleSet);
+                CircuitBreaker cb = getExistingSameCbOrNew(rule);
+                if (cb == null) {
+                    RecordLog.warn("[DegradeRuleManager] Unknown circuit breaking strategy, ignoring: " + rule);
+                    continue;
                }
-                ruleSet.add(rule);


Inside getExistingSameCbOrNew method, the design of reuse the circuit breaker if the rule remains unchanged.is smart. I tried with DegradeRuleManager.loadRules in different places, it works. A small question is that, if using DegradeRuleManager.loadRules(rules) first time, and rules have two same DegradeRule , there will be two CircuitBreaker created, since in the first time, the static circuitBreakers variable is null.

Yes, it can be improved.

cdfive · 2020-05-20T09:43:17Z

sentinel-core/src/main/java/com/alibaba/csp/sentinel/slots/block/degrade/DegradeSlot.java

+            return;
+        }
+
+        if (curEntry.getBlockError() == null) {


This judgement seems can be removed since Line#63.

It's here in case the blockError was modified.

Is it really concurrent? And the time lapse is really tiny here.

CodingSinger · 2020-05-21T01:16:46Z

where is the proposal for this pr?

sczyh30 · 2020-06-05T03:21:03Z

where is the proposal for this pr?

I'll link the proposal here soon.

wavesZh · 2020-06-08T06:38:06Z

...java/com/alibaba/csp/sentinel/slots/block/degrade/circuitbreaker/AbstractCircuitBreaker.java

+    public boolean tryPass() {
+        // Template implementation.
+        if (currentState.get() == State.CLOSED) {
+            return true;
+        }
+        if (currentState.get() == State.OPEN) {
+            // For half-open state we allow a request for trial.
+            return retryTimeoutArrived() && fromOpenToHalfOpen();
+        }
+        return false;
+    }


If thare are two breakers and curState is open, one breaker tryPass and change state to half open, but other breaker tryPass return false, it is right that no request be allowed. but the below requests will not be allowed because two breakers tryPass can't both be true.

@sczyh30 Any idea for the scene？Was I wrong？

I think it's ok for this scenario.

B1(open) & B2(open) -> no request at all
B1(open -> half) & B2(open) -> no request at all, but B1 may transform to closed leaving B2 alone, that is:
B1(closed) & B2(open)

Sorry, i cannot find how to change half-open to close.
Method fromHalfOpenToClose is invoked in DegradeSlot#exist when no BlockingException. But if B1(open -> half) & B2(open), the follow requests check by CircuitBreaker#tryPass must return false then throw DegradeException.

Requests is blocked by tryPass() in abstract parent and the state of breakers are maintained in different implementations when requests complete (post-request).

So they are not collided.

public boolean tryPass() { // Template implementation. if (currentState.get() == State.CLOSED) { return true; } if (currentState.get() == State.OPEN) { // For half-open state we allow a request for trial. return retryTimeoutArrived() && fromOpenToHalfOpen(); } return false; }

我已经自我怀疑了🤨😢 是我理解错了吗？？

```java public boolean tryPass() { // Template implementation. if (currentState.get() == State.CLOSED) { return true; } if (currentState.get() == State.OPEN) { // For half-open state we allow a request for trial. return retryTimeoutArrived() && fromOpenToHalfOpen(); } return false; }

我已经自我怀疑了🤨😢 是我理解错了吗？？

emmm.........仔细想了下，你考虑的问题的确存在，而且貌似还存在其它限流规则的问题，因为这里并没有判断是不是DegradeException，待我下午拿Test测一下

@sczyh30 Hi, Eric

Here're some boundaries which lead to new issues while we should discuss more how to make the recovery logic better:

Multiple degrading rules for same resource(Especially after introducing prefix/postfix matching)

Another rule after degrading rule in half-open state(Though there's no such kind of rule but it's possible in future)

These will lead to the incorrect workflow:

entry: R0(open -> half) -> R1(block)
exit: XXXBlockException -> skip the detection for stalling half state

For straightforward patch i think the onComplete method is intentioned to be call after a successfully transforming into half. We should make the exiting logic more simple and clear.

@jasonjoo2010 @wavesZh Yes, it's a fatal bug indeed and should be resolved immediately. However we need a better temporary workaround for this (for the half-open case). I'll merge this PR first and we may discuss it in a new issue. It's a common problem for all rules that have their own metrics and rely on onComplete.

...java/com/alibaba/csp/sentinel/slots/block/degrade/circuitbreaker/AbstractCircuitBreaker.java

CLAassistant · 2020-06-12T07:20:52Z

All committers have signed the CLA.

...inel-core/src/main/java/com/alibaba/csp/sentinel/slots/block/degrade/DegradeRuleManager.java

jasonjoo2010 · 2020-07-04T02:26:16Z

sentinel-core/src/main/java/com/alibaba/csp/sentinel/slots/block/degrade/DegradeSlot.java

+            return;
+        }
+
+        if (curEntry.getBlockError() == null) {


Is it really concurrent? And the time lapse is really tiny here.

...ibaba/csp/sentinel/slots/block/degrade/circuitbreaker/CircuitBreakerStateChangeObserver.java

...rc/main/java/com/alibaba/csp/sentinel/slots/block/degrade/circuitbreaker/CircuitBreaker.java

...java/com/alibaba/csp/sentinel/slots/block/degrade/circuitbreaker/CircuitBreakerStrategy.java

jasonjoo2010 · 2020-07-05T03:11:47Z

sentinel-core/src/main/java/com/alibaba/csp/sentinel/slots/block/degrade/DegradeRule.java

     *
     * @since 1.7.0
     */
-    private int rtSlowRequestAmount = RuleConstant.DEGRADE_DEFAULT_SLOW_REQUEST_AMOUNT;


Here is a missing property in new implementation. Is that acceptable in compatibility?

It could bring breaking changes (and break the SEMVER spec) indeed. In convention, we should mark it deprecated and remove it until 2.x. But actually it's rarely used and often regarded as a "hidden" attribute (not even appeared in the dashboard)...

Yeah you're right. But i just afraid the users who integrating manually suffering broken compatibility.😂 We can make decision here whether doing it carefully this time. Decide and record it and it would be ok.

IMHO we could drop it (the legacy bad design) and add notes in relevant documents :)

jasonjoo2010 · 2020-07-05T03:18:29Z

sentinel-core/src/main/java/com/alibaba/csp/sentinel/slots/block/degrade/DegradeSlot.java

    @Override
-    public void exit(Context context, ResourceWrapper resourceWrapper, int count, Object... args) {
-        fireExit(context, resourceWrapper, count, args);
+    public void exit(Context context, ResourceWrapper r, int count, Object... args) {


And here's a problem that if something went wrong which may cause RT down to more than 10s (100ms for normal) . Our breaker will cut the traffic after 10s. Maybe we could improve it (or as a new issue and do it in future) that unfinished entries could be recorded as statIntervalMs when slidingCounter is reset.

Yes, it's necessary to improve the scenarios regarding slow in-flight requests. See #1405

Oh they are not the same issue i think.

How to calculate a more reasonable RT is one thing while how to make a more reasonable degradation is quite another thing. The primary reason is that the calculating window is separated from general RT, isn't?

And sure we can improve it later but it's more important than making it more reasonable calculating RT. Maybe another time window is required to make it under tracing.

We could open a new issue to discuss it.

…rove strategy * Add `CircuitBreaker` abstraction (with half-open state) and add circuit breaker state change event observer support. * Improve circuit breaking strategy (avg RT → slow request ratio) and make statistics of each rule dependent (to support arbitrary statistic interval). * Add simple "trial" mechanism (aka. half-open). * Refactor mechanism of metric recording and state change handling for circuit breakers: record RT and error when requests have completed (i.e. `onExit`, based on alibaba#1420). Signed-off-by: Eric Zhao <sczyh16@gmail.com>

Signed-off-by: Eric Zhao <sczyh16@gmail.com>

sczyh30 · 2020-07-28T02:25:42Z

@jasonjoo2010 I've updated the PR, please check.

Note: please use "rebase and merge" for this PR instead of "squash and merge".

jasonjoo2010

It's ok for me now. @wavesZh @cdfive Any further problem?

sczyh30 added area/circuit-breaking Issues or PRs related to circuit breaking kind/feature Category issues or prs related to feature request. kind/refactor Issue related to functional refactoring. labels May 18, 2020

This was linked to issues May 18, 2020

Support half-open state for circuit breaking in Sentinel #154

Closed

Support event listener of circuit breaker state change event #951

Closed

Wrong Circuit Breaker Implementation #1032

Closed

sczyh30 added this to the 1.8.0 milestone May 18, 2020

sczyh30 force-pushed the refactor/circuit-breaker branch from df75e22 to c017b12 Compare May 18, 2020 15:27

sczyh30 added the to-review To review label May 18, 2020

sczyh30 requested review from linlinisme, cdfive, jasonjoo2010 and a team May 18, 2020 15:28

sczyh30 marked this pull request as ready for review May 18, 2020 15:32

cdfive reviewed May 20, 2020

View reviewed changes

sczyh30 requested a review from zhaoyuguang May 22, 2020 07:39

sczyh30 added the size/XXL Indicate a PR that changes 1000+ lines. label Jun 5, 2020

wavesZh reviewed Jun 8, 2020

View reviewed changes

wavesZh reviewed Jun 9, 2020

View reviewed changes

...java/com/alibaba/csp/sentinel/slots/block/degrade/circuitbreaker/AbstractCircuitBreaker.java Outdated Show resolved Hide resolved

sczyh30 linked an issue Jun 15, 2020 that may be closed by this pull request

[RFC] Changing statistic time window in degrade rule dynamically #308

Closed

sczyh30 mentioned this pull request Jun 28, 2020

降级DEGRADE_GRADE_EXCEPTION_RATIO，设置的是每秒异常比例，能否支持任意统计窗口时长 #1569

Closed

sczyh30 linked an issue Jun 29, 2020 that may be closed by this pull request

降级DEGRADE_GRADE_EXCEPTION_RATIO，设置的是每秒异常比例，能否支持任意统计窗口时长 #1569

Closed

jasonjoo2010 requested changes Jul 5, 2020

View reviewed changes

sczyh30 added 2 commits July 28, 2020 10:16

Update test cases for circuit breaking

1743001

Signed-off-by: Eric Zhao <sczyh16@gmail.com>

Update demo for circuit breaking (DegradeRule)

17395c0

Signed-off-by: Eric Zhao <sczyh16@gmail.com>

sczyh30 force-pushed the refactor/circuit-breaker branch from 4c043ed to 17395c0 Compare July 28, 2020 02:17

jasonjoo2010 approved these changes Jul 28, 2020

View reviewed changes

sczyh30 merged commit 19db20f into alibaba:master Jul 30, 2020

sczyh30 removed the to-review To review label Jul 30, 2020

This was referenced Jul 30, 2020

Separate rule action from rule definition #56

Closed

add half-open and fix degrade-bug #553

Closed

[DISCUSSION] Circuit breaker will remain half-open state forever when the request is blocked by upcoming rules #1638

Open

jasonjoo2010 mentioned this pull request Aug 4, 2020

Refactor the flow of request to fix #1638 for circuit breakers #1645

Merged

single-wolf mentioned this pull request Apr 10, 2021

Degrade by RT should consider inflight request #1405

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor degrade hierarchy with new circuit breaker mechanism and improve strategy #1490

Refactor degrade hierarchy with new circuit breaker mechanism and improve strategy #1490

sczyh30 commented May 18, 2020 •

edited

Loading

codecov-commenter commented May 19, 2020 •

edited

Loading

cdfive May 20, 2020

sczyh30 Jun 18, 2020

cdfive May 20, 2020

sczyh30 Jun 18, 2020

jasonjoo2010 Jul 4, 2020

CodingSinger commented May 21, 2020

sczyh30 commented Jun 5, 2020

wavesZh Jun 8, 2020 •

edited

Loading

wavesZh Jul 28, 2020

jasonjoo2010 Jul 28, 2020

wavesZh Jul 29, 2020

jasonjoo2010 Jul 29, 2020

wavesZh Jul 29, 2020

jasonjoo2010 Jul 29, 2020

jasonjoo2010 Jul 29, 2020 •

edited

Loading

sczyh30 Jul 30, 2020

sczyh30 Jul 30, 2020

CLAassistant commented Jun 12, 2020 •

edited

Loading

jasonjoo2010 Jul 4, 2020

jasonjoo2010 Jul 5, 2020

sczyh30 Jul 27, 2020

jasonjoo2010 Jul 27, 2020

sczyh30 Jul 28, 2020

jasonjoo2010 Jul 5, 2020

sczyh30 Jul 27, 2020

jasonjoo2010 Jul 27, 2020

sczyh30 Jul 30, 2020

sczyh30 commented Jul 28, 2020

jasonjoo2010 left a comment

Refactor degrade hierarchy with new circuit breaker mechanism and improve strategy #1490

Refactor degrade hierarchy with new circuit breaker mechanism and improve strategy #1490

Conversation

sczyh30 commented May 18, 2020 • edited Loading

Describe what this PR does / why we need it

Does this pull request fix one issue?

Describe how you did it

Describe how to verify it

Special notes for reviews

codecov-commenter commented May 19, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CodingSinger commented May 21, 2020

sczyh30 commented Jun 5, 2020

wavesZh Jun 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasonjoo2010 Jul 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Jun 12, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sczyh30 commented Jul 28, 2020

jasonjoo2010 left a comment

Choose a reason for hiding this comment

sczyh30 commented May 18, 2020 •

edited

Loading

codecov-commenter commented May 19, 2020 •

edited

Loading

wavesZh Jun 8, 2020 •

edited

Loading

jasonjoo2010 Jul 29, 2020 •

edited

Loading

CLAassistant commented Jun 12, 2020 •

edited

Loading