Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Graceful shutdown bugs #1254

Merged
merged 6 commits into from
Jun 13, 2021

Conversation

justxuewei
Copy link
Member

@justxuewei justxuewei commented Jun 11, 2021

What this PR does:

  • Cancel calling exporter.Unexport() in registryProtocol.Destory()
  • Close zkListener before closing zkClient
  • Fix timeout logic for graceful shutdown
  • Fix bug that RequestsFinished always be true
  • Fix bug that RejectRequest not works

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@justxuewei justxuewei marked this pull request as draft June 11, 2021 08:00
@codecov-commenter
Copy link

codecov-commenter commented Jun 11, 2021

Codecov Report

Merging #1254 (f1879b4) into 3.0 (968650f) will decrease coverage by 3.77%.
The diff coverage is 50.60%.

Impacted file tree graph

@@            Coverage Diff             @@
##              3.0    #1254      +/-   ##
==========================================
- Coverage   59.53%   55.75%   -3.78%     
==========================================
  Files         259      272      +13     
  Lines       12737    12899     +162     
==========================================
- Hits         7583     7192     -391     
- Misses       4199     4808     +609     
+ Partials      955      899      -56     
Impacted Files Coverage Δ
cluster/cluster_impl/available_cluster_invoker.go 66.66% <ø> (ø)
cluster/cluster_impl/broadcast_cluster_invoker.go 76.47% <ø> (ø)
cluster/cluster_impl/failfast_cluster_invoker.go 66.66% <ø> (ø)
cluster/cluster_impl/mock_cluster.go 0.00% <0.00%> (ø)
cluster/loadbalance/random.go 100.00% <ø> (ø)
...router/v3router/judger/list_string_match_judger.go 0.00% <0.00%> (ø)
...ster/router/v3router/judger/method_match_judger.go 0.00% <0.00%> (ø)
...er/router/v3router/judger/url_label_match_judge.go 0.00% <0.00%> (ø)
common/extension/auth.go 0.00% <ø> (ø)
common/extension/cluster.go 0.00% <ø> (ø)
... and 277 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ef3eff...f1879b4. Read the comment docs.

@justxuewei justxuewei marked this pull request as ready for review June 11, 2021 08:26
Copy link
Contributor

@zhaoyunxing92 zhaoyunxing92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved these changes

@AlexStocks AlexStocks merged commit 7fedc22 into apache:3.0 Jun 13, 2021
@@ -60,6 +60,9 @@ func (gf *gracefulShutdownFilter) Invoke(ctx context.Context, invoker protocol.I
return gf.getRejectHandler().RejectedExecution(invoker.GetURL(), invocation)
}
atomic.AddInt32(&gf.activeCount, 1)
if gf.shutdownConfig != nil && gf.activeCount > 0 {
gf.shutdownConfig.RequestsFinished = false
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

没有锁,会导致 data race。

@@ -163,6 +163,7 @@ func waitForReceivingRequests() {
// ignore this step
return
}
providerConfig.ShutdownConfig.RejectRequest = true
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

graceful shutdown 时拒绝新请求不合适,会导致请求失败。

参考 grpc,先是关闭 listener,拒绝新连接,而不是请求。

@win5do
Copy link

win5do commented Jun 13, 2021

@justxuewei
Copy link
Member Author

https://github.com/xavier-niu/dubbo-go/blob/f1879b4cd72d06bed0e69f4e58d4c2ce3b19fd38/filter/filter_impl/graceful_shutdown_filter.go#L36

init 时拿到的 shutdownConfig 总是 nil。

ShutdownConfig should be not nil if it is configured in server.yml or client.yml.

justxuewei added a commit to justxuewei/dubbo-go that referenced this pull request Jun 13, 2021
justxuewei added a commit to justxuewei/dubbo-go that referenced this pull request Jun 13, 2021
remove unused comments

fix import cycle

append apache license header

fix gracefulShutdownFilter unittest bug

go fmt

fix gracefulShutdownConfig unittest bug

fix gracefulShutdownConfig unittest bug

go fmt
@LaurenceLiZhixin LaurenceLiZhixin added this to the v3.0.0-rc2 milestone Jun 16, 2021
AlexStocks pushed a commit that referenced this pull request Jun 24, 2021
* supplementary fix #1254

remove unused comments

fix import cycle

append apache license header

fix gracefulShutdownFilter unittest bug

go fmt

fix gracefulShutdownConfig unittest bug

fix gracefulShutdownConfig unittest bug

go fmt

* improve formatting based on code style

* go fmt

* set RequestsFinished explicitly

* use mutex to protect variables of ShutdownConfig

* ftr: add config (#1258)

* recover gracefulShutdownFilter logic

* remove unused mutex

Co-authored-by: Laurence <45508533+LaurenceLiZhixin@users.noreply.github.com>
mark4z pushed a commit to mark4z/dubbo-go that referenced this pull request Jul 5, 2021
* supplementary fix apache#1254

remove unused comments

fix import cycle

append apache license header

fix gracefulShutdownFilter unittest bug

go fmt

fix gracefulShutdownConfig unittest bug

fix gracefulShutdownConfig unittest bug

go fmt

* improve formatting based on code style

* go fmt

* set RequestsFinished explicitly

* use mutex to protect variables of ShutdownConfig

* ftr: add config (apache#1258)

* recover gracefulShutdownFilter logic

* remove unused mutex

Co-authored-by: Laurence <45508533+LaurenceLiZhixin@users.noreply.github.com>
AlexStocks added a commit that referenced this pull request Jul 5, 2021
…1300)

* build(deps): bump actions/cache from v2.1.4 to v2.1.5

Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5.
- [Release notes](https://github.com/actions/cache/releases)
- [Commits](actions/cache@v2.1.4...1a9e213)

Signed-off-by: dependabot[bot] <support@github.com>

* improve etcd version and change create to put (#1203)

* make the package v3router/judger test coverage rate reach 80% (#1260)

* make the package v3router/judger test coverage rate reach 80%

* add router_chain unit test

* refactor imports and some code

* remove blank lines

Co-authored-by: dongjianhui <dongjianhui@yuanfudao.com>

* Fix: Graceful shutdown bugs(supplement #1254) (#1257)

* supplementary fix #1254

remove unused comments

fix import cycle

append apache license header

fix gracefulShutdownFilter unittest bug

go fmt

fix gracefulShutdownConfig unittest bug

fix gracefulShutdownConfig unittest bug

go fmt

* improve formatting based on code style

* go fmt

* set RequestsFinished explicitly

* use mutex to protect variables of ShutdownConfig

* ftr: add config (#1258)

* recover gracefulShutdownFilter logic

* remove unused mutex

Co-authored-by: Laurence <45508533+LaurenceLiZhixin@users.noreply.github.com>

* refine grpc test code (#1266)

* refine grpc test code

* fix test

* remove useless code

* config test grpc server

* registry 默认值问题 (#1275)

Co-authored-by: wangxiaowei14227 <wangxiaowei14227@autohome.com.cn>

* config center for more parameters (#1277)

* nacos config center optimize

* up remote config

* add protocol chick

* up:代码优化

* go fmt

* fix: add arch picture in readme and delete unused router field. (#1279)

* fix

* fix: delete notify

* performance optimization: change time.After => time.NewTimer

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xin.Zh <dragoncharlie@foxmail.com>
Co-authored-by: AlexStocks <alexstocks@foxmail.com>
Co-authored-by: randy <ztelur@gmail.com>
Co-authored-by: LaurenceLiZhixin <382673304@qq.com>
Co-authored-by: Mulavar <978007503@qq.com>
Co-authored-by: dongjianhui <dongjianhui@yuanfudao.com>
Co-authored-by: XavierNiu <a@nxw.name>
Co-authored-by: Laurence <45508533+LaurenceLiZhixin@users.noreply.github.com>
Co-authored-by: gaoxinge <xg.gao@tianrang-inc.com>
Co-authored-by: wangxw666 <2484713618@qq.com>
Co-authored-by: wangxiaowei14227 <wangxiaowei14227@autohome.com.cn>
Co-authored-by: 赵云兴 <2385585770@qq.com>
@justxuewei justxuewei deleted the fix/graceful-shutdown-bug branch January 27, 2022 04:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants