fix:#1143 Feature/reduce etcd registry conn; wait group modify #1297

WilliamLeaves · 2021-07-02T10:50:10Z

What this PR does:
reduce etcd registry conn when register to etcd
and modify the logic of wait group when add() and done() of registries
Which issue(s) this PR fixes:

Fixes #1143

Special notes for your reviewer:

Does this PR introduce a user-facing change?: NO

3.0

codecov-commenter · 2021-07-02T11:08:12Z

Codecov Report

Merging #1297 (e5941ad) into 3.0 (6b1d9dd) will decrease coverage by 0.41%.
The diff coverage is 55.83%.

❗ Current head e5941ad differs from pull request most recent head c79d3c2. Consider uploading reports for the commit c79d3c2 to get more accurate results

@@            Coverage Diff             @@
##              3.0    #1297      +/-   ##
==========================================
- Coverage   56.09%   55.68%   -0.42%     
==========================================
  Files         273      273              
  Lines       12779    14998    +2219     
==========================================
+ Hits         7169     8351    +1182     
- Misses       4705     5738    +1033     
- Partials      905      909       +4

Impacted Files	Coverage Δ
common/extension/auth.go	`0.00% <0.00%> (ø)`
common/extension/metadata_remote.go	`0.00% <0.00%> (ø)`
common/extension/metadata_service.go	`0.00% <0.00%> (ø)`
common/extension/metadata_service_proxy_factory.go	`0.00% <0.00%> (ø)`
config/config_api.go	`76.71% <ø> (-1.01%)`	⬇️
config/config_loader.go	`51.34% <0.00%> (+3.97%)`	⬆️
config/reference_config.go	`81.29% <0.00%> (-0.53%)`	⬇️
filter/auth/consumer_sign_filter.go	`44.44% <0.00%> (ø)`
filter/auth/sign_util.go	`62.96% <ø> (ø)`
filter/generic/filter.go	`69.23% <0.00%> (ø)`
... and 309 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3ab0ab6...c79d3c2. Read the comment docs.

LaurenceLiZhixin · 2021-07-02T12:49:01Z

remoting/etcdv3/facade.go

+		case <-r.Client().GetCtx().Done():
+			r.RestartCallBack()
+			// re-register all services
+			time.Sleep(10 * time.Microsecond)


为什么这里要Sleep？
我看RestartCallBack的逻辑是重新注册，下面的逻辑是重建client，并且有重试逻辑。使用RestartCallBack的目的是什么呢？

代码实现参照zk对于连接过多的优化，做了类似的实现。
<-GetCtx().Done()意味着已经断连，需要调用RestartCallBack重新注册

LaurenceLiZhixin · 2021-07-06T15:56:12Z

可否简单叙述一下这么优化的思路在哪里？ @WilliamLeaves

ztelur · 2021-07-08T02:32:50Z

remoting/etcdv3/facade.go

-		err       error
-		failTimes int
-	)
-
 	defer r.WaitGroup().Done()


r.WaitGroup()的done，wait和add使用散落在各处，remoting作为基础组件，是否可能有同学直接调用 HandleClientRestart ，然后函数结束出现 negative 的问题？

有可能，用once限制一下HandleClientRestart的调用次数是否可以呢；该方法在init包被引入的时候调用，只调用一次。

AlexStocks · 2021-07-13T05:13:13Z

@WilliamLeaves pls fix the conflict file remoting/etcdv3/facade.go

WilliamLeaves · 2021-07-13T14:19:08Z

@WilliamLeaves pls fix the conflict file remoting/etcdv3/facade.go

already fixed

…m.cnpmjs.org/WilliamLeaves/dubbo-go into feature/reduce-etcd-registry-conn

…s registry

cityiron · 2021-07-23T02:08:20Z

pls Conflicting files go.sum

LaurenceLiZhixin · 2021-07-23T11:57:22Z

Pls fix your conflict and push me to merge.

…m.cnpmjs.org/WilliamLeaves/dubbo-go into feature/reduce-etcd-registry-conn

AlexStocks · 2021-07-24T19:40:31Z

registry/zookeeper/registry.go

@@ -312,3 +313,8 @@ func (r *zkRegistry) getCloseListener(conf *common.URL) (*RegistryConfigurationL

 	return zkListener, nil
 }
+
+func (r *zkRegistry) handleClientRestart() {


AsyncHandleClientRestart

AlexStocks · 2021-07-24T19:40:49Z

registry/etcdv3/registry.go

@@ -174,3 +174,8 @@ func (r *etcdV3Registry) DoSubscribe(svc *common.URL) (registry.Listener, error)
 func (r *etcdV3Registry) DoUnsubscribe(conf *common.URL) (registry.Listener, error) {
 	return nil, perrors.New("DoUnsubscribe is not support in etcdV3Registry")
 }
+
+func (r *etcdV3Registry) handleClientRestart() {


AsyncHandleClientRestart

* fix: change register event chan to unbundent chan (#1330) * fix: #1323 delete zk registry when set defualt consumer/provider config (#1324) * Ftr: Generic invocation supports Generalizer (#1315) * generic filter * sync * feat: get java name from POJO * todo: unittests for generic filter * unittests for generic filter * map generalizer * adjust import block * generic service filter is ready for generalizer * add unittests for GenericServiceFilter * ReferenceConfig supports new format of generic key * go fmt * fix license * fix license * fix bugs encountering in integrated tests * go fmt * fix serialization bugs * ftr: generic invocation supports protobuf-json * go fmt * go fmt * disable protobuf-json * go fmt * update comments * fix logging level (#1341) * Ftr: add lumberjack log config (#1335) * add lumberjack & union test * fix go fmt error,modify go.sum file * Ftr: add json generalizer (#1343) * Ftr: add json generalizer Fix: go fmt * Fix: hessian pojo * fix: restart ci * Fix: imports format * style(*): format imports with dubbogo/tools/imports-formatter (#1346) * fix Fixed the problem that the timeout period of the original method level does not take effect (#1336) * fix Fixed the problem that the timeout period of the original method level does not take effect * fix Fixed the problem that the timeout period of the original method level does not take effect Co-authored-by: 董振兴 <dongzhenxing@xiaomi.com> * fix:#1143 Feature/reduce etcd registry conn; wait group modify (#1297) * modify map use in nacos service_discovery * fix map double write bug * modify etcd registry restart mechanism * add sync.once for HandleClientRestart * modify the logic when add() or done() of wait group in zk,etcd and k8s registry * modify the logic when add() or done() of wait group in zk,etcd and k8s registry * add handleClientRestart func * add handleClientRestart func * go fmt * fix: update linter Co-authored-by: yexiaobo <yexiaobo@luojilab.com> Co-authored-by: LaurenceLiZhixin <382673304@qq.com> * ftr: triple multi params support (#1344) * ftr: triple multi params support * fix: import * fix: fix ci and gomod * fix: fmt * fix: add develop back to branch * fix: remove unused getty import * fix: mock zk registry (#1351) * 优化Nacos下的ServiceName的命名风格 (#1352) Co-authored-by: Changeden <chenzhiduan@unizone.tech> * remove zk test (#1357) * Use class name as the default reference name (#1339) * build(deps): bump actions/cache from v2.1.4 to v2.1.5 Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.4...1a9e213) Signed-off-by: dependabot[bot] <support@github.com> * improve etcd version and change create to put (#1203) * Remove RPC Service * use type assertion before reflect * modify comment * modify comment of BaseMetadataService * add the type alias of interface{} * modify RPCService Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xin.Zh <dragoncharlie@foxmail.com> Co-authored-by: AlexStocks <alexstocks@foxmail.com> Co-authored-by: randy <ztelur@gmail.com> * Feat/upgrade getty (#1345) * feat(getty): upgrade getty version to 1.4.4 * feat(getty): delete outdated getty params * delete time.Sleep in unit test * upgrade getty version to 1.4.5 * style(getty): format package getty imports Co-authored-by: dongjianhui03 <dongjianhui03@meituan.com> * support multi pb (#1361) * 修复当getty的tcp read/write timeout小于1s时panic的问题 * TCP Read/Write最小值提取为全局变量，优化parseTimeDurationByRange逻辑 * logo (#1382) * Change the key of a mock EchoFilter (#1381) * build(deps): bump actions/cache from v2.1.4 to v2.1.5 Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.4...1a9e213) Signed-off-by: dependabot[bot] <support@github.com> * improve etcd version and change create to put (#1203) * change the mock filter key Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xin.Zh <dragoncharlie@foxmail.com> Co-authored-by: AlexStocks <alexstocks@foxmail.com> Co-authored-by: randy <ztelur@gmail.com> * Change the key of a mock EchoFilter (#1381) * build(deps): bump actions/cache from v2.1.4 to v2.1.5 Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.4...1a9e213) Signed-off-by: dependabot[bot] <support@github.com> * improve etcd version and change create to put (#1203) * change the mock filter key Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xin.Zh <dragoncharlie@foxmail.com> Co-authored-by: AlexStocks <alexstocks@foxmail.com> Co-authored-by: randy <ztelur@gmail.com> * fix: nacos service disc error log bug (#1390) * 优化代码逻辑 * fix:registry timeout not pars (#1392) * fix:registry timeout not pars * up:fmt * up:统一超时时间key * fix:修改测试顺序问题 * up:fmt * change dependabot target branch * build(deps): bump github.com/fsnotify/fsnotify from 1.4.9 to 1.5.0 (#1400) * build(deps): bump actions/cache from v2.1.4 to v2.1.5 Bumps [actions/cache](https://github.com/actions/cache) from v2.1.4 to v2.1.5. - [Release notes](https://github.com/actions/cache/releases) - [Commits](actions/cache@v2.1.4...1a9e213) Signed-off-by: dependabot[bot] <support@github.com> * improve etcd version and change create to put (#1203) * build(deps): bump github.com/dubbogo/gost from 1.11.14 to 1.11.16 (#1387) Bumps [github.com/dubbogo/gost](https://github.com/dubbogo/gost) from 1.11.14 to 1.11.16. - [Release notes](https://github.com/dubbogo/gost/releases) - [Commits](dubbogo/gost@v1.11.14...v1.11.16) --- updated-dependencies: - dependency-name: github.com/dubbogo/gost dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump google.golang.org/protobuf from 1.26.0 to 1.27.1 (#1386) Bumps [google.golang.org/protobuf](https://github.com/protocolbuffers/protobuf-go) from 1.26.0 to 1.27.1. - [Release notes](https://github.com/protocolbuffers/protobuf-go/releases) - [Changelog](https://github.com/protocolbuffers/protobuf-go/blob/master/release.bash) - [Commits](protocolbuffers/protobuf-go@v1.26.0...v1.27.1) --- updated-dependencies: - dependency-name: google.golang.org/protobuf dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump go.etcd.io/etcd/client/v3 from 3.5.0-alpha.0 to 3.5.0 (#1383) Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd) from 3.5.0-alpha.0 to 3.5.0. - [Release notes](https://github.com/etcd-io/etcd/releases) - [Changelog](https://github.com/etcd-io/etcd/blob/main/CHANGELOG-3.5.md) - [Commits](etcd-io/etcd@v3.5.0-alpha.0...v3.5.0) --- updated-dependencies: - dependency-name: go.etcd.io/etcd/client/v3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): bump github.com/fsnotify/fsnotify from 1.4.9 to 1.5.0 Bumps [github.com/fsnotify/fsnotify](https://github.com/fsnotify/fsnotify) from 1.4.9 to 1.5.0. - [Release notes](https://github.com/fsnotify/fsnotify/releases) - [Changelog](https://github.com/fsnotify/fsnotify/blob/master/CHANGELOG.md) - [Commits](fsnotify/fsnotify@v1.4.9...v1.5.0) --- updated-dependencies: - dependency-name: github.com/fsnotify/fsnotify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xin.Zh <dragoncharlie@foxmail.com> Co-authored-by: AlexStocks <alexstocks@foxmail.com> Co-authored-by: randy <ztelur@gmail.com> * build(deps): bump github.com/emicklei/go-restful/v3 from 3.4.0 to 3.5.2 (#1403) Bumps [github.com/emicklei/go-restful/v3](https://github.com/emicklei/go-restful) from 3.4.0 to 3.5.2. - [Release notes](https://github.com/emicklei/go-restful/releases) - [Changelog](https://github.com/emicklei/go-restful/blob/v3/CHANGES.md) - [Commits](emicklei/go-restful@v3.4.0...v3.5.2) --- updated-dependencies: - dependency-name: github.com/emicklei/go-restful/v3 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * 更新TCP读写超时最小值命名;不再使用范围比较; * del protoc-gen-dubbo3 & protoc-gen-dubbo (#1313) Co-authored-by: Mulavar <978007503@qq.com> Co-authored-by: XavierNiu <a@nxw.name> Co-authored-by: 氕氘氚 <cjp_ca@163.com> Co-authored-by: EnableAsync <43645467+EnableAsync@users.noreply.github.com> Co-authored-by: XX <45752021+Threadalive@users.noreply.github.com> Co-authored-by: 董振兴 <dongzhenxing@xiaomi.com> Co-authored-by: WilliamLeaves <williamtheleaves@gmail.com> Co-authored-by: yexiaobo <yexiaobo@luojilab.com> Co-authored-by: ChangedenChan <changeden520@gmail.com> Co-authored-by: Changeden <chenzhiduan@unizone.tech> Co-authored-by: YuDong Tang <583125614@qq.com> Co-authored-by: alchemy-lee <2769566951@qq.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Xin.Zh <dragoncharlie@foxmail.com> Co-authored-by: AlexStocks <alexstocks@foxmail.com> Co-authored-by: randy <ztelur@gmail.com> Co-authored-by: dongjianhui03 <dongjianhui03@meituan.com> Co-authored-by: gaoxinge <xg.gao@tianrang-inc.com> Co-authored-by: wangxw666 <2484713618@qq.com> Co-authored-by: zhaoyunxing <2385585770@qq.com> Co-authored-by: Joe Zou <joezou@apache.org>

yexiaobo and others added 5 commits April 21, 2021 14:58

modify map use in nacos service_discovery

8ec8fd5

fix map double write bug

1721de0

Merge pull request #2 from apache/3.0

cbbbd8d

3.0

Merge pull request #4 from apache/3.0

a3e1e13

3.0

modify etcd registry restart mechanism

7979c1b

WilliamLeaves changed the title ~~Feature/reduce etcd registry conn~~ fix:#1143 Feature/reduce etcd registry conn Jul 2, 2021

LaurenceLiZhixin reviewed Jul 2, 2021

View reviewed changes

LaurenceLiZhixin added the enhancement New feature or request label Jul 6, 2021

LaurenceLiZhixin added this to the v3.0.0 milestone Jul 6, 2021

LaurenceLiZhixin removed the enhancement New feature or request label Jul 6, 2021

AlexStocks requested a review from gaoxinge July 7, 2021 11:27

ztelur reviewed Jul 8, 2021

View reviewed changes

zouyx marked this pull request as draft July 12, 2021 10:40

zouyx marked this pull request as ready for review July 12, 2021 10:40

Merge branch '3.0' into feature/reduce-etcd-registry-conn

9684690

yexiaobo added 2 commits July 13, 2021 22:29

add sync.once for HandleClientRestart

dc9f978

Merge branch 'feature/reduce-etcd-registry-conn' of https://github.co…

dc66fe4

…m.cnpmjs.org/WilliamLeaves/dubbo-go into feature/reduce-etcd-registry-conn

WilliamLeaves requested review from ztelur and LaurenceLiZhixin July 13, 2021 14:33

WilliamLeaves changed the title ~~fix:#1143 Feature/reduce etcd registry conn~~ [WIP]fix:#1143 Feature/reduce etcd registry conn; wait group modify Jul 16, 2021

yexiaobo added 2 commits July 19, 2021 23:25

modify the logic when add() or done() of wait group in zk,etcd and k8…

fd54619

…s registry

modify the logic when add() or done() of wait group in zk,etcd and k8…

64fbfbd

…s registry

WilliamLeaves changed the title ~~[WIP]fix:#1143 Feature/reduce etcd registry conn; wait group modify~~ fix:#1143 Feature/reduce etcd registry conn; wait group modify Jul 19, 2021

georgehao approved these changes Jul 21, 2021

View reviewed changes

ztelur approved these changes Jul 21, 2021

View reviewed changes

LaurenceLiZhixin approved these changes Jul 23, 2021

View reviewed changes

WilliamLeaves and others added 4 commits July 23, 2021 21:27

Merge branch '3.0' into feature/reduce-etcd-registry-conn

e5941ad

add handleClientRestart func

84c2602

Merge branch 'feature/reduce-etcd-registry-conn' of https://github.co…

cd567b0

…m.cnpmjs.org/WilliamLeaves/dubbo-go into feature/reduce-etcd-registry-conn

add handleClientRestart func

a6adc3b

AlexStocks reviewed Jul 24, 2021

View reviewed changes

yexiaobo and others added 2 commits July 29, 2021 21:36

go fmt

c702599

fix: update linter

c79d3c2

LaurenceLiZhixin merged commit fb0d226 into apache:3.0 Jul 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix:#1143 Feature/reduce etcd registry conn; wait group modify #1297

fix:#1143 Feature/reduce etcd registry conn; wait group modify #1297

WilliamLeaves commented Jul 2, 2021 •

edited

Loading

codecov-commenter commented Jul 2, 2021 •

edited

Loading

LaurenceLiZhixin Jul 2, 2021

WilliamLeaves Jul 9, 2021 •

edited

Loading

LaurenceLiZhixin commented Jul 6, 2021

ztelur Jul 8, 2021

WilliamLeaves Jul 9, 2021

AlexStocks commented Jul 13, 2021

WilliamLeaves commented Jul 13, 2021

cityiron commented Jul 23, 2021

LaurenceLiZhixin commented Jul 23, 2021

AlexStocks Jul 24, 2021

AlexStocks Jul 24, 2021

fix:#1143 Feature/reduce etcd registry conn; wait group modify #1297

fix:#1143 Feature/reduce etcd registry conn; wait group modify #1297

Conversation

WilliamLeaves commented Jul 2, 2021 • edited Loading

codecov-commenter commented Jul 2, 2021 • edited Loading

Codecov Report

LaurenceLiZhixin Jul 2, 2021

Choose a reason for hiding this comment

WilliamLeaves Jul 9, 2021 • edited Loading

Choose a reason for hiding this comment

LaurenceLiZhixin commented Jul 6, 2021

ztelur Jul 8, 2021

Choose a reason for hiding this comment

WilliamLeaves Jul 9, 2021

Choose a reason for hiding this comment

AlexStocks commented Jul 13, 2021

WilliamLeaves commented Jul 13, 2021

cityiron commented Jul 23, 2021

LaurenceLiZhixin commented Jul 23, 2021

AlexStocks Jul 24, 2021

Choose a reason for hiding this comment

AlexStocks Jul 24, 2021

Choose a reason for hiding this comment

WilliamLeaves commented Jul 2, 2021 •

edited

Loading

codecov-commenter commented Jul 2, 2021 •

edited

Loading

WilliamLeaves Jul 9, 2021 •

edited

Loading