Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

使用北极星作为服务发现时,在注册多个grpc client时,启动报错:concurrent map iteration and map write #3361

Closed
JackChanw opened this issue Jul 9, 2024 · 4 comments
Labels
bug Something isn't working

Comments

@JackChanw
Copy link

What happened:

INFO msg=watcher's ctx cancel : context canceled
INFO msg=[resolver] update instances: [{"id":"7e55bf0237ad7a213c338824f08a89552fa2a1fb","name":"bv_media_server.grpc","version":"","metadata":{"env":"prod","env_id":"0","env_name":"","env_tag":"","protocol":"grpc","version":"","weight":"100"},"endpoints":["grpc://golden-us04.domob.cn:9789"]},{"id":"d742c26c247bbc377f8a99ac9c10e402f04da209","name":"bv_media_server.grpc","version":"","metadata":{"env":"prod","env_id":"0","env_name":"","env_tag":"","protocol":"grpc","version":"","weight":"100"},"endpoints":["grpc://golden-us03.domob.cn:9789"]}]
INFO msg=[resolver] update instances: [{"id":"7e55bf0237ad7a213c338824f08a89552fa2a1fb","name":"bv_media_server.grpc","version":"","metadata":{"env":"prod","env_id":"0","env_name":"","env_tag":"","protocol":"grpc","version":"","weight":"100"},"endpoints":["grpc://golden-us04.domob.cn:9789"]},{"id":"d742c26c247bbc377f8a99ac9c10e402f04da209","name":"bv_media_server.grpc","version":"","metadata":{"env":"prod","env_id":"0","env_name":"","env_tag":"","protocol":"grpc","version":"","weight":"100"},"endpoints":["grpc://golden-us03.domob.cn:9789"]}]
fatal error: concurrent map iteration and map write

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

goroutine 170 [running]:
github.com/go-kratos/kratos/v2/registry.(*ServiceInstance).Equal(0xc00089f630, {0x1636220?, 0xc00089f630})
/home/gitlab-runner/go/pkg/mod/github.com/go-kratos/kratos/v2@v2.7.3/registry/registry.go:88 +0x225
google.golang.org/grpc/attributes.(*Attributes).Equal(0x16092a0?, 0xc00067c260)
/home/gitlab-runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/attributes/attributes.go:96 +0x1b1
google.golang.org/grpc/resolver.addressMapEntryList.find(...)
/home/gitlab-runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/resolver/map.go:64
google.golang.org/grpc/resolver.(*AddressMap).Get(0xc000b5d9d8?, {{0xc000028047, 0x19}, {0xc00067a138, 0x12}, 0xc00067c260, 0x0, {0x0, 0x0}})
/home/gitlab-runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/resolver/map.go:75 +0x185
google.golang.org/grpc/balancer/base.(*baseBalancer).UpdateClientConnState(0xc000c91f00, {{{0xc000c91e80, 0x2, 0x2}, {0xc00093e9c0, 0x2, 0x2}, 0x0, 0x0}, {0x0, ...}})
/home/gitlab-runner/go/pkg/mod/google.golang.org/grpc@v1.63.2/balancer/base/balancer.go:125 +0x4a5
google.golang.org/grpc/internal/balancer/gracefulswitch.(*Balancer).UpdateClientConnState(0xffffffffffffffff?, {{{0xc000c91e80, 0x2

Environment:

  • Kratos version (use kratos -v):
  • Go version (use go version):
  • OS (e.g: cat /etc/os-release):
  • Others:
@JackChanw JackChanw added the bug Something isn't working label Jul 9, 2024
@JackChanw
Copy link
Author

目前服务为Task模式,启动时使用discovery 链接多个grpc服务,其中有部分服务属于同一个实例(endpoint一致)

@kratos-ci-bot
Copy link
Collaborator

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿


The current service is in Task mode. Discovery is used to link multiple grpc services when starting. Some of the services belong to the same instance (the endpoints are consistent).

@JackChanw
Copy link
Author

完整错误栈
goroutine 157 [running]:
github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.parseAttributes(...)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/resolver.go:111
github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*discoveryResolver).update(0x140009bfe60, {0x14000af8410, 0xa, 0xa})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/resolver.go:81 +0x368
github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*discoveryResolver).watch(0x140009bfe60)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/resolver.go:47 +0x74
created by github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*builder).Build
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/builder.go:116 +0x414

goroutine 1 [runnable]:
google.golang.org/grpc.(*ccResolverWrapper).start(0x14000a16980)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/resolver_wrapper.go:84 +0xc4
google.golang.org/grpc.(*ClientConn).exitIdleMode(0x14000afc400)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/clientconn.go:328 +0x50
google.golang.org/grpc.(*idler).ExitIdleMode(0x0?)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/clientconn.go:311 +0x1c
google.golang.org/grpc/internal/idle.(*Manager).ExitIdleMode(0x140000b8480)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/idle/idle.go:236 +0xb8
google.golang.org/grpc.DialContext({0x105bdf8e8, 0x14000134050}, {0x14000802220, 0x1f}, {0x140009bfec0, 0x5, 0x0?})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/clientconn.go:231 +0x1a0
github.com/go-kratos/kratos/v2/transport/grpc.dial({0x105bdf8e8, 0x14000134050}, 0x1, {0x14000667b48, 0x4, 0x1050d6fac?})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/client.go:204 +0xa90
github.com/go-kratos/kratos/v2/transport/grpc.DialInsecure(...)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/client.go:151
git.domob-inc.cn/bluevision/bv_commons/gkratos_x/kxgrpc.(*ClientFactory).CreateConnWithTimeout(0x14000667e08, {0x14000802220, 0x1f}, 0x2540be400)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/git.domob-inc.cn/bluevision/bv_commons/gkratos_x/kxgrpc/client_factory.go:48 +0x2c8
bv_order_server/internal/data/client.NewPayCenterLakalaClient(0x14000910a80, 0x1400074f040?)
/Users/chanw/Domob/Bluevision/bv_order_server/internal/data/client/grpc_client.go:29 +0x60
main.initApp(0x140002db900, 0x14000910a80, {0x105bcbd00, 0x1400074f040})
/Users/chanw/Domob/Bluevision/bv_order_server/cmd/bv_order_server/wire_gen.go:119 +0x4f4
main.main()
/Users/chanw/Domob/Bluevision/bv_order_server/cmd/bv_order_server/main.go:58 +0x14c

goroutine 36 [chan receive]:
github.com/polarismesh/polaris-go/pkg/clock.(*clockImpl).updateTime(0x140003e37f0)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/clock/clock.go:57 +0x88
created by github.com/polarismesh/polaris-go/pkg/clock.init.0
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/clock/clock.go:74 +0x120

goroutine 10 [select]:
github.com/patrickmn/go-cache.(*janitor).Run(0x14000624c60, 0x0?)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0x78
created by github.com/patrickmn/go-cache.runJanitor
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0x100

goroutine 24 [select]:
github.com/polarismesh/polaris-go/plugin/metrics/prometheus.(*PullAction).doAggregation(0x140002a40f0, {0x105bdf8b0, 0x14000258380})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/metrics/prometheus/reporter.go:299 +0xa4
created by github.com/polarismesh/polaris-go/plugin/metrics/prometheus.(*PullAction).Run
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/metrics/prometheus/reporter.go:312 +0xa4

goroutine 26 [chan receive]:
github.com/natefinch/lumberjack.(*Logger).millRun(0x14000132720)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/natefinch/lumberjack/lumberjack.go:379 +0x44
created by github.com/natefinch/lumberjack.(*Logger).mill.func1
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/natefinch/lumberjack/lumberjack.go:390 +0xa4

goroutine 14 [chan receive]:
github.com/natefinch/lumberjack.(*Logger).millRun(0x14000132300)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/natefinch/lumberjack/lumberjack.go:379 +0x44
created by github.com/natefinch/lumberjack.(*Logger).mill.func1
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/natefinch/lumberjack/lumberjack.go:390 +0xa4

goroutine 50 [select]:
github.com/polarismesh/polaris-go/pkg/network.(*connectionManager).doSwitchRoutine(0x14000160a80)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/network/impl.go:493 +0xe4
created by github.com/polarismesh/polaris-go/pkg/network.NewConnectionManager
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/network/impl.go:307 +0x590

goroutine 72 [select]:
github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).doSend(0x1400022a780)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:216 +0xc8
created by github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).StartUpdateRoutines
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:123 +0x124

goroutine 73 [select]:
github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).doRetry(0x1400022a780)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:171 +0xd0
created by github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).StartUpdateRoutines
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:124 +0x168

goroutine 74 [select]:
github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).doLog(0x1400022a780)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:142 +0xd4
created by github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).StartUpdateRoutines
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:125 +0x1ac

goroutine 75 [select]:
github.com/polarismesh/polaris-go/plugin/localregistry/inmemory.(*LocalCache).logServiceMap(0x1400020a0f0)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/localregistry/inmemory/inmemory.go:195 +0xd4
created by github.com/polarismesh/polaris-go/plugin/localregistry/inmemory.(*LocalCache).Start
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/localregistry/inmemory/inmemory.go:220 +0xb4

goroutine 76 [select]:
github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).runPeriod(0x14000808be0)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:179 +0xc4
created by github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).start
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:109 +0x2e8

goroutine 77 [select]:
github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).runPeriod(0x14000808c80)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:179 +0xc4
created by github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).start
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:109 +0x2e8

goroutine 25 [IO wait]:
internal/poll.runtime_pollWait(0x12e8220d8, 0x72)
/Users/chanw/Go/go1.19.1/src/runtime/netpoll.go:305 +0xa0
internal/poll.(*pollDesc).wait(0x14000796000?, 0x105a4a420?, 0x0)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000796000)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_unix.go:614 +0x1d0
net.(*netFD).accept(0x14000796000)
/Users/chanw/Go/go1.19.1/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x14000282078)
/Users/chanw/Go/go1.19.1/src/net/tcpsock_posix.go:142 +0x28
net.(*TCPListener).Accept(0x14000282078)
/Users/chanw/Go/go1.19.1/src/net/tcpsock.go:288 +0x2c
net/http.(*Server).Serve(0x1400079c000, {0x105bde510, 0x14000282078})
/Users/chanw/Go/go1.19.1/src/net/http/server.go:3070 +0x30c
net/http.Serve(...)
/Users/chanw/Go/go1.19.1/src/net/http/server.go:2568
github.com/polarismesh/polaris-go/plugin/metrics/prometheus.(*PullAction).Run.func1()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/metrics/prometheus/reporter.go:327 +0x354
created by github.com/polarismesh/polaris-go/plugin/metrics/prometheus.(*PullAction).Run
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/metrics/prometheus/reporter.go:313 +0xe8

goroutine 44 [chan receive]:
github.com/natefinch/lumberjack.(*Logger).millRun(0x14000132660)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/natefinch/lumberjack/lumberjack.go:379 +0x44
created by github.com/natefinch/lumberjack.(*Logger).mill.func1
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/natefinch/lumberjack/lumberjack.go:390 +0xa4

goroutine 147 [select]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-sql-driver/mysql/connection.go:614 +0x7c
created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-sql-driver/mysql/connection.go:611 +0x110

goroutine 106 [select]:
github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).doSend(0x140004e0180)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:216 +0xc8
created by github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).StartUpdateRoutines
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:123 +0x124

goroutine 107 [select]:
github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).doRetry(0x140004e0180)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:171 +0xd0
created by github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).StartUpdateRoutines
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:124 +0x168

goroutine 108 [select]:
github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).doLog(0x140004e0180)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:142 +0xd4
created by github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).StartUpdateRoutines
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:125 +0x1ac

goroutine 48 [select]:
database/sql.(*DB).connectionOpener(0x140004d2ea0, {0x105bdf8b0, 0x14000274040})
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1224 +0x80
created by database/sql.OpenDB
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:792 +0x180

goroutine 105 [select]:
github.com/polarismesh/polaris-go/plugin/localregistry/inmemory.(*LocalCache).logServiceMap(0x14000a02000)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/localregistry/inmemory/inmemory.go:195 +0xd4
created by github.com/polarismesh/polaris-go/plugin/localregistry/inmemory.(*LocalCache).Start
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/localregistry/inmemory/inmemory.go:220 +0xb4

goroutine 82 [select]:
github.com/polarismesh/polaris-go/pkg/network.(*connectionManager).doSwitchRoutine(0x140004cc000)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/network/impl.go:493 +0xe4
created by github.com/polarismesh/polaris-go/pkg/network.NewConnectionManager
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/network/impl.go:307 +0x590

goroutine 109 [select]:
github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).runPeriod(0x140002a5090)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:179 +0xc4
created by github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).start
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:109 +0x2e8

goroutine 110 [select]:
github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).runPeriod(0x140002a5130)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:179 +0xc4
created by github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).start
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:109 +0x2e8

goroutine 111 [select]:
github.com/polarismesh/polaris-go/plugin/metrics/prometheus.(*PullAction).doAggregation(0x140002a5220, {0x105bdf8b0, 0x14000258d00})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/metrics/prometheus/reporter.go:299 +0xa4
created by github.com/polarismesh/polaris-go/plugin/metrics/prometheus.(*PullAction).Run
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/metrics/prometheus/reporter.go:312 +0xa4

goroutine 137 [IO wait]:
internal/poll.runtime_pollWait(0x12e821fe8, 0x72)
/Users/chanw/Go/go1.19.1/src/runtime/netpoll.go:305 +0xa0
internal/poll.(*pollDesc).wait(0x14000796780?, 0x105a4a420?, 0x0)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000796780)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_unix.go:614 +0x1d0
net.(*netFD).accept(0x14000796780)
/Users/chanw/Go/go1.19.1/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x14000283008)
/Users/chanw/Go/go1.19.1/src/net/tcpsock_posix.go:142 +0x28
net.(*TCPListener).Accept(0x14000283008)
/Users/chanw/Go/go1.19.1/src/net/tcpsock.go:288 +0x2c
net/http.(*Server).Serve(0x1400079c3c0, {0x105bde510, 0x14000283008})
/Users/chanw/Go/go1.19.1/src/net/http/server.go:3070 +0x30c
net/http.(*Server).ListenAndServe(0x1400079c3c0)
/Users/chanw/Go/go1.19.1/src/net/http/server.go:2999 +0x84
net/http.ListenAndServe({0x1400049d678, 0x6}, {0x0?, 0x0})
/Users/chanw/Go/go1.19.1/src/net/http/server.go:3255 +0x94
created by git.domob-inc.cn/bluefocus/golang_libs/bfomonitor.start
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/git.domob-inc.cn/bluefocus/golang_libs/bfomonitor/pmonitor.go:111 +0x1d4

goroutine 140 [select]:
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).processQueue(0x140006f94a0)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/go.opentelemetry.io/otel/sdk/trace/batch_span_processor.go:293 +0xd8
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/go.opentelemetry.io/otel/sdk/trace/batch_span_processor.go:128 +0x58
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/go.opentelemetry.io/otel/sdk/trace/batch_span_processor.go:126 +0x2a4

goroutine 141 [select]:
database/sql.(*DB).connectionOpener(0x140001d8340, {0x105bdf8b0, 0x1400064d9c0})
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1224 +0x80
created by database/sql.OpenDB
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:792 +0x180

goroutine 144 [select]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-sql-driver/mysql/connection.go:614 +0x7c
created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-sql-driver/mysql/connection.go:611 +0x110

goroutine 166 [select]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0x140006dd720, {0x14000431480, 0x5, 0x5})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/transport.go:193 +0x74
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0x140006dd720, {0x14000431480?, 0x140005fa3d8?, 0x14000987740?})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/transport.go:173 +0x148
google.golang.org/grpc/internal/transport.(*transportReader).Read(0x14000431230, {0x14000431480?, 0x140009877c8?, 0x104b900b8?})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/transport.go:525 +0x34
io.ReadAtLeast({0x105bce540, 0x14000431230}, {0x14000431480, 0x5, 0x5}, 0x5)
/Users/chanw/Go/go1.19.1/src/io/io.go:332 +0xa0
io.ReadFull(...)
/Users/chanw/Go/go1.19.1/src/io/io.go:351
google.golang.org/grpc/internal/transport.(*Stream).Read(0x14000922c60, {0x14000431480, 0x5, 0x5})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/transport.go:509 +0x98
google.golang.org/grpc.(*parser).recvMsg(0x14000431470, 0x3200000)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/rpc_util.go:614 +0x48
google.golang.org/grpc.recvAndDecompress(0x14000431470, 0x14000922c60, {0x0, 0x0}, 0x3200000, 0x0, {0x0, 0x0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/rpc_util.go:753 +0x58
google.golang.org/grpc.recv(0x140009879f8?, {0x12e82c508, 0x1068bd750}, 0x1045f3e18?, {0x0?, 0x0?}, {0x105b4a4c0, 0x1400022b380}, 0x65?, 0x0?, ...)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/rpc_util.go:833 +0x64
google.golang.org/grpc.(*csAttempt).recvMsg(0x140001d85b0, {0x105b4a4c0?, 0x1400022b380}, 0x140008bf6d8?)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/stream.go:1085 +0x240
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xd900000106e8ca68?)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/stream.go:928 +0x28
google.golang.org/grpc.(*clientStream).withRetry(0x14000922a20, 0x14000987c38, 0x14000987c28)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/stream.go:761 +0x568
google.golang.org/grpc.(*clientStream).RecvMsg(0x14000922a20, {0x105b4a4c0?, 0x1400022b380?})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/stream.go:927 +0xbc
github.com/polarismesh/specification/source/go/api/v1/service_manage.(*polarisGRPCDiscoverClient).Recv(0x14000737850)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/specification/source/go/api/v1/service_manage/grpcapi.pb.go:110 +0x58
github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*StreamingClient).receiveAndNotify(0x140001b2230)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:479 +0x138
created by github.com/polarismesh/polaris-go/plugin/serverconnector/common.(*DiscoverConnector).newStream
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/plugin/serverconnector/common/discover.go:604 +0x828

goroutine 139 [IO wait]:
internal/poll.runtime_pollWait(0x12e821ef8, 0x72)
/Users/chanw/Go/go1.19.1/src/runtime/netpoll.go:305 +0xa0
internal/poll.(*pollDesc).wait(0x140000ce000?, 0x105a4a420?, 0x0)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x140000ce000)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_unix.go:614 +0x1d0
net.(*netFD).accept(0x140000ce000)
/Users/chanw/Go/go1.19.1/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x14000a00288)
/Users/chanw/Go/go1.19.1/src/net/tcpsock_posix.go:142 +0x28
net.(*TCPListener).Accept(0x14000a00288)
/Users/chanw/Go/go1.19.1/src/net/tcpsock.go:288 +0x2c
net/http.(*Server).Serve(0x14000a020f0, {0x105bde510, 0x14000a00288})
/Users/chanw/Go/go1.19.1/src/net/http/server.go:3070 +0x30c
net/http.(*Server).ListenAndServe(0x14000a020f0)
/Users/chanw/Go/go1.19.1/src/net/http/server.go:2999 +0x84
net/http.ListenAndServe(...)
/Users/chanw/Go/go1.19.1/src/net/http/server.go:3255
git.domob-inc.cn/bluevision/bv_commons/gkratos_x/kxserverob.InitServerObserver.func1()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/git.domob-inc.cn/bluevision/bv_commons/gkratos_x/kxserverob/server_observer.go:13 +0x40
created by git.domob-inc.cn/bluevision/bv_commons/gkratos_x/kxserverob.InitServerObserver
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/git.domob-inc.cn/bluevision/bv_commons/gkratos_x/kxserverob/server_observer.go:12 +0x30

goroutine 47 [select]:
database/sql.(*DB).connectionCleaner(0x140001d8340, 0x0?)
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1067 +0x90
created by database/sql.(*DB).startCleanerLocked
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1054 +0x108

goroutine 148 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140009f0e70, {0x105bdf8b0, 0x14000275500})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 118 [select]:
github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-sql-driver/mysql/connection.go:614 +0x7c
created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-sql-driver/mysql/connection.go:611 +0x110

goroutine 119 [select]:
database/sql.(*DB).connectionCleaner(0x140004d2ea0, 0x0?)
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1067 +0x90
created by database/sql.(*DB).startCleanerLocked
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1054 +0x108

goroutine 120 [select]:
database/sql.(*DB).connectionOpener(0x140004e4dd0, {0x105bdf8b0, 0x14000259f80})
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1224 +0x80
created by database/sql.OpenDB
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:792 +0x180

goroutine 123 [select]:
database/sql.(*DB).connectionCleaner(0x140004e4dd0, 0x0?)
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1067 +0x90
created by database/sql.(*DB).startCleanerLocked
/Users/chanw/Go/go1.19.1/src/database/sql/sql.go:1054 +0x108

goroutine 124 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14000270580, {0x105bdf8b0, 0x1400074e180})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 125 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140002705b0, {0x105bdf8b0, 0x1400074e200})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 126 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140002705e0, {0x105bdf8b0, 0x1400074e280})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 152 [runnable]:
encoding/json.stateInString(0x140007d5ec0?, 0x30?)
/Users/chanw/Go/go1.19.1/src/encoding/json/scanner.go:337 +0x1d8
encoding/json.Indent(0x14000a3d8b0, {0x14000b0a000, 0x1dee, 0x1045fd370?}, {0x0, 0x0}, {0x10553139a, 0x2})
/Users/chanw/Go/go1.19.1/src/encoding/json/indent.go:89 +0xf8
encoding/json.MarshalIndent({0x105a72940?, 0x140007d5e80?}, {0x0, 0x0}, {0x10553139a, 0x2})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:180 +0x68
google.golang.org/grpc/internal/pretty.ToJSON({0x105a72940, 0x140007d5e80})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/pretty/pretty.go:56 +0x198
google.golang.org/grpc.(*ccResolverWrapper).addChannelzTraceEvent(0x14000797280, {{0x1400080c000, 0xa, 0xa}, {0x1400017c780, 0xa, 0xa}, 0x0, 0x0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/resolver_wrapper.go:197 +0x2e0
google.golang.org/grpc.(*ccResolverWrapper).UpdateState(0x14000797280, {{0x1400080c000, 0xa, 0xa}, {0x1400017c780, 0xa, 0xa}, 0x0, 0x0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/resolver_wrapper.go:133 +0x1b0
github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*discoveryResolver).update(0x140009bf7a0, {0x140004e3d60, 0xa, 0xa})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/resolver.go:90 +0x5e4
github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*discoveryResolver).watch(0x140009bf7a0)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/resolver.go:47 +0x74
created by github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*builder).Build
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/builder.go:116 +0x414

goroutine 180 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14000271190, {0x105bdf8b0, 0x1400074f0c0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 179 [runnable]:
google.golang.org/grpc/attributes.(*Attributes).String(0x14000626990?)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/attributes/attributes.go:113 +0x100
google.golang.org/grpc/attributes.(*Attributes).MarshalJSON(0x1059eeee0?)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/attributes/attributes.go:140 +0x1c
encoding/json.marshalerEncoder(0x140009d8a80, {0x1059eeee0?, 0x14000656720?, 0x14000783b00?}, {0x0?, 0x10?})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:478 +0xc4
encoding/json.structEncoder.encode({{{0x1400027b680?, 0x5?, 0x8?}, 0x140008fb230?}}, 0x140009d8a80, {0x105ad6a80?, 0x14000656700?, 0x1047457d4?}, {0x0?, 0x1?})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:760 +0x178
encoding/json.arrayEncoder.encode({0x0?}, 0x140009d8a80, {0x10587b380?, 0x140008fc2c0?, 0x0?}, {0x0?, 0x0?})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:915 +0xc0
encoding/json.sliceEncoder.encode({0x1400066d608?}, 0x140009d8a80, {0x10587b380?, 0x140008fc2c0?, 0x2600000000000000?}, {0xc?, 0x0?})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:888 +0x298
encoding/json.structEncoder.encode({{{0x140007898c0?, 0x105bf41d8?, 0x0?}, 0x140008fb410?}}, 0x140009d8a80, {0x105a72940?, 0x140008fc2c0?, 0x105a72940?}, {0x48?, 0xd7?})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:760 +0x178
encoding/json.(*encodeState).reflectValue(0x1400066d7c8?, {0x105a72940?, 0x140008fc2c0?, 0x1400066d818?}, {0x2?, 0x0?})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:359 +0x70
encoding/json.(*encodeState).marshal(0x1071ebfff?, {0x105a72940?, 0x140008fc2c0?}, {0x0?, 0x4?})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:331 +0x120
encoding/json.Marshal({0x105a72940, 0x140008fc2c0})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:160 +0x3c
encoding/json.MarshalIndent({0x105a72940?, 0x140008fc2c0?}, {0x0, 0x0}, {0x10553139a, 0x2})
/Users/chanw/Go/go1.19.1/src/encoding/json/encode.go:175 +0x34
google.golang.org/grpc/internal/pretty.ToJSON({0x105a72940, 0x140008fc2c0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/pretty/pretty.go:56 +0x198
google.golang.org/grpc.(*ccResolverWrapper).addChannelzTraceEvent(0x14000a16900, {{0x14000656500, 0xa, 0xa}, {0x14000260b40, 0xa, 0xa}, 0x0, 0x0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/resolver_wrapper.go:197 +0x2e0
google.golang.org/grpc.(*ccResolverWrapper).UpdateState(0x14000a16900, {{0x14000656500, 0xa, 0xa}, {0x14000260b40, 0xa, 0xa}, 0x0, 0x0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/resolver_wrapper.go:133 +0x1b0
github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*discoveryResolver).update(0x140005b3920, {0x14000651d60, 0xa, 0xa})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/resolver.go:90 +0x5e4
github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*discoveryResolver).watch(0x140005b3920)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/resolver.go:47 +0x74
created by github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*builder).Build
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/builder.go:116 +0x414

goroutine 149 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140009f0ea0, {0x105bdf8b0, 0x14000275640})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 150 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140009f0ed0, {0x105bdf8b0, 0x14000275680})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 165 [select]:
google.golang.org/grpc.newClientStreamWithParams.func4()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/stream.go:392 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/stream.go:391 +0xc44

goroutine 163 [runnable]:
internal/poll.runtime_pollWait(0x12e821b38, 0x72)
/Users/chanw/Go/go1.19.1/src/runtime/netpoll.go:305 +0xa0
internal/poll.(*pollDesc).wait(0x14000909680?, 0x14000630000?, 0x0)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14000909680, {0x14000630000, 0x8000, 0x8000})
/Users/chanw/Go/go1.19.1/src/internal/poll/fd_unix.go:167 +0x1e0
net.(*netFD).Read(0x14000909680, {0x14000630000?, 0x140008c3ca8?, 0x1046f271c?})
/Users/chanw/Go/go1.19.1/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x140006265f8, {0x14000630000?, 0x140008c3d18?, 0x104b7d80c?})
/Users/chanw/Go/go1.19.1/src/net/net.go:183 +0x34
bufio.(*Reader).Read(0x14000911a40, {0x140004cc3c0, 0x9, 0x0?})
/Users/chanw/Go/go1.19.1/src/bufio/bufio.go:237 +0x1e0
io.ReadAtLeast({0x105bc75a0, 0x14000911a40}, {0x140004cc3c0, 0x9, 0x9}, 0x9)
/Users/chanw/Go/go1.19.1/src/io/io.go:332 +0xa0
io.ReadFull(...)
/Users/chanw/Go/go1.19.1/src/io/io.go:351
golang.org/x/net/http2.readFrameHeader({0x140004cc3c0?, 0x9?, 0x14000430330?}, {0x105bc75a0?, 0x14000911a40?})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/golang.org/x/net/http2/frame.go:237 +0x58
golang.org/x/net/http2.(*Framer).ReadFrame(0x140004cc380)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/golang.org/x/net/http2/frame.go:498 +0x84
google.golang.org/grpc/internal/transport.(*http2Client).reader(0x14000788b40, 0x0?)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1602 +0x1b8
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/http2_client.go:409 +0x15b8

goroutine 164 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0x140006dd540, 0x1)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:418 +0x154
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0x1400025f0a0)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:551 +0x7c
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/http2_client.go:463 +0x70
created by google.golang.org/grpc/internal/transport.newHTTP2Client
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/transport/http2_client.go:461 +0x1a78

goroutine 27 [select]:
github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).runTakePriority(0x14000808960)
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:133 +0xd0
created by github.com/polarismesh/polaris-go/pkg/flow/schedule.(*taskRoutine).start
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/polarismesh/polaris-go/pkg/flow/schedule/routines.go:106 +0x1b4

goroutine 153 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140006249e0, {0x105bdf8b0, 0x14000a566c0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 154 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14000624a20, {0x105bdf8b0, 0x14000a56740})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 155 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14000624a70, {0x105bdf8b0, 0x14000a56780})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 181 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140002711d0, {0x105bdf8b0, 0x1400074f640})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 182 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14000271220, {0x105bdf8b0, 0x1400074f6c0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 158 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140006254a0, {0x105bdf8b0, 0x14000a568c0})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 159 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x140006254e0, {0x105bdf8b0, 0x14000a56940})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 160 [select]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0x14000625510, {0x105bdf8b0, 0x14000a56980})
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:76 +0xc8
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/google.golang.org/grpc/internal/grpcsync/callback_serializer.go:52 +0x144

goroutine 194 [runnable]:
github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*builder).Build.func2()
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/builder.go:116
runtime.goexit()
/Users/chanw/Go/go1.19.1/src/runtime/asm_arm64.s:1172 +0x4
created by github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery.(*builder).Build
/Users/chanw/Domob/Bluevision/bv_order_server/vendor/github.com/go-kratos/kratos/v2/transport/grpc/resolver/discovery/builder.go:116 +0x414

@shenqidebaozi
Copy link
Member

Can you help fix it directly

shenqidebaozi pushed a commit that referenced this issue Aug 19, 2024
… (#3386)

* fix(registry/polaris): fix concurrent map iteration and map write #3361

* fix(test): support go 1.20.x

* fix(test): fix test mock concurrent read

* fix: fix Metadata use rmd & remove dup space
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants