grpc: v22.1.2: nil pointer dereference in transport.(*Stream).getState

[cockroach-teamcity]

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://sentry.io/organizations/cockroach-labs/issues/3670615603/?referrer=webhooks_plugin

Panic message:

panic.go:1038: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:1038
| runtime.panicmem
| GOROOT/src/runtime/panic.go:221
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:735
| google.golang.org/grpc/internal/transport.(*Stream).getState
| google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go:314
| google.golang.org/grpc/internal/transport.(*http2Client).Write
| google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go:958
| google.golang.org/grpc.(*csAttempt).sendMsg
| google.golang.org/grpc/external/org_golang_google_grpc/stream.go:935
| google.golang.org/grpc.(*clientStream).SendMsg.func2
| google.golang.org/grpc/external/org_golang_google_grpc/stream.go:800
| google.golang.org/grpc.(*clientStream).withRetry
| google.golang.org/grpc/external/org_golang_google_grpc/stream.go:680
| google.golang.org/grpc.(*clientStream).SendMsg
| google.golang.org/grpc/external/org_golang_google_grpc/stream.go:806
| github.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).SendMsg
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpc_interceptor.go:432
| github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Send
| github.com/cockroachdb/cockroach/pkg/gossip/bazel-out/k8-opt/bin/pkg/gossip/gossip_go_proto_/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:631
| github.com/cockroachdb/cockroach/pkg/gossip.(*client).sendGossip
| github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/client.go:211
| github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip
| github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/client.go:383
| github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1
| github.com/cockroachdb/cockroach/pkg/gossip/pkg/gossip/client.go:125
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:494
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1581
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:1038: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/panic.go#L1037-L1039 in runtime.gopanic
GOROOT/src/runtime/panic.go#L220-L222 in runtime.panicmem
GOROOT/src/runtime/signal_unix.go#L734-L736 in runtime.sigpanic
google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go#L313-L315 in google.golang.org/grpc/internal/transport.(*Stream).getState
google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go#L957-L959 in google.golang.org/grpc/internal/transport.(*http2Client).Write
google.golang.org/grpc/external/org_golang_google_grpc/stream.go#L934-L936 in google.golang.org/grpc.(*csAttempt).sendMsg
google.golang.org/grpc/external/org_golang_google_grpc/stream.go#L799-L801 in google.golang.org/grpc.(*clientStream).SendMsg.func2
google.golang.org/grpc/external/org_golang_google_grpc/stream.go#L679-L681 in google.golang.org/grpc.(*clientStream).withRetry
google.golang.org/grpc/external/org_golang_google_grpc/stream.go#L805-L807 in google.golang.org/grpc.(*clientStream).SendMsg

cockroach/pkg/util/tracing/grpc_interceptor.go

Lines 431 to 433 in fc456a2

	func (cs *tracingClientStream) SendMsg(m interface{}) error {
	err := cs.ClientStream.SendMsg(m)
	if err != nil {

in pkg/util/tracing.(*tracingClientStream).SendMsg
https://github.com/cockroachdb/cockroach/blob/fc456a26830067b6dfbb8cd87e093a28d9b833d1/pkg/gossip/gossip.pb.go#L630-L632 in pkg/gossip.(*gossipGossipClient).Send
https://github.com/cockroachdb/cockroach/blob/fc456a26830067b6dfbb8cd87e093a28d9b833d1/pkg/gossip/pkg/gossip/client.go#L210-L212 in pkg/gossip.(*client).sendGossip
https://github.com/cockroachdb/cockroach/blob/fc456a26830067b6dfbb8cd87e093a28d9b833d1/pkg/gossip/pkg/gossip/client.go#L382-L384 in pkg/gossip.(*client).gossip
https://github.com/cockroachdb/cockroach/blob/fc456a26830067b6dfbb8cd87e093a28d9b833d1/pkg/gossip/pkg/gossip/client.go#L124-L126 in pkg/gossip.(*client).startLocked.func1

cockroach/pkg/util/stop/stopper.go

Lines 493 to 495 in fc456a2

	f(ctx)
	}()

in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
GOROOT/src/runtime/asm_amd64.s#L1580-L1582 in runtime.goexit

GOROOT/src/runtime/panic.go in runtime.gopanic at line 1038
GOROOT/src/runtime/panic.go in runtime.panicmem at line 221
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 735
google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go in google.golang.org/grpc/internal/transport.(*Stream).getState at line 314
google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go in google.golang.org/grpc/internal/transport.(*http2Client).Write at line 958
google.golang.org/grpc/external/org_golang_google_grpc/stream.go in google.golang.org/grpc.(*csAttempt).sendMsg at line 935
google.golang.org/grpc/external/org_golang_google_grpc/stream.go in google.golang.org/grpc.(*clientStream).SendMsg.func2 at line 800
google.golang.org/grpc/external/org_golang_google_grpc/stream.go in google.golang.org/grpc.(*clientStream).withRetry at line 680
google.golang.org/grpc/external/org_golang_google_grpc/stream.go in google.golang.org/grpc.(*clientStream).SendMsg at line 806
pkg/util/tracing/grpc_interceptor.go in pkg/util/tracing.(*tracingClientStream).SendMsg at line 432
pkg/gossip/gossip.pb.go in pkg/gossip.(*gossipGossipClient).Send at line 631
pkg/gossip/pkg/gossip/client.go in pkg/gossip.(*client).sendGossip at line 211
pkg/gossip/pkg/gossip/client.go in pkg/gossip.(*client).gossip at line 383
pkg/gossip/pkg/gossip/client.go in pkg/gossip.(*client).startLocked.func1 at line 125
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 494
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1581

Tag	Value
Cockroach Release	v22.1.2
Cockroach SHA:	fc456a2
Platform	linux amd64
Distribution	CCL
Environment	v22.1.2
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-20523

[yuzefovich]

I think I hit this problem when running tpch_concurrency roachtest using GOMEMLIMIT=7.5GiB (on the custom branch) with concurrency 1088 on Q11. The cluster was in complete overload mode.

grpc.zip

[yuzefovich]

This seems like an issue upstream that couldn't be reproduced.

[yuzefovich]

Hey @howardjohn, I ran into an issue similar to what you opened about a year ago. It appears that the fix has likely been merged in grpc/grpc-go#5323. Just wanted to let you know :)

[yuzefovich]

Dup of #81227 - will be fixed once we upgrade gRPC dep.