Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | runtime.panicmem ... #97556

Closed
cockroach-teamcity opened this issue Feb 23, 2023 · 1 comment
Closed

sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | runtime.panicmem ... #97556

cockroach-teamcity opened this issue Feb 23, 2023 · 1 comment
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Feb 23, 2023 •
edited by cockroach-jira-scripts
Loading

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://cockroach-labs.sentry.io/issues/3954098901/?referrer=webhooks_plugin

Panic message:

panic.go:884: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
-- stack trace:
| runtime.gopanic
| GOROOT/src/runtime/panic.go:884
| runtime.panicmem
| GOROOT/src/runtime/panic.go:260
| runtime.sigpanic
| GOROOT/src/runtime/signal_unix.go:835
| google.golang.org/grpc/internal/transport.(*Stream).compareAndSwapState
| google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go:311
| google.golang.org/grpc/internal/transport.(*http2Client).Write
| google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go:954
| google.golang.org/grpc.(*clientStream).CloseSend.func1
| google.golang.org/grpc/external/org_golang_google_grpc/stream.go:872
| google.golang.org/grpc.(*clientStream).withRetry
| google.golang.org/grpc/external/org_golang_google_grpc/stream.go:688
| google.golang.org/grpc.(*clientStream).CloseSend
| google.golang.org/grpc/external/org_golang_google_grpc/stream.go:879
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.(*tracingClientStream).CloseSend
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:398
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).SendSnapshot.func1
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:640
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).SendSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:644
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).followerSendSnapshot.func5
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:2885
| github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout
| github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:91
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).followerSendSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:2883
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleDelegatedSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:180
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).DelegateRaftSnapshot
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:356
| github.com/cockroachdb/cockroach/pkg/kv/kvserver.MultiRaft_DelegateRaftSnapshot_Handler
| github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-opt/bin/pkg/kv/kvserver/kvserver_go_proto/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:296
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1
| github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:163
| google.golang.org/grpc.chainStreamInterceptors.func1.1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1408
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:272
| google.golang.org/grpc.chainStreamInterceptors.func1.1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1411
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:241
| github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
| github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:341
| github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2
| github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:240
| google.golang.org/grpc.chainStreamInterceptors.func1.1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1411
| google.golang.org/grpc.chainStreamInterceptors.func1
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1413
| google.golang.org/grpc.(*Server).processStreamingRPC
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1549
| google.golang.org/grpc.(*Server).handleStream
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:1624
| google.golang.org/grpc.(*Server).serveStreams.func1.2
| google.golang.org/grpc/external/org_golang_google_grpc/server.go:922
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:884: *withstack.withStack (top exception)

Stacktrace (expand for inline code snippets):

GOROOT/src/runtime/panic.go#L883-L885 in runtime.gopanic
GOROOT/src/runtime/panic.go#L259-L261 in runtime.panicmem
GOROOT/src/runtime/signal_unix.go#L834-L836 in runtime.sigpanic
google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go#L310-L312 in google.golang.org/grpc/internal/transport.(*Stream).compareAndSwapState
google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go#L953-L955 in google.golang.org/grpc/internal/transport.(*http2Client).Write
google.golang.org/grpc/external/org_golang_google_grpc/stream.go#L871-L873 in google.golang.org/grpc.(*clientStream).CloseSend.func1
google.golang.org/grpc/external/org_golang_google_grpc/stream.go#L687-L689 in google.golang.org/grpc.(*clientStream).withRetry
google.golang.org/grpc/external/org_golang_google_grpc/stream.go#L878-L880 in google.golang.org/grpc.(*clientStream).CloseSend

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 397 to 399 in 07a53a3

func (cs *tracingClientStream) CloseSend() error {
err := cs.ClientStream.CloseSend()
if err != nil {
in pkg/util/tracing/grpcinterceptor.(*tracingClientStream).CloseSend
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L639-L641 in pkg/kv/kvserver.(*RaftTransport).SendSnapshot.func1
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L643-L645 in pkg/kv/kvserver.(*RaftTransport).SendSnapshot
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go#L2884-L2886 in pkg/kv/kvserver.(*Replica).followerSendSnapshot.func5

cockroach/pkg/util/contextutil/context.go

Lines 90 to 92 in 07a53a3

start := timeutil.Now()
err := fn(ctx)
if err != nil && errors.Is(ctx.Err(), context.DeadlineExceeded) {
in pkg/util/contextutil.RunWithTimeout
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go#L2882-L2884 in pkg/kv/kvserver.(*Replica).followerSendSnapshot
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L179-L181 in pkg/kv/kvserver.(*Store).HandleDelegatedSnapshot
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go#L355-L357 in pkg/kv/kvserver.(*RaftTransport).DelegateRaftSnapshot
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/kv/kvserver/storage_services.pb.go#L295-L297 in pkg/kv/kvserver._MultiRaft_DelegateRaftSnapshot_Handler

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 162 to 164 in 07a53a3

}
err = handler(srv, ss)
if err != nil {
in pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1407-L1409 in google.golang.org/grpc.chainStreamInterceptors.func1.1
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L271-L273 in pkg/rpc.NewServerEx.func4
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1410-L1412 in google.golang.org/grpc.chainStreamInterceptors.func1.1
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L240-L242 in pkg/rpc.NewServerEx.func2.1

cockroach/pkg/util/stop/stopper.go

Lines 340 to 342 in 07a53a3

return f(ctx)
}
in pkg/util/stop.(*Stopper).RunTaskWithErr
https://github.com/cockroachdb/cockroach/blob/07a53a36601e9ca5fcffcff55f69b43c6dfbf1c1/pkg/rpc/pkg/rpc/context.go#L239-L241 in pkg/rpc.NewServerEx.func2
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1410-L1412 in google.golang.org/grpc.chainStreamInterceptors.func1.1
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1412-L1414 in google.golang.org/grpc.chainStreamInterceptors.func1
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1548-L1550 in google.golang.org/grpc.(*Server).processStreamingRPC
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L1623-L1625 in google.golang.org/grpc.(*Server).handleStream
google.golang.org/grpc/external/org_golang_google_grpc/server.go#L921-L923 in google.golang.org/grpc.(*Server).serveStreams.func1.2
GOROOT/src/runtime/asm_amd64.s#L1593-L1595 in runtime.goexit

GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 835
google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/transport.go in google.golang.org/grpc/internal/transport.(*Stream).compareAndSwapState at line 311
google.golang.org/grpc/internal/transport/external/org_golang_google_grpc/internal/transport/http2_client.go in google.golang.org/grpc/internal/transport.(*http2Client).Write at line 954
google.golang.org/grpc/external/org_golang_google_grpc/stream.go in google.golang.org/grpc.(*clientStream).CloseSend.func1 at line 872
google.golang.org/grpc/external/org_golang_google_grpc/stream.go in google.golang.org/grpc.(*clientStream).withRetry at line 688
google.golang.org/grpc/external/org_golang_google_grpc/stream.go in google.golang.org/grpc.(*clientStream).CloseSend at line 879
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.(*tracingClientStream).CloseSend at line 398
pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).SendSnapshot.func1 at line 640
pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).SendSnapshot at line 644
pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go in pkg/kv/kvserver.(*Replica).followerSendSnapshot.func5 at line 2885
pkg/util/contextutil/context.go in pkg/util/contextutil.RunWithTimeout at line 91
pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go in pkg/kv/kvserver.(*Replica).followerSendSnapshot at line 2883
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).HandleDelegatedSnapshot at line 180
pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go in pkg/kv/kvserver.(*RaftTransport).DelegateRaftSnapshot at line 356
pkg/kv/kvserver/storage_services.pb.go in pkg/kv/kvserver._MultiRaft_DelegateRaftSnapshot_Handler at line 296
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.StreamServerInterceptor.func1 at line 163
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainStreamInterceptors.func1.1 at line 1408
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func4 at line 272
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainStreamInterceptors.func1.1 at line 1411
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2.1 at line 241
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 341
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func2 at line 240
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainStreamInterceptors.func1.1 at line 1411
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.chainStreamInterceptors.func1 at line 1413
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).processStreamingRPC at line 1549
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).handleStream at line 1624
google.golang.org/grpc/external/org_golang_google_grpc/server.go in google.golang.org/grpc.(*Server).serveStreams.func1.2 at line 922
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
Tag Value
Cockroach Release v22.2.2
Cockroach SHA: 07a53a3
Platform linux amd64
Distribution CCL
Environment v22.2.2
Command server
Go Version ``
# of CPUs
# of Goroutines

Jira issue: CRDB-24757
@cockroach-teamcity cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Feb 23, 2023
@yuzefovich
Copy link
Member

dup of #81227

@exalate-issue-sync exalate-issue-sync bot changed the title sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | runtime.panicmem ... sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | runtime.panicmem ... Feb 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yuzefovich @cockroach-teamcity