Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix NPE in GCP archival #4626

Merged
merged 2 commits into from
Nov 10, 2021
Merged

Fix NPE in GCP archival #4626

merged 2 commits into from
Nov 10, 2021

Conversation

longquanzheng
Copy link
Contributor

What changed?
Fix NPE in GCP archival

Why?

error:  "runtime error: invalid memory address or nil pointer dereference"    
  level:  "error"    
  logging-call-at:  "panic.go:45"    
  msg:  "Panic is captured"    
  service:  "cadence-frontend"    
  stacktrace:  "github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error
	/cadence/common/log/loggerimpl/logger.go:131
github.com/uber/cadence/common/log.CapturePanic
	/cadence/common/log/panic.go:45
runtime.gopanic
	/usr/local/go/src/runtime/panic.go:679
runtime.panicmem
	/usr/local/go/src/runtime/panic.go:199
runtime.sigpanic
	/usr/local/go/src/runtime/signal_unix.go:394
github.com/uber/cadence/common/archiver/gcloud.(*historyArchiver).Get
	/cadence/common/archiver/gcloud/historyArchiver.go:227
github.com/uber/cadence/service/frontend.(*WorkflowHandler).getArchivedHistory
	/cadence/service/frontend/workflowHandler.go:4039
github.com/uber/cadence/service/frontend.(*WorkflowHandler).GetWorkflowExecutionHistory
	/cadence/service/frontend/workflowHandler.go:1996
github.com/uber/cadence/service/frontend.(*DCRedirectionHandlerImpl).GetWorkflowExecutionHistory.func2
	/cadence/service/frontend/dcRedirectionHandler.go:240
github.com/uber/cadence/service/frontend.(*NoopRedirectionPolicy).WithDomainNameRedirect
	/cadence/service/frontend/dcRedirectionPolicy.go:115
github.com/uber/cadence/service/frontend.(*DCRedirectionHandlerImpl).GetWorkflowExecutionHistory
	/cadence/service/frontend/dcRedirectionHandler.go:236
github.com/uber/cadence/service/frontend.(*AccessControlledWorkflowHandler).GetWorkflowExecutionHistory
	/cadence/service/frontend/accessControlledHandler.go:204
github.com/uber/cadence/service/frontend.ThriftHandler.GetWorkflowExecutionHistory
	/cadence/service/frontend/thriftHandler.go:103
github.com/uber/cadence/.gen/go/cadence/workflowserviceserver.handler.GetWorkflowExecutionHistory
	/cadence/.gen/go/cadence/workflowserviceserver/server.go:890
go.uber.org/yarpc/encoding/thrift.thriftUnaryHandler.Handle
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/encoding/thrift/inbound.go:64
go.uber.org/yarpc/internal/observability.(*Middleware).Handle
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/internal/observability/middleware.go:166
go.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/api/middleware/inbound.go:71
go.uber.org/yarpc/api/transport.InvokeUnaryHandler
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/api/transport/handler_invoker.go:70
go.uber.org/yarpc/transport/tchannel.handler.callHandler
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:233
go.uber.org/yarpc/transport/tchannel.handler.handle
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:121
go.uber.org/yarpc/transport/tchannel.handler.Handle
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:110
github.com/uber/tchannel-go.channelHandler.Handle
	/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/handlers.go:126
github.com/uber/tchannel-go.(*Connection).dispatchInbound
	/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/inbound.go:203"    
  sys-stack-trace:  "goroutine 434834 [running]:
runtime/debug.Stack(0x2888b80, 0x28688a0, 0x4b0c8f0)
	/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
github.com/uber/cadence/common/log.CapturePanic(0x341c5c0, 0xc00032f2e0, 0xc0005c25f0)
	/cadence/common/log/panic.go:43 +0x8f
panic(0x28688a0, 0x4b0c8f0)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/uber/cadence/common/archiver/gcloud.(*historyArchiver).Get(0xc001f3fb30, 0x3405040, 0xc001d04330, 0x342c280, 0xc001e36138, 0xc0017444e0, 0xc001f3fb30, 0x0, 0x0)
	/cadence/common/archiver/gcloud/historyArchiver.go:227 +0x97c
github.com/uber/cadence/service/frontend.(*WorkflowHandler).getArchivedHistory(0xc00004c770, 0x3405040, 0xc001d04330, 0xc0021060f0, 0xc001468c60, 0x24, 0x3429640, 0xc001d04a50, 0xc05ad56f1b880401, 0x244bae71a41, ...)
	/cadence/service/frontend/workflowHandler.go:4039 +0x113d
github.com/uber/cadence/service/frontend.(*WorkflowHandler).GetWorkflowExecutionHistory(0xc00004c770, 0x3405040, 0xc001d04330, 0xc0021060f0, 0x0, 0x0, 0x0)
	/cadence/service/frontend/workflowHandler.go:1996 +0x4ad9
github.com/uber/cadence/service/frontend.(*DCRedirectionHandlerImpl).GetWorkflowExecutionHistory.func2(0xc000059b00, 0x6, 0x3429601, 0xc002006180)
	/cadence/service/frontend/dcRedirectionHandler.go:240 +0x1db
github.com/uber/cadence/service/frontend.(*NoopRedirectionPolicy).WithDomainNameRedirect(0xc0004ee8c0, 0x3405040, 0xc001d04330, 0xc0015e20c0, 0x11, 0x2d03ce1, 0x1b, 0xc002006180, 0x29bbce0, 0xc001d043f0)
	/cadence/service/frontend/dcRedirectionPolicy.go:115 +0x3c
github.com/uber/cadence/service/frontend.(*DCRedirectionHandlerImpl).GetWorkflowExecutionHistory(0xc000a06b90, 0x3405040, 0xc001d04330, 0xc0021060f0, 0x3429640, 0x0, 0x0)
	/cadence/service/frontend/dcRedirectionHandler.go:236 +0x254
github.com/uber/cadence/service/frontend.(*AccessControlledWorkflowHandler).GetWorkflowExecutionHistory(0xc000d8a990, 0x3405040, 0xc001d04330, 0xc0021060f0, 0x3405040, 0xc001d04330, 0x0)
	/cadence/service/frontend/accessControlledHandler.go:204 +0x17a
github.com/uber/cadence/service/frontend.ThriftHandler.GetWorkflowExecutionHistory(0x3449ba0, 0xc000d8a990, 0x3405040, 0xc001d04300, 0xc0021060a0, 0x2, 0x0, 0x0)
	/cadence/service/frontend/thriftHandler.go:103 +0x11c
github.com/uber/cadence/.gen/go/cadence/workflowserviceserver.handler.GetWorkflowExecutionHistory(0x3447aa0, 0xc0004ee8d0, 0x3405040, 0xc001d04300, 0xc, 0x0, 0x0, 0x0, 0x0, 0xc000aa4900, ...)
	/cadence/.gen/go/cadence/workflowserviceserver/server.go:890 +0x1ad
go.uber.org/yarpc/encoding/thrift.thriftUnaryHandler.Handle(0xc0007ad040, 0x3407e80, 0x4b5e5e8, 0x46eb00, 0x3405040, 0xc001d04270, 0xc001af4210, 0x33f6500, 0xc002121b60, 0x0, ...)
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/encoding/thrift/inbound.go:64 +0x215
go.uber.org/yarpc/internal/observability.(*Middleware).Handle(0xc00074ccc0, 0x3405040, 0xc001d04270, 0xc001af4210, 0x7fb5bec62948, 0xc000aa4780, 0x33a8660, 0xc0007ad5a0, 0x0, 0x0)
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/internal/observability/middleware.go:166 +0x1f1
go.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle(...)
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/api/middleware/inbound.go:71
go.uber.org/yarpc/api/transport.InvokeUnaryHandler(0x3405040, 0xc001d04270, 0xc05ad56f1b85c6a8, 0x244bae4dce0, 0x4b3b6c0, 0xc001af4210, 0x7fb5bec62948, 0xc000aa4780, 0x33a84c0, 0xc0007adce0, ...)
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/api/transport/handler_invoker.go:70 +0x108
go.uber.org/yarpc/transport/tchannel.handler.callHandler(0xc000d8b5f0, 0x7fb5bec62568, 0xc0005e96e0, 0x33f7bc0, 0x4b5e1c0, 0x0, 0xc00074ca80, 0x2de2d50, 0x3405040, 0xc001d04270, ...)
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:233 +0xac4
go.uber.org/yarpc/transport/tchannel.handler.handle(0xc000d8b5f0, 0x7fb5bec62568, 0xc0005e96e0, 0x33f7bc0, 0x4b5e1c0, 0x0, 0xc00074ca80, 0x2de2d50, 0x3407a00, 0xc001aac0b0, ...)
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:121 +0x1fe
go.uber.org/yarpc/transport/tchannel.handler.Handle(...)
	/go/pkg/mod/go.uber.org/yarpc@v1.52.1-0.20210303193224-b2caa40d56b6/transport/tchannel/handler.go:110
github.com/uber/tchannel-go.channelHandler.Handle(0xc000738e00, 0x3407a00, 0xc001aac0b0, 0xc00165e180)
	/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/handlers.go:126 +0x8c
github.com/uber/tchannel-go.(*Connection).dispatchInbound(0xc001f8e000, 0x600000090, 0xc00165e180, 0xc0021022a0)
	/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/inbound.go:203 +0x380
created by github.com/uber/tchannel-go.(*Connection).handleCallReq
	/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/inbound.go:125 +0xc97

How did you test it?
Eye ball.
It's the same code as in S3 archiver https://github.com/uber/cadence/blob/04cd354f8d80cda913e6d54932f8b4533e1fff48/common/archiver/s3store/historyArchiver.go#L407 but missing in GCP archiver

Potential risks
No

Release notes
Will release in 0.23.4 patch

Documentation Changes

@longquanzheng longquanzheng enabled auto-merge (squash) November 10, 2021 04:36
@longquanzheng longquanzheng merged commit 32123b9 into master Nov 10, 2021
@longquanzheng longquanzheng deleted the qlong-archival-npe branch November 10, 2021 05:08
@coveralls
Copy link

Pull Request Test Coverage Report for Build e79e152d-f900-453c-8590-68eb4c329ad2

  • 1 of 3 (33.33%) changed or added relevant lines in 1 file are covered.
  • 52 unchanged lines in 10 files lost coverage.
  • Overall coverage increased (+0.02%) to 56.912%

Changes Missing Coverage Covered Lines Changed/Added Lines %
common/archiver/gcloud/historyArchiver.go 1 3 33.33%
Files with Coverage Reduction New Missed Lines %
common/persistence/sql/sqlExecutionStore.go 1 62.69%
service/history/execution/mutable_state_task_refresher.go 1 73.82%
common/persistence/executionManager.go 2 76.57%
common/persistence/statsComputer.go 2 96.43%
common/task/weightedRoundRobinTaskScheduler.go 2 89.64%
common/cache/lru.go 3 90.73%
service/history/queue/timer_gate.go 3 95.83%
service/history/shard/context.go 9 66.22%
service/history/queue/transfer_queue_processor.go 12 57.24%
service/history/execution/context.go 17 74.69%
Totals Coverage Status
Change from base Build 7e177af3-6aad-4f85-a354-8788060ff8f6: 0.02%
Covered Lines: 82365
Relevant Lines: 144723

💛 - Coveralls

longquanzheng added a commit that referenced this pull request Nov 11, 2021
longquanzheng added a commit that referenced this pull request Nov 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants