ccl/streamingccl/streamingest: TestTenantStreamingMultipleNodes failed

[cockroach-teamcity]

ccl/streamingccl/streamingest.TestTenantStreamingMultipleNodes failed with artifacts on release-23.1 @ aaedb3dd0d97097cff7afeb2532c6265d3d62818:

          | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
          | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
          | runtime.goexit
          | 	GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) server stop before successful start
        Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/90c1b75d835f45b8488807abb5b1092d/logTestTenantStreamingMultipleNodes1968593793
I230727 07:17:43.176907 390411 server/server_controller_channel_orchestrator.go:284  [-] 2  propagate-close task terminating
I230727 07:17:43.392232 351227 server/server_controller_orchestration.go:127  [T1,n4] 3  tenant "source" has changed service mode, should now start
I230727 07:17:43.445535 347945 server/server_controller_orchestration.go:127  [T1,n3] 4  tenant "source" has changed service mode, should now start
I230727 07:17:43.448072 390691 server/server_controller_new_server.go:146  [n4] 5  creating tenant server
I230727 07:17:43.479808 390771 server/server_controller_new_server.go:146  [n3] 6  creating tenant server
I230727 07:17:43.496151 390691 3@pebble/event.go:689  [n4,s?,pebble] 7  [JOB 1] MANIFEST created 000001
I230727 07:17:43.502635 390691 3@pebble/event.go:717  [n4,s?,pebble] 8  [JOB 1] WAL created 000002
I230727 07:17:43.520624 390771 3@pebble/event.go:689  [n3,s?,pebble] 9  [JOB 1] MANIFEST created 000001
I230727 07:17:43.522978 390691 3@pebble/event.go:685  [n4,s?,pebble] 10  upgraded to format version: 002
I230727 07:17:43.523095 390771 3@pebble/event.go:717  [n3,s?,pebble] 11  [JOB 1] WAL created 000002
I230727 07:17:43.526047 390771 3@pebble/event.go:685  [n3,s?,pebble] 12  upgraded to format version: 002
I230727 07:17:43.527097 390691 3@pebble/event.go:685  [n4,s?,pebble] 13  upgraded to format version: 003
I230727 07:17:43.528390 390691 3@pebble/event.go:685  [n4,s?,pebble] 14  upgraded to format version: 004
I230727 07:17:43.529482 390691 3@pebble/event.go:685  [n4,s?,pebble] 15  upgraded to format version: 005
I230727 07:17:43.530569 390691 3@pebble/event.go:685  [n4,s?,pebble] 16  upgraded to format version: 006
I230727 07:17:43.531690 390691 3@pebble/event.go:685  [n4,s?,pebble] 17  upgraded to format version: 007
I230727 07:17:43.532776 390771 3@pebble/event.go:685  [n3,s?,pebble] 18  upgraded to format version: 003
I230727 07:17:43.533784 390691 3@pebble/event.go:685  [n4,s?,pebble] 19  upgraded to format version: 008
I230727 07:17:43.534863 390771 3@pebble/event.go:685  [n3,s?,pebble] 20  upgraded to format version: 004
I230727 07:17:43.536108 390691 3@pebble/event.go:685  [n4,s?,pebble] 21  upgraded to format version: 009
I230727 07:17:43.537627 390771 3@pebble/event.go:685  [n3,s?,pebble] 22  upgraded to format version: 005
I230727 07:17:43.538394 390691 3@pebble/event.go:685  [n4,s?,pebble] 23  upgraded to format version: 010
I230727 07:17:43.564216 390771 3@pebble/event.go:685  [n3,s?,pebble] 24  upgraded to format version: 006
I230727 07:17:43.565846 390771 3@pebble/event.go:685  [n3,s?,pebble] 25  upgraded to format version: 007
I230727 07:17:43.568010 390771 3@pebble/event.go:685  [n3,s?,pebble] 26  upgraded to format version: 008
I230727 07:17:43.570171 390771 3@pebble/event.go:685  [n3,s?,pebble] 27  upgraded to format version: 009
I230727 07:17:43.572336 390771 3@pebble/event.go:685  [n3,s?,pebble] 28  upgraded to format version: 010
I230727 07:17:43.574340 390911 3@pebble/event.go:709  [n4,stemp,pebble] 29  [JOB 4] all initial table stats loaded
I230727 07:17:43.581862 390913 3@pebble/event.go:709  [n3,s?,pebble] 30  [JOB 4] all initial table stats loaded
I230727 07:17:43.703412 390691 server/server_controller_channel_orchestrator.go:365  [T10,n4,tenant-orchestration,tenant=source,start-server] 31  starting tenant server
I230727 07:17:43.704040 390691 1@server/tenant.go:685  [T10,n4,tenant-orchestration,tenant=source,start-server] 32  starting https server at 127.0.0.1:34787 (use: 127.0.0.1:34787)
I230727 07:17:43.704184 390691 1@server/tenant.go:692  [T10,n4,tenant-orchestration,tenant=source,start-server] 33  starting postgres server at 127.0.0.1:38439 (use: 127.0.0.1:38439)
I230727 07:17:43.704279 390691 1@server/tenant.go:696  [T10,n4,tenant-orchestration,tenant=source,start-server] 34  starting grpc server at 127.0.0.1:43899
I230727 07:17:43.704356 390691 1@server/tenant.go:697  [T10,n4,tenant-orchestration,tenant=source,start-server] 35  advertising SQL server node at 127.0.0.1:43899
I230727 07:17:43.739618 391222 2@rpc/context.go:2581  [T10,n4,rnode=?,raddr=127.0.0.1:42695,class=system,rpc] 36  connection is now ready
I230727 07:17:43.757027 391220 kv/kvclient/kvtenant/setting_overrides.go:68  [T10,n4,tenant-connector] 37  received initial tenant settings
I230727 07:17:43.757239 390691 kv/kvclient/kvtenant/connector.go:313  [T10,n4] 38  kv connector tenant settings started
I230727 07:17:43.758581 390691 kv/kvclient/kvtenant/connector.go:310  [T10,n4] 39  kv connector gossip subscription started
I230727 07:17:43.758643 390691 server/settingswatcher/settings_watcher.go:379  [T10,n4] 40  updating storage cluster cached version from: 0.0 to: 23.1
I230727 07:17:43.758761 391338 kv/kvclient/rangefeed/rangefeedcache/watcher.go:335  [T10,n4] 41  settings-watcher: established range feed cache
I230727 07:17:43.758979 391337 server/settingswatcher/settings_watcher.go:379  [T10,n4] 42  updating storage cluster cached version from: 23.1 to: 23.1
--- FAIL: TestTenantStreamingMultipleNodes (22.89s)

Parameters: TAGS=bazel,gss

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-30159

[adityamaru]

Should be fixed by #107666.

[lidorcarmel]

should be fixed by #107666