Controller crashes when edge router reconnects (Client Hello)

[arunsworld]

Notes:

• I'm using binary compiled from the fix-xctrl-startup-race branch (though I suspect this issue is applicable across versions)
• Edge router running on my Mac laptop; controller running in the cloud (both same version)
• Not reproducible on demand but has happened several times now to indicate a pattern

Stack trace from controller:

{"file":"/go/src/github.com/openziti/ziti/controller/handler_ctrl/accept.go:93","func":"github.com/openziti/ziti/controller/handler_ctrl.(*CtrlAccepter).Bind","level":"warning","msg":"no advertised listeners","routerId":"5Wd7iovsP","time":"2023-10-11T04:49:56.586Z"}
{"file":"/go/src/github.com/openziti/ziti/controller/handler_ctrl/accept.go:117","func":"github.com/openziti/ziti/controller/handler_ctrl.(*CtrlAccepter).Bind","level":"info","msg":"accepted new router connection [r/5Wd7iovsP]","routerId":"5Wd7iovsP","time":"2023-10-11T04:49:56.587Z"}
{"file":"/go/src/github.com/openziti/ziti/controller/network/router_messaging.go:192","func":"github.com/openziti/ziti/controller/network.(*routerChangedEvent).handle","level":"info","msg":"calculating router updates for router 5Wd7iovsP, connected=true","time":"2023-10-11T04:49:56.587Z"}
{"file":"/go/src/github.com/openziti/ziti/controller/env/broker.go:104","func":"github.com/openziti/ziti/controller/env.(*Broker).RouterConnected.func1","level":"info","msg":"broker detected edge router with id 5Wd7iovsP connecting","routerFingerprint":"e69a130b27690e6b82ff5fdea9e5a55bff36573c","routerId":"5Wd7iovsP","routerName":"arun-mac-router","time":"2023-10-11T04:49:56.587Z"}
{"file":"/go/src/github.com/openziti/ziti/controller/sync_strats/sync_instant.go:185","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).RouterConnected","level":"info","msg":"edge router connected, adding to sync routerConnectedQueue","routerFingerprint":"e69a130b27690e6b82ff5fdea9e5a55bff36573c","routerId":"5Wd7iovsP","routerName":"arun-mac-router","syncStatus":"SYNC_QUEUED","sync_strategy":"instant","time":"2023-10-11T04:49:56.587Z"}
{"file":"/go/src/github.com/openziti/ziti/controller/sync_strats/sync_instant.go:366","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).hello","level":"info","msg":"edge router sync starting","routerChannelIsOpen":true,"routerFingerprint":"e69a130b27690e6b82ff5fdea9e5a55bff36573c","routerId":"5Wd7iovsP","routerName":"arun-mac-router","routerTxId":"gIlfL6X1t","strategy":"instant","time":"2023-10-11T04:49:56.587Z"}
{"file":"/go/src/github.com/openziti/ziti/controller/sync_strats/sync_instant.go:375","func":"github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).hello","level":"info","msg":"sending edge router hello","routerChannelIsOpen":true,"routerFingerprint":"e69a130b27690e6b82ff5fdea9e5a55bff36573c","routerId":"5Wd7iovsP","routerName":"arun-mac-router","routerTxId":"gIlfL6X1t","strategy":"instant","syncStatus":"SYNC_HELLO","time":"2023-10-11T04:49:56.587Z"}
{"file":"/go/src/github.com/openziti/ziti/controller/network/network.go:873","func":"github.com/openziti/ziti/controller/network.(*Network).Run","level":"info","msg":"changed router","routerId":"5Wd7iovsP","time":"2023-10-11T04:49:56.837Z"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1cc4311]

goroutine 3500394 [running]:
github.com/openziti/channel/v2.(*channelImpl).rxer.func1()
        /go/pkg/mod/github.com/openziti/channel/v2@v2.0.99/impl.go:308 +0x3f
panic({0x2113340?, 0x4196650?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/openziti/ziti/controller/sync_strats.(*InstantStrategy).ReceiveClientHello(0xc001a3f380, 0xc002810e10, 0xc003d0a000)
        /go/src/github.com/openziti/ziti/controller/sync_strats/sync_instant.go:500 +0x1051
github.com/openziti/ziti/controller/handler_edge_ctrl.(*helloHandler).HandleReceive(0xc001230170, 0xc001600330, {0x2bc82e0, 0xc001d28c00})
        /go/src/github.com/openziti/ziti/controller/handler_edge_ctrl/hello.go:58 +0x4a8
github.com/openziti/channel/v2.(*channelImpl).rxer(0xc00028a9c0?)
        /go/pkg/mod/github.com/openziti/channel/v2@v2.0.99/impl.go:357 +0x571
created by github.com/openziti/channel/v2.(*channelImpl).startMultiplex in goroutine 68
        /go/pkg/mod/github.com/openziti/channel/v2@v2.0.99/impl.go:288 +0xc7

[plorenz]

Hi @arunsworld
I pushed a fix to the fix-router-connect-panic branch. I suspect a race condition on brief connections. I also merged the fix for fix-xctrl-startup-race to release-next, so this branch includes that fix. I've not seen this error before, so if you get a chance to see if the change addresses your problem, that would be much appreciated.

[arunsworld]

Thanks - I've deployed it. Will need a few days of longevity test to confirm if the issue recurs. But appreciate your looking at these issues so promptly. Thanks,