Could not deregister runner on new deployment

[gak]

On deployment, ftl-controller can not deregister runners (maybe from the previous deploy).

Jul 14 00:34:38.761 ftl-controller Streaming RPC failed: runner callback failed: unavailable: failed to connect to runner: context deadline exceeded: /xyz.block.ftl.v1.ControllerService/RegisterRunner
Jul 14 00:34:38.761 ftl-controller Could not deregister runner http://xxx: not found

[matt2e]

Looks like this is what's happening:

• Runner starts, starts stream to controller
• Controller receives first message of stream and tries to ping the runner https://github.com/TBD54566975/ftl/blob/main/backend/controller/controller.go#L572
  • Controller fails to ping runner, retries for 10s, then gives up (by cancelling context): https://github.com/TBD54566975/ftl/blob/main/backend/controller/controller.go#L612
• Controller tries to deregister the runner https://github.com/TBD54566975/ftl/blob/main/backend/controller/controller.go#L565
  • This fails with Could not deregister runner http://xxx: not found because we have not yet reached the code that upserts the runner https://github.com/TBD54566975/ftl/blob/main/backend/controller/controller.go#L581