Failed to heartbeat controller

[matt2e]

failed to heartbeat controller: duplicate key value violates unique constraint "controller_endpoint_not_dead_idx": conflict

Might be related to this one: #2045

[matt2e]

Assigned it to me because I think the next steps are just seeing if other fixes (like controller not panicing when getting an error back from calling a verb) make this go away.

[matt2e]

Havent seen this error the past few times we restarted the controllers, closing for now

[matt2e]

This might be back...

[wesbillman]

Once we have a k8s env setup for local debugging, we should did into this again.

[matt2e]

Looks like this is the cause:

• Controller is alive
• We take down the controller
• We bring up another controller. The controller ends up having the same endpoint as the original controller
• New controller tries to heartbeat the db
  • This fails because there is already a controller row with state = 'live', a matching endpoint, and a different controller key
• A controller runs the reapStaleControllers job which finds live controller in the db which haven't been seen in the past 10s. Maybe the first time this happens we are still in the 10s window. But it shouldn't take long for one of these jobs to find and mark the old controller's row as dead
• New controller keeps trying to heartbeat, and will eventually succeed in upserting its row into the db.

You can repro locally by doing this:

• Commenting out code in the if !recraete block: https://github.com/TBD54566975/ftl/blob/main/backend/controller/sql/databasetesting/devel.go#L76
• Making the integer used for the controller's key random like it is in non-local environments: https://github.com/TBD54566975/ftl/blob/main/cmd/ftl/cmd_serve.go#L122

When I did this, the first heartbeat failed like in the issue description, then the staler controller was reaped, then the next controller heartbeat worked. Seems like everything recovered fine.