If WaitForMastership fails log_signer sits around doing nothing

[rolandshoemaker]

If the call to WaitForMastership fails in electionRunner.Run it returns from the function and sits around doing nothing for that log (i.e. masterFor calls will always return empty and nothing can be done). This can be triggered by recoverable intermittent etcd issues.

The return should either be replaced with continue (perhaps with a backoff?) when hitting errors from WaitforMastership or the electionRunner should be removed from l.electionRunner[logID] when it exits so a new election goroutine can be created the next time masterFor is called. I'd probably go for the first of the two options.

[pav-kv]

Hi Roland, thanks for filing this. I am working right now on factoring out the election runner, and also noticed this issue. I have a work-in-progress PR that largely interferes with this code. Would you like to review that?

[rolandshoemaker]

Ah cool, I'll take a look, thanks!

[Martin2112]

Did #1144 fix your issue?

[pav-kv]

No, that was a refactoring, and it ended up postponed.

[pav-kv]

This is effectively fixed now. Election code has migrated to util/election (types Runner and MasterTracker). If election fails at any step, it will be attempted again.

[pav-kv]

No, actually the problem seems to be still there:

trillian/log/operation_manager.go

Line 242 in 85d73fd

r.Run(innerCtx, o.pendingResignations)