Monitor heartbeat attempt can time out, but succeed

[bcwaldon]

An attempt to heartbeat machine presence can succeed, but timeout client-side. This causes all subsequent attempts to fail as the presence is created using prevExist=false.

Aug 06 06:43:21 node-02 fleet[668]: I0806 06:43:21.360308 00668 monitor.go:54] Monitor heartbeat function returned err, retrying in 2.5s: timeout reached
…
Aug 06 06:43:25 node-02 fleet[668]: I0806 06:43:25.354149 00668 monitor.go:54] Monitor heartbeat function returned err, retrying in 2.5s: 105: Key already exists (/_coreos.com/fleet/machines/00b7e1bbb2464ed9a4ef4282b11f2161/object)

[jonboulle]

Somewhat related: #615

[wuqixuan]

@jonboulle @bcwaldon It's not totally same as #615.
There are 3 cases should be taken care.

1. prevent other machine with the same id (agent: check for existing machine-ids on startup #615)
2. allow the existing same id when you second time heartbeat ( Monitor heartbeat attempt can time out, but succeed #750)
3. prevent another daemon in the same host.

[kayrus]

Does anyone know how to reproduce this issue?

[dongsupark]

1. prevent other machine with the same id (agent: check for existing machine-ids on startup #615)
2. allow the existing same id when you second time heartbeat ( Monitor heartbeat attempt can time out, but succeed #750)
3. prevent another daemon in the same host.

The 1st one, preventing other machines from being registered with the same ID, was addressed by #1561.
I'm not sure about what we can do about the 3rd one, preventing another daemon in the same host.

Let me get the 2nd one straight, "allowing the existing same id when doing heartbeat for the second time". AFAIK no PR addressed this issue.
Could it be simply done, if we would do s.hrt.Register() only for the 1st attempt, and do s.hrt.Beat() starting from the 2nd attempt? Then this issue could be relatively simply addressed.

Though I'm not sure if this is supposed to be done inside the milestone v0.13.

[antrik]

At first I thought 3) would be addressed by the fix for 1) too -- but I realised now that it is indeed a separate issue: if you start two fleetds on the same machine, they will be "fighting" for the machine ID... Not sure this is really a problem that needs to be fixed though -- at least not an urgent one.

Anyway, this issue here is clearly about case 2.

[dongsupark]

1. prevent other machine with the same id (agent: check for existing machine-ids on startup #615)
2. allow the existing same id when you second time heartbeat ( Monitor heartbeat attempt can time out, but succeed #750)
3. prevent another daemon in the same host.

The 1st one is already fixed by #1561.
As for the 2nd issue, originally I thought I could fix it via #1563, but it was not the case. It actually caused other side effects. So I had to come up with another solution, which was already merged to #1561.
I don't know about the 3rd issue. At least I don't think it should be fixed with this issue.

So I think this issue can be closed.

[tixxdz]

Closing this one.

Thanks all!