agent: check for existing machine-ids on startup

[jonboulle]

If multiple systems in a cluster happen to have the same machine-id, all hell can break loose. It would be nice if the agent could be able to do a simple sanity check when first registering itself in the cluster; if there's an existing machine with its machine-id, it can log something useful and/or back off and try again (e.g. in case a previously-pushed machineState just hasn't TTLed yet)

[bcwaldon]

What situations can cause this to happen?

[jonboulle]

Cloned VMs (come up a few times on the mailing list).
Using KVM (until the very newest version of systemd, which we're not using yet).
Someone trying to run multiple instances of fleet on a single machine.

[ibuildthecloud]

+1

[bcwaldon]

If we do this, I believe we would lose the ability for an agent to go down with SIGKILL (i.e. no state purge) and come back up without any service interruption. If we make an agent wait on its state key to time out before it can rejoin, that will cause unnecessary churn in the schedule. Once we switch to a more direct presence mechanism (direct API requests) this change won't be as disruptive.

[carmstrong]

Ah, I thought #697 fixed this, but it looks like it didn't. I must've misread the release notes.

[carmstrong]

(The whole 'accidentally running fleet instead of fleetctl' issue)

[bcwaldon]

@carmstrong we should really just move fleet to fleetd.

[carmstrong]

we should really just move fleet to fleetd.

👍

[ibuildthecloud]

+1 renaming to fleetd

[jonboulle]

#807

[steveej]

I strongly vote for including a sanity check to ensure unique machine-ids. I came across this due to cloned VMs and spent hours of debugging until @jonboulle pointed me in the right direction.

[jasonkeene]

+1 this was a head scratcher

[tixxdz]

@antrik could you take this one, and it seems there is a PR #1288