Agent.Initialize causes major race conditions on startup

[bcwaldon]

When fleet starts, it attempts to unload any units locally that it knows it started in a past life, but are now scheduled elsewhere [0]. As soon as fleet kicks off the unload routine for that unit, it subscribes to events. If the unit that fleet just started unloading is then scheduled back to that machine, fleet will start it [1]. These two routines are now racing to finish first. If the unload finishes first, everything is ok. If the unload doesn't finish before the unit is scheduled, you'll get into this situation:

$ fleetctl list-units
UNIT                STATE       LOAD        ACTIVE      SUB     DESC    MACHINE
foo.service         launched    not-found   inactive    failed  -       af6ffd86.../192.0.2.32

The unit clearly has state published for it, but it's "not-found". This is because the unload routine finished after the unit was started, and destroyed the unit file on disk.

[0]

fleet/agent/agent.go

Line 111 in 733fa11

go func() {

[1]

fleet/agent/agent.go

Line 111 in 733fa11

go func() {

[jonboulle]

What do you propose here; queuing actions related to a specific unit?

[bcwaldon]

@jonboulle at a minimum, we should reconsider the use of "replace" when calling dbus methods.

[bcwaldon]

Agent.initialize no longer exists!

[jonboulle]

Not to burst your bubble pal, but Reconcile is basically just Initialize by another name, so I'd say this ticket still applies. But if you want to consider it addressed by #646 that is probably sufficient.

[bcwaldon]

That's what I was implying. Reconcile is different than Initialize, however, as it does not run while events are also mutating state. #646 covers the whole issue at this point