fleet ignoring manual unit state changes

[bcwaldon]

Create a unit foo.service with the following contents:

[Service]
ExecStart=/bin/sleep infinity

Start it with fleetctl:

$ fleetctl start foo.service
Job foo.service launched on 9181c53e.../172.17.8.101
$ fleetctl list-units
UNIT        STATE       LOAD    ACTIVE  SUB DESC    MACHINE
foo.service launched    loaded  active  running -   9181c53e.../172.17.8.101

Stop it manually with systemctl:

$ systemctl stop foo.service
$ systemctl status foo.service
● foo.service
   Loaded: loaded (/run/fleet/units/foo.service; linked-runtime)
   Active: inactive (dead)

Jul 08 18:20:17 core-01 systemd[1]: Starting foo.service...
Jul 08 18:20:17 core-01 systemd[1]: Started foo.service.
Jul 08 18:20:29 core-01 systemd[1]: Stopping foo.service...
Jul 08 18:20:29 core-01 systemd[1]: Stopped foo.service.

fleet doesn't get the message, however:

$ fleetctl list-units
UNIT        STATE       LOAD    ACTIVE  SUB DESC    MACHINE
foo.service launched    loaded  active  running -   9181c53e.../172.17.8.101

[jonboulle]

[bcwaldon]

@jonboulle I know...

[bcwaldon]

More interesting:

$ fleetctl load foo.service
Job foo.service loaded on 9181c53e.../172.17.8.101
$ fleetctl list-units
UNIT        STATE   LOAD    ACTIVE      SUB DESC    MACHINE
foo.service loaded  loaded  inactive    dead    -   9181c53e.../172.17.8.101
$ systemctl start foo.service
$ fleetctl list-units
UNIT        STATE   LOAD    ACTIVE  SUB DESC    MACHINE
foo.service loaded  loaded  active  running -   9181c53e.../172.17.8.101

Maybe this goes back to #352?

[bcwaldon]

This is because we don't react to nil UnitStatus objects coming out of dbus at all.

[jonboulle]

What do you mean "nil UnitStatus objects coming out of dbus"? What does that look like?

[bcwaldon]

@jonboulle https://github.com/coreos/fleet/blob/master/agent/event.go#L93

[jonboulle]

@bcwaldon I meant in terms of what we get from dbus/go-systemd itself - seems like a bug to me that the status channel ever returns nils

[bcwaldon]

@jonboulle nil does mean something, and we could totally react to it. We just don't. We could react by fetching the current state of the Unit and publishing that, which would actually fix this bug. We've gone back and forth on how we are managing unit state, but we've never really sat down and figured out the 100% correct solution. Let's talk in person.

[jonboulle]

Is this covered by #651 ?

[bcwaldon]

yes

[bcwaldon]

Fixed by #651