Fleet starts units before required ones are loaded after a reboot

[digital-wonderland]

I'm trying to run an Elasticsearch cluster with fleet. Each elasticsearch@.service has -data & -discovery sidekicks on which the main unit has a Requires=.

To get it up and running I first load all necessary units and then start the elasticsearch@.service units which in turn starts their respective sidekicks.

This works nicely except after a reboot when fleet tries to start the main service before its sidekicks are loaded:

Oct 24 21:32:51 core-02 fleetd[591]: INFO server.go:148: Starting server components
Oct 24 21:32:54 core-02 fleetd[591]: INFO manager.go:218: Writing systemd unit elasticsearch@3.service (1343b)
Oct 24 21:32:54 core-02 fleetd[591]: INFO manager.go:142: Instructing systemd to reload units
Oct 24 21:32:54 core-02 fleetd[591]: INFO reconcile.go:274: AgentReconciler completed task: type=LoadUnit job=elasticsearch@3.service reason="unit scheduled here but not loaded"
Oct 24 21:32:54 core-02 fleetd[591]: ERROR manager.go:80: Failed to trigger systemd unit elasticsearch@3.service start: Unit elasticsearch-data@3.service failed to load: No such file or directory.
Oct 24 21:32:54 core-02 fleetd[591]: INFO reconcile.go:274: AgentReconciler completed task: type=StartUnit job=elasticsearch@3.service reason="unit currently loaded but desired state is launched"
Oct 24 21:32:59 core-02 fleetd[591]: INFO manager.go:218: Writing systemd unit elasticsearch-data@3.service (370b)
Oct 24 21:32:59 core-02 fleetd[591]: INFO manager.go:142: Instructing systemd to reload units
Oct 24 21:32:59 core-02 fleetd[591]: INFO manager.go:218: Writing systemd unit elasticsearch-discovery@3.service (515b)
Oct 24 21:32:59 core-02 fleetd[591]: INFO reconcile.go:274: AgentReconciler completed task: type=LoadUnit job=elasticsearch-data@3.service reason="unit scheduled here but not loaded"
Oct 24 21:32:59 core-02 fleetd[591]: INFO manager.go:142: Instructing systemd to reload units
Oct 24 21:32:59 core-02 fleetd[591]: INFO reconcile.go:274: AgentReconciler completed task: type=LoadUnit job=elasticsearch-discovery@3.service reason="unit scheduled here but not loaded"

To reproduce this follow the instructions from here. After the cluster is up restart one of the hosts.

My uneducated guess would be that deployment should first load all units, which are scheduled for a host, before trying to start one of them.

This also seems to be similar to #997 except that here sidekicks get only loaded but not started (since the Requires= in the main unit takes care of that).

[bcwaldon]

Please see #1134