Integral support for Sidekick units

[yaronr]

Hi

Whether it's to support discovery, or something else - it seems that having 'sidekick' services is a very prevalent use case.

Although sidekick units 'work' - they quickly become a problem as you scale your system and number of unit files. In my case, many of my units have not one, but two 'sidekicks'.

This is a problem because

1. It increases the amount of work needed to make something work
2. It increases the possibility of something breaking - and the amount of scripting that needs to be maintained
3. It is error prone. If I have unit X@, and unit X-sidekick@, I have to remember to manually start X-sidekick@ and destroy it (it's not a global unit, and I don't know in advance how many I'll have)
4. problem --> nightmare, when a lot of this has to happen within 'bash -c' or within a script, which has to be stored, versioned, and maintained elsewhere, and be fetched and updated by the unit

I'm afraid I don't have a complete solution to offer; I have a few thoughts I will list below, but more importantly - I would like to start a discussion around this topic, because I think it's important and could make using CoreOS a whole lot easier.

Ideas:

1. Have fleet automatically start/destroy sidekicks. When I start X, fleet will start X-sidekick (all of them)
2. Add Sidekick (or watchdog?) functionality to units as an integral part
3. Create a super-duper master sidekick that will do various sidekicking for any unit that wants it

One of the goals of this should be making unit files easier to read and understand, therefore (3) is less ideal, since some of the knowledge of what a process does - is externalised. (PS, this is also one of the biggest problems with unit files executing scripts that are maintained 'somewhere')

[dbason]

The guys I work with are going to be looking at idea 3 at some point. We're already using https://github.com/digital-wonderland/docker-logstash-forwarder for logging which listens to the docker socket so I'm guessing you could do something similar to watch for starting containers and populate etcd from there.

[yaronr]

Hi @dbason
I am certainly aware that this option exists and that people have used it (also progrium/registrator, and others).

What I was suggesting was to include this as an integral part of the framework, since it is very common, it will greatly improve the user experience / usability of CoreOS, and the effort should be minimal (at least for people who program in Go. I'm a Java guy)

[leonidlm]

I love the concept of pods in Kubernetes.

It is a convenient way of bundling multiple containers which will be scheduled together into one unit.
A higher level concept like that, which will start the underlying containers based on their definition order (for example) may ease the management pain @yaronr mentioned (with which I definitely agree).

I think it is very important to be able to say "start elasticsearch" instead of the current situation when we need to say "start elasticsearch and elasticsearch-discovery and elasticsearch-monitoring and ..."

[yaronr]

..And he named them Service Groups, for they were Groups of Services. :)

[leonidlm]

How about adding an addition fleet specific unit file option:

[X-Fleet]
Groupby=elasticsearch

This way, we can reuse the already existing systemd dependency definitions and just create a convenient way of grouping services together on a cluster level.

Fleet could support all the basic unit operations on the group level:

fleetctl start/stop elasticsearch@1.group
fleetctl start/stop elasticsearch@{1,2,3}.group

Another option would be to introduce a tagging mechanism, that will allow to tag each unit file, and then to ask fleetctl to perform a start/stop based on the units selected by a tag.

Something like:

fleetctl start/stop -tag elasticsearch-1

@yaronr @dbason @bcwaldon what do you think ?

[yaronr]

Hey @leonidlm
I think it's definitely in the right direction, but I'm not sure I fully understand.
Say, you have:
elasticsearch@.service
elasticsearch-sidekick@.service
elasticsearch-discovery@.service

How would you start/stop according to your syntax? and how does the tagging work?

I actually think that the information is already there in the unit file, because all 3 units are 'partOf' and 'wants' etc. so all it takes is for fleet to parse the unit files in a folder and start/stop the dependent units

[bcwaldon]

As I understand it, you've basically got a set of linked processes. You want them to come up and go down together, and you don't want to have to manage them as independent entities. The easy answer here is to use something that already has these features (kubernetes, rocket). Both kubernetes and rocket have models for grouping independent processes together into single entities. If you're still interested in doing this in fleet, then you should head over to #1055 and make sure this use-case is covered.

[jonboulle]

@leonidlm I think the behaviour you described in #1052 (comment) is essentially described by (or at least overlaps with) #464 - perhaps you could chime in there.