Some form of "standard library"?

[08d2]

I'm using makesure as a sort of ersatz Chef/Puppet/Ansible. Each Makesurefile defines how to install, start, and uninstall a versioned application on a host. Each of these high-level outcomes usually can be achieved with a small set of low-level primitives: a goal to fetch an artifact from a remote source, create a user,install a file, make a symlink, basic interactions with systemd, etc. So far, I've just been copying and pasting these goals between Makesurefiles, modifying the specific parameters directly. But it's now become enough repetition, and enough subtlety and complexity in each primitive goal, that copying and pasting isn't really working any more.

The core requirement here is, given N Makesurefiles with broadly similar structure and goals, to be able reduce each of those files to just the "important bits" so to speak. I want to say "install_file SRC DST OWNER MODE" without needing to re-define the steps required to do that work in each file.

The lib directive is a good step, but for this I think it's not quite sufficient. My approach at the moment is essentially a preprocessor, which transforms parameterized !goal directives to proper @goal expressions with a form of templating. This works, but I wanted to solicit your opinion on alternative approaches, or potentially new capabilities in the core tool.

[xonixx]

Although you described your use-case really good, I would be really glad, if possible, to take a look at your Makesurefiles as well as the preprocessor.

[08d2]

Sure, here's a pretty standard example of what I'm trying to reduce.

@define PROGRAM="mysvc"
@define SERVICE_USER="mysvcuser"

@goal automated-build
@depends_on artifact-defined
	export BUILDVER=$(git rev-parse --short HEAD 2>/dev/null || echo hack)
	export BUILDDATE=$(date -u +%Y%m%dT%H%M%SZ)
	env GOOS=linux GOARCH=amd64 go build -ldflags="-X mysvc/debug.BuildVersion=${BUILDVER} -X mysvc/debug.BuildDate=${BUILDDATE}" -o ${ARTIFACT} ./cmd/mysvc

#
# installed
#

@goal installed
@depends_on artifact-defined artifact-downloaded user-created symlink-installed servicefile-installed secretenv-installed config-installed libwasmvm

@goal artifact-downloaded @private
@depends_on artifact-defined
@reached_if [ -x /opt/mystuff/artifacts/${ARTIFACT} ]
	s3cmd --quiet get s3://artifacts0/${ARTIFACT} /opt/mystuff/artifacts/${ARTIFACT}
	chmod +x /opt/mystuff/artifacts/${ARTIFACT}

@goal user-created @private
@reached_if id ${SERVICE_USER} >/dev/null 2>&1
	adduser --system --no-create-home --gecos '' --disabled-password --disabled-login --group ${SERVICE_USER}

@goal symlink-installed @private
@depends_on artifact-defined artifact-downloaded
@reached_if [ "$(readlink /opt/mystuff/bin/${PROGRAM})" = "/opt/mystuff/artifacts/${ARTIFACT}" ]
	ln -f -s /opt/mystuff/artifacts/${ARTIFACT} /opt/mystuff/bin/${PROGRAM}
	touch /opt/mystuff/dirty/${PROGRAM}

@goal servicefile-installed @private
@reached_if cmp --silent mysvc.service /etc/systemd/system/mysvc.service
	cp mysvc.service /etc/systemd/system/mysvc.service
	touch /opt/mystuff/dirty/${PROGRAM}

@goal secretenv-installed @private
@reached_if [ "$(stat --format=%U:%G:%a /opt/mystuff/conf/mysvc.secret.env 2>/dev/null)" = "${SERVICE_USER}:${SERVICE_USER}:400" ]
	install --mode=0400 --owner=${SERVICE_USER} --group=${SERVICE_USER} /root/.mysvc.secret.env /opt/mystuff/conf/mysvc.secret.env
	touch /opt/mystuff/dirty/${PROGRAM}

@goal config-installed @private
@reached_if [ "$(stat --format=%U:%G:%a /opt/mystuff/conf/mysvc.conf 2>/dev/null)" = "${SERVICE_USER}:${SERVICE_USER}:400" ] && cmp --silent mysvc.conf /opt/mystuff/conf/mysvc.conf
	install --mode=0400 --owner=${SERVICE_USER} --group=${SERVICE_USER} mysvc.conf /opt/mystuff/conf/mysvc.conf
	touch /opt/mystuff/dirty/${PROGRAM}

@goal daemon-reloaded @private
@depends_on servicefile-installed
@reached_if systemctl show mysvc.service 2>/dev/null | grep NeedDaemonReload=no >/dev/null
	systemctl --quiet daemon-reload
	touch /opt/mystuff/dirty/${PROGRAM}

#
# running
#

@goal running
@depends_on service-active

@goal service-active @private
@depends_on service-enabled daemon-reloaded
@reached_if systemctl --quiet is-active mysvc.service 2>/dev/null && [ ! -f /opt/mystuff/dirty/${PROGRAM} ]
	systemctl --quiet restart mysvc.service
	rm -f /opt/mystuff/dirty/${PROGRAM}

@goal service-enabled @private
@depends_on daemon-reloaded symlink-exists
@reached_if systemctl --quiet is-enabled mysvc.service 2>/dev/null
	systemctl --quiet enable mysvc.service

#
# uninstalled
#

@goal uninstalled
@depends_on service-inactive service-disabled servicefile-removed config-removed symlink-removed user-removed

@goal service-inactive @private
@reached_if ! systemctl --quiet is-active mysvc.service
	systemctl --quiet stop mysvc.service

@goal service-disabled @private
@depends_on service-inactive
@reached_if ! systemctl --quiet is-enabled mysvc.service 2>/dev/null
	systemctl --quiet disable mysvc.service
	systemctl --quiet reset-failed

@goal servicefile-removed @private
@depends_on service-disabled
@reached_if [ ! -f /etc/systemd/system/mysvc.service ]
	rm /etc/systemd/system/mysvc.service
	systemctl --quiet daemon-reload

@goal secretenv-removed @private
@depends_on service-disabled
@reached_if [ ! -f /opt/mystuff/conf/mysvc.secret.env ]
	rm -f /opt/mystuff/conf/mysvc.secret.env

@goal config-removed @private
@depends_on service-disabled
@reached_if [ ! -f /opt/mystuff/conf/mysvc.conf ]
	rm -f /opt/mystuff/conf/mysvc.conf

@goal symlink-removed @private
@depends_on service-disabled
@reached_if [ ! -f /opt/mystuff/bin/mysvc ]
	rm -f /opt/mystuff/bin/mysvc

@goal user-removed
@depends_on service-disabled
@reached_if ! id ${SERVICE_USER} >/dev/null 2>&1
	deluser ${USERNAME}
	delgroup ${USERNAME}

#
# misc
#

@goal artifact-defined @private
@reached_if [ ! -z "${ARTIFACT}" ]
	echo ARTIFACT not defined
	exit 1

@goal symlink-exists @private
@reached_if [ -x /opt/mystuff/bin/mysvc ]
	echo /opt/mystuff/bin/mysvc missing or not executable
	exit 1

@goal servicefile-exists @private
@reached_if [ -f /etc/systemd/system/mysvc.service ]
	echo /etc/systemd/system/mysvc.service missing
	exit 1

I'm still working on the preprocessor, but my goal is to turn e.g.

@goal con-installed @private
@reached_if [ "$(stat --format=%U:%G:%a /opt/mystuff/conf/mysvc.conf 2>/dev/null)" = "${SERVICE_USER}:${SERVICE_USER}:400" ]
	install --mode=0400 --owner=${SERVICE_USER} --group=${SERVICE_USER} mysvc.conf /opt/mystuff/conf/mysvc.conf
	touch /opt/mystuff/dirty/${PROGRAM}

into something like

!goal installfile GOAL=conf-installed SRC=mysvc.conf DST=/opt/mystuff/conf/mysvc.conf OWNER=${SERVICE_USER} MODE=400 PROGRAM=${PROGRAM}

which at the moment is accomplished with a template file for each !goal + envsubst.

[xonixx]

Thank you. This is interesting case. Looks like this requires some sort of parameterized goals. This is what I always resisted to add. Firstly, due to increased complexity that will not be needed in a majority of typical usage scenarios. But mostly, because it’s tricky to do while preserving the declarative semantics of dependencies.
You see, dependency of one goal on another is fundamentally different from function call.

But this is not impossible.
For example, just does have parameterized goals.

I need some time to think on this in depth.

[08d2]

Just to be clear, I completely respect and agree with your goal of keeping the tool laser-focused, and am completely happy with the preprocessor approach, so please don't feel like you have to accommodate me if you don't want to 😇

I would also be interested to hear if you had a better approach for doing what I'm trying to do here.

[xonixx]

Well, with current implementation I guess not much that you can do without resorting to external machinery (as you do).

Probably the most ideomatic way now (provide that the structure of all your Makesurefiles is almost the same) is to extract all changing parts to @define-s, so essentially use single Makesurefile while passing all the parameters externally.
But most likely this is not realistic in your case since the changes are more substantial. Am I right?

I don't think I'm completely opposed to the idea. I even had my own case when something like parameterized goals may be in help. But at that point I didn't get enough motivation (my though repetitive code worked good enough) and my approach to solution design in #96 was not what I liked. In fact I decided that I won't add a feature without clear need and scenarios of usage. Now looks like we have at least two such usage scenarios.

It's just that adding the feature requires (from me) lots of thoroughful consideration to:

• come up with good syntax: easy to use and easy to parse
• accidentally not to introduce alternative ways of doing the same
• avoid considerable complication of implementation and adding lots of KB to code size.
• understand all possible implications of new feature to the existing ones, so make sure they play well in all reasonable combinations

Not to mention, that for your use case we'll additionaly need some form of @incude to include the shared lib with generic goals in multiple Makesurefiles. Is this correct? @include in its turn brings lots of additional questions like: do we want to allow includes from includes (meaning we need to detect cycles), how do we override @define-s and many more.

[08d2]

. . . extract all changing parts to @define-s . . .

I have a Makesurefile for each program that I want to provision, and there is a lot of commonality between them, but I don't think they're so uniform that @define could be a solution. As a simple example, programs will install an arbitrary number of files to arbitrary locations.

we'll additionaly need some form of @incude to include the shared lib with generic goals in multiple Makesurefiles. Is this correct? @include in its turn brings lots of additional questions like: do we want to allow includes from includes (meaning we need to detect cycles), how do we override @define-s and many more.

I defer to your judgment, but my intention here was not necessarily to extend the grammar/capabilities of Makesure itself. Rather I was hoping for something like a set of built-in goals that would solve common tasks — or at least what I think are common tasks ;) — like installing files, adding users, interacting with systemd, etc. These goals would by their nature need to be parameterized.

[xonixx]

I see. I would prefer not to do it via the built-ins.
Firstly, I want to keep the core tool simple.
Secondly, I don't want to support the "standard lib" myself. I believe that if we manage to add proper tools for extending, other people may come up with much better/useful libs.

[xonixx]

Hey @08d2! Could you please take a look at #115. Do you think this design approach can help with your use-case?

[08d2]

Interesting! Let me try to grok it...

[xonixx]

Hey @08d2! I've released v0.9.20 with parameterized goals support!
I think this should be applicable to your use-case.
Please see the documentation https://github.com/xonixx/makesure#parameterized-goals.