Remake, a build system that bridges the gap between make and redo

As with make, remake uses a centralized rule file, which is named Remakefile. It contains rules with a make-like syntax:

target1 target2 ... : prerequisite1 prerequisite2 ...
	shell script
	that builds
	the targets

A target is known to be up-to-date if all its prerequisites are. If it has no known prerequisites yet the file already exits, it is assumed to be up-to-date. Obsolete targets are rebuilt thanks to the shell script provided by the rule.

As with redo, remake supports dynamic dependencies in addition to these static dependencies. Whenever a script executes remake prerequisite4 prerequisite5 ..., these prerequisites are rebuilt if they are obsolete. (So remake acts like redo-ifchange.) Moreover, all the dependencies are stored in file .remake so that they are remembered in subsequent runs. Note that dynamic dependencies from previous runs are only used to decide whether a target is obsolete; they are not automatically rebuilt when they are obsolete yet a target depends on them. They will only be rebuilt once the dynamic call to remake is executed.

In other words, the following two rules have almost the same behavior.

target1 target2 ... : prerequisite1 prerequisite2 ...
	shell script

target1 target2 ... :
	remake prerequisite1 prerequisite2 ...
	shell script

(There is a difference if the targets already exist, have never been built before, and the prerequisites are either younger or obsolete, since the targets will not be rebuilt in the second case.)

The above usage of dynamic dependencies is hardly useful. Their strength lies in the fact that they can be computed on the fly:

%.o : %.c
	gcc -MMD -MF $@.d -o $@ -c $<
	remake -r < $@.d
	rm $@.d

%.cmo : %.ml
	ocamldep $< | remake -r $@
	ocamlc -c $<

after.xml: before.xml rules.xsl
	xsltproc --load-trace -o after.xml rules.xsl before.xml 2> deps
	remake `sed -n -e "\\,//,! s,^.*URL=\"\\([^\"]*\\).*\$,\\1,p" deps`
	rm deps

Note that the first rule fails if any of the header files included by a C source file has to be automatically generated. In that case, one should perform a first call to remake them before calling the compiler. (Dependencies from several calls to remake are cumulative, so they will all be remembered the next time.)

Usage

Usage: remake options targets

Options:

• -B, --always-make: Unconditionally make all targets.
• -d: Echo script commands.
• -f FILE: Read FILE as Remakefile.
• -j[N], --jobs=[N]: Allow N jobs at once; infinite jobs with no argument.
• -k, --keep-going: Keep going when some targets cannot be made.
• -r: Look up targets from the dependencies on standard input.
• -s, --silent, --quiet: Do not echo targets.

Syntax

Lines starting with a space character or a tabulation are assumed to be rule scripts. They are only allowed after a rule header.

Lines starting with # are considered to be comments and are ignored. They do interrupt rule scripts though.

Any other line is either a variable definition or a rule header. If such a line ends with a backslash, the following line break is ignored and the line extends to the next one.

Variable definitions are a single name followed by equal followed by a list of names, possibly empty.

Rule headers are a nonempty list of names, followed by a colon, followed by another list of names, possibly empty. Basically, the syntax of a rule is as follows:

targets : prerequisites
	shell script

List of names are space-separated sequences of names. If a name contains a space character, it should be put into double quotes. Names cannot be any of the following special characters :$(),=". Again, quotation should be used. Quotation marks can be escaped by a backslash inside quoted names.

Variables

Variables can be used to factor lists of targets or prerequisites. They are expanded as they are encountered during Remakefile parsing.

VAR2 = a
VAR1 = c d
VAR2 += $(VAR1) b
$(VAR2) e :

Variable assignments can appear instead of prerequisites inside non-generic rules with no script. They are then expanded inside the corresponding generic rule.

foo.o: CFLAGS += -DBAR

%.o : %.c
	gcc $(CFLAGS) -MMD -MF $@.d -o $@ -c $<
	remake -r < $@.d
	rm $@.d

Note: contrarily to make, variable names have to be enclosed in parentheses. For instance, $y is not a shorthand for $(y) and is left unexpanded.

Automatic variables

The following special symbols can appear inside scripts:

• $< expands to the first static prerequisite of the rule.
• $^ expands to all the static prerequisites of the rule, including duplicates if any.
• $@ expands to the first target of the rule.
• $* expands to the string that matched % in a generic rule.
• $$ expands to a single dollar symbol.

Note: contrarily to make, there are no corresponding variables. For instance, $^ is not a shorthand for $(^). Another difference is that $@ is always the first target, not the one that triggered the rule.

Built-in functions

remake also supports a few built-in functions inspired from make.

• $(addprefix prefix, list) returns the list obtained by prepending its first argument to each element of its second argument.
• $(addsuffix suffix, list) returns the list obtained by appending its first argument to each element of its second argument.

Order-only prerequisites

If the static prerequisites of a rule contain a pipe symbol, prerequisites on its right do not cause the targets to become obsolete if they are newer (unless they are also dynamically registered as dependencies). They are meant to be used when the targets do not directly depend on them, but the computation of their dynamic dependencies does.

%.o : %.c | parser.h
	gcc -MMD -MF $@.d -o $@ -c $<
	remake -r < $@.d
	rm $@.d

parser.c parser.h: parser.y
	yacc -d -o parser.c parser.y

Static pattern rules

A rule with the following structure is expanded into several rules, one per target.

targets: pattern1 pattern2 ...: prerequisites

Every target is matched against one of the patterns containing the % character. A rule is then created using the patterns as targets, after having substituted % in the patterns and prerequisites. The automatic variable $* can be used in the script of the rule.

Special targets

Target .PHONY marks its prerequisites as being always obsolete.

Special variables

Variable .OPTIONS is handled specially. Its content enables some features of remake that are not enabled by default.

• variable-propagation: When a variable is set in the prerequisite part of a rule, it is propagated to the rules of all the targets this rule depends on. This option also enables variables to be set on the command line. Note that, as in make, this features introduces non-determinism: the content of some variables will depend on the build order.

Semantics

When are targets obsolete?

A target is obsolete:

• if there is no file corresponding to the target, or to one of its siblings in a multi-target rule,
• if any of its dynamic prerequisites from a previous run or any of its static prerequisites is obsolete,
• if the latest file corresponding to its siblings or itself is older than any of its dynamic prerequisites or static prerequisites.

In all the other cases, it is assumed to be up-to-date (and so are all its siblings). Note that the last rule above says "latest" and not "earliest". While it might cause some obsolete targets to go unnoticed in corner cases, it allows for the following kind of rules:

config.h stamp-config_h: config.h.in config.status
	./config.status config.h
	touch stamp-config_h

A config.status file generally does not update header files (here config.h) if they would not change. As a consequence, if not for the stamp-config_h file above, a header would always be considered obsolete once one of its prerequisites is modified. Note that touching config.h rather than stamp-config_h would defeat the point of not updating it in the first place, since the program files would need to be rebuilt.

Once all the static prerequisites of a target have been rebuilt, remake checks whether the target still needs to be built. If it was obsolete only because its prerequisites needed to be rebuilt and none of them changed, the target is assumed to be up-to-date.

How are targets (re)built?

There are two kinds of rules. If any of the targets or prerequisites contains a % character, the rule is said to be generic. All the targets of the rule shall then contain a single % character. All the other rules are said to be specific.

A rule is said to match a given target:

• if it is specific and the target appears inside its target list,
• if it is generic and there is a way to replace the % character from one of its targets so that it matches the given target.

When remake tries to build a given target, it looks for a specific rule that matches it. If there is one and its script is nonempty, it uses it to rebuild the target.

Otherwise, it looks for a generic rule that matches the target. If there are several matching rules, it chooses the one with the shortest pattern (and if there are several ones, the earliest one). It then looks for specific rules that match each target of the generic rule. All the prerequisites of these specific rules are added to those of the generic rule. The script of the generic rule is used to build the target.

Example:

t%1 t2%: p1 p%2
	commands building t%1 and t2%

t2z: p4
	commands building t2z

ty1: p3

# t2x is built by the first rule (which also builds tx1) and its prerequisites are p1, px2
# t2y is built by the first rule (which also builds ty1) and its prerequisites are p1, py2, p3
# t2z is built by the second rule and its prerequisite is p4

The set of rules from Remakefile is ill-formed:

• if any specific rule matching a target of the generic rule has a nonempty script,
• if any target of the generic rule is matched by a generic rule with a shorter pattern.

Compilation

• On Linux, MacOSX, and BSD: g++ -o remake remake.cpp
• On Windows: g++ -o remake.exe remake.cpp -lws2_32

Installing remake is needed only if Remakefile does not specify the path to the executable for its recursive calls. Thanks to its single source file, remake can be shipped inside other packages and built at configuration time.

Differences with other build systems

Differences with make:

• Dynamic dependencies are supported.
• For rules with multiple targets, the shell script is executed only once and is assumed to build all the targets. There is no need for convoluted rules that are robust enough for parallel builds. For generic rules, this is similar to the behavior of pattern rules from gmake.
• As with redo, only one shell is run when executing a script, rather than one per script line. Note that the shells are run with option -e, thus causing them to exit as soon as an error is encountered.
• The prerequisites of generic rules (known as implicit rules in make lingo) are not used to decide between several of them, which means that remake does not select one for which it could satisfy the dependencies.
• Variables and built-in functions are expanded as they are encountered during Remakefile parsing.
• Target-specific variables are not propagated, unless specifically enabled, since this causes non-deterministic builds. This is the same for variables set on the command line.

Differences with redo:

• As with make, it is possible to write the following kind of rules in remake.

    Remakefile: Remakefile.in ./config.status
    	./config.status Remakefile
  

• If a target is already built the first time remake runs, it still uses the static prerequisites of rules mentioning it to check whether it needs to be rebuilt. It does not assume it to be up-to-date. As with redo though, if its obsolete status would be due to a dynamic prerequisite, it will go unnoticed; it should be removed beforehand.

• Multiple targets are supported.

• remake has almost no features: no checksum-based dependencies, no compatibility with job servers, etc.

Limitations

• If a rule script calls remake, the current working directory should be the directory containing Remakefile (or the working directory from the original remake if it was called with option -f).
• As with make, variables passed on the command line should keep the same values, to ensure deterministic builds.
• Some cases of ill-formed rules are not caught by remake and can thus lead to unpredictable behaviors.

Links

See http://cr.yp.to/redo.html for the philosophy of redo and https://github.com/apenwarr/redo for an implementation and some comprehensive documentation.

Licensing

Copyright (C) 2012-2020 by Guillaume Melquiond guillaume.melquiond@gmail.com

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.