Make failing tests possible, improve test output. #123
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Until now it has been possible to test only that something succeeds
(specifically, that the output matches exactly). This limited the
tests to cases where we had determinism and where the expected output
was exactly known.
This commit extends test.sh to maintain the old functionality, but now
ALSO allows checking scripts to be used instead of the old
.out
files.A checking script is an executable whose name is that of a test, but
with the ending
.chk
-- if the test source if calledfoo.enc
, thechecking script is called
foo.chk
and it is executable. The filetestutils.sh contains a few functions to make it easier to write tests
and to make maintenance of tests easier, should error output change
(then only this file has to be updated).
The
make test
command will now compilefoo.enc
and try to run it(if compilation succeeds). The output of both compilation and
execution will be piped into the checking script. The checking script
can analyse the input and return successfully to signal that
everything is ok, or with a failure code to signal an error.
An example:
File fail.enc:
File fail.chk: