Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make failing tests possible, improve test output. #123

Merged
merged 1 commit into from
Apr 18, 2015
Merged

Make failing tests possible, improve test output. #123

merged 1 commit into from
Apr 18, 2015

Conversation

kaeluka
Copy link
Contributor

@kaeluka kaeluka commented Apr 16, 2015

Until now it has been possible to test only that something succeeds
(specifically, that the output matches exactly). This limited the
tests to cases where we had determinism and where the expected output
was exactly known.

This commit extends test.sh to maintain the old functionality, but now
ALSO allows checking scripts to be used instead of the old .out files.

A checking script is an executable whose name is that of a test, but
with the ending .chk -- if the test source if called foo.enc, the
checking script is called foo.chk and it is executable. The file
testutils.sh contains a few functions to make it easier to write tests
and to make maintenance of tests easier, should error output change
(then only this file has to be updated).

The make test command will now compile foo.enc and try to run it
(if compilation succeeds). The output of both compilation and
execution will be piped into the checking script. The checking script
can analyse the input and return successfully to signal that
everything is ok, or with a failure code to signal an error.

An example:

File fail.enc:

class Main
  def main() : void
    repeat i <- 100 {
      print x -- error should occur here
    }

File fail.chk:

#!/usr/bin/env bash
source testutils.sh

error_at 4
error_msg "Unbound variable 'x'"

Until now it has been possible to test only that something succeeds
(specifically, that the output matches exactly). This limited the
tests to cases where we had determinism and where the expected output
was exactly known.

This commit extends test.sh to maintain the old functionality, but now
ALSO allows checking scripts to be used instead of the old `.out` files.

A checking script is an executable whose name is that of a test, but
with the ending `.chk` -- if the test source if called `foo.enc`, the
checking script is called `foo.chk` and it is executable. The file
testutils.sh contains a few functions to make it easier to write tests
and to make maintenance of tests easier, should error output change
(then only this file has to be updated).

The `make test` command will now compile `foo.enc` and try to run it
(if compilation succeeds). The output of both compilation and
execution will be piped into the checking script. The checking script
can analyse the input and return successfully to signal that
everything is ok, or with a failure code to signal an error.

An example:

File fail.enc:

    class Main
      def main() : void
        repeat i <- 100 {
          print x -- error should occur here
        }

File fail.chk:

    #!/usr/bin/env bash
    source testutils.sh

    error_at 4
    error_msg "Unbound variable 'x'"
@kaeluka
Copy link
Contributor Author

kaeluka commented Apr 16, 2015

This is a new pull request (instead of #121), with a small testing lib that'll make writing tests clearer and easier.

@TheGrandmother
Copy link
Contributor

👍
This is great!

supercooldave pushed a commit that referenced this pull request Apr 18, 2015
Make failing tests possible, improve test output.
@supercooldave supercooldave merged commit 4ca238d into parapluu:master Apr 18, 2015
@kaeluka kaeluka deleted the failing-tests branch April 18, 2015 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants