Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test for error output #494

Merged
merged 3 commits into from
Sep 21, 2015
Merged

Test for error output #494

merged 3 commits into from
Sep 21, 2015

Conversation

saper
Copy link
Member

@saper saper commented Sep 19, 2015

Enable testing for error output
and sass compiler failures.

If files "error" and "status" are existing
in the test directory, do expect errors
and test against them.

Errors returned by the Sass engines are
now treated as test failures, not errors.

Also create error and status files with --nuke.
An error will be reported in this case anyway.

Important Now the tests cannot be ran in parallel,
because we need to capture standard error reliably.

Should fix: #136 #133

@saper
Copy link
Member Author

saper commented Sep 19, 2015

As the Travis shows, this change work only with adding compiler via the -c option (can be sassc or Ruby sass CLI). Unfortunately, calling Ruby sass internally presents the incompatible interface (this needs to be fixed, too).

@saper
Copy link
Member Author

saper commented Sep 20, 2015

So after resolving #495, #497, #499 the tests finally pass...

@saper
Copy link
Member Author

saper commented Sep 20, 2015

This should be enough for rudimentary error testing. Should we need more, we should be aware of the following limitations:

Convert --skip/-s to no-op, as this is now the default
behavior.

Rationale: As we now expect errors to be thrown by
the spec scripts it makes no sense to stop whenever
a first error is encountered.

Unfortunately we cannot print deprecation warning
in order not to break formats like tap or mess
with stderr (Windows PowerShell thinks for example
that if there is anything on stderr, the command failed).
We cannot run tests in parallel
when redirecting $stderr into a single buffer

Fixes: sass#499
If files "error" and "status" are existing
in the test directory, do expect errors
and test against them.

The errors returned by the Sass engines are
now treated as test failures (not fatal errors).

--unexpected-pass reports an error whenever
a test marked as "todo" does pass. Failing
"todo" errors are silently marked as "passed".

Also re-create output test files with --nuke.
"status" and "error" files are re-created if necessary.
Any failures will be reported in this case normally.

Fixes: sass#136
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot test error reporting
1 participant