Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure all Haskell test cases still run within same file when there are failed test cases #543

Merged

Conversation

donny-wong
Copy link
Contributor

Description:
When running Haskell tests within a file, and there is at least 1 failed tests, all tests fail in the file even though some of them or passing. The issue is If any test case fails, the stack runghc ... command will have an exit status of 1, not 0. This will cause the subprocess.run call to raise an error, because of the check=True argument.

Solution:
A try-except block was used around the subprocess.run call. If the return code is 1 we won't raise an error but will still raise it for other errors.

Testing
Currently markus-autotesting does not have a framework to do unit-testing for Haskell, and due to time, I did not write unit tests for this.

However, to do a manual test, I used a Haskell test file that has 1 passing and 1 failing test case where I named the file Test.hs, with the following contents (note: ".hs" extension file is not supported on GitHub, so I am displaying the contents here.):

module Test where
import Test.QuickCheck

prop_true :: () -> Bool
prop_true _ = True

prop_false :: () -> Bool
prop_false _ = False

It should show 1 passing and 1 failing test in the results after running a test.

@donny-wong donny-wong changed the title fixed Haskell failed test error Ensure all Haskell test cases still run within same file when there are failed test cases Aug 23, 2024
@pretendWhale pretendWhale added this to the v2.5.1 milestone Aug 27, 2024
@david-yz-liu david-yz-liu merged commit 50a0ad7 into MarkUsProject:master Aug 27, 2024
13 checks passed
pretendWhale pushed a commit to pretendWhale/markus-autotesting that referenced this pull request Aug 27, 2024
…re failed test cases (MarkUsProject#543)

Co-authored-by: Donny Wong <donnywong@cs.toronto.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants