Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a --fail_fast switch to common test #1979

Merged
merged 1 commit into from
Dec 18, 2018
Merged

Conversation

ferd
Copy link
Collaborator

@ferd ferd commented Dec 17, 2018

This switch aborts the rebar3 run after the first test failure is
reported.

Since Common Test does not support this feature and it can be useful for
interactive development (a demand from one of the teams I work with), I
decided to add the feature as an experimental CT hook.

The hook notices any test failure, and forcefully aborts the current
script execution, but only during the initialization phase of the next
test; this ensure that all other hooks for a post-test failure have the
time to do things right.

The CT logs will be interupted, and hooks of all kinds may also
suffer.

Since this might be a bit tricky to support internally, I will be fine
with a review that results in "no, Fred. Please test this in a project
plugin first" or if I get told to use profiles to mature the feature
(i.e. rebar3 as failfast ct ...) with a custom hook declaration.

This switch aborts the rebar3 run after the first test failure is
reported.

Since Common Test does not support this feature and it can be useful for
interactive development (a demand from one of the teams I work with), I
decided to add the feature as an experimental CT hook.

The hook notices any test failure, and forcefully aborts the current
script execution, but only during the initialization phase of the next
test; this ensure that all other hooks for a post-test failure have the
time to do things right.

The CT logs _will_ be interupted, and hooks of all kinds may also
suffer.

Since this might be a bit tricky to support internally, I will be fine
with a review that results in "no, Fred. Please test this in a project
plugin first" or if I get told to use profiles to mature the feature
(i.e. `rebar3 as failfast ct ...`) with a custom hook declaration.
@ferd
Copy link
Collaborator Author

ferd commented Dec 17, 2018

I could also move this feature to cth_readable without a problem, and keep the integration the same fwiw

@ferd ferd changed the title [proposal] Add a --fail_fast switch to common test Add a --fail_fast switch to common test Dec 18, 2018
@ferd ferd merged commit 968458b into erlang:master Dec 18, 2018
@tsenturk
Copy link

tsenturk commented Jan 8, 2019

Hello,

This switch consider skipped test as failed. Are you sure about this behavior? Skipped or not_implemented cts should NOT consider as failed.

Sample log;

%%% mambo_jambo_SUITE ==> client.client_fail.client_additional_verification.download_mambo_jambo_additional_check: SKIPPED
%%% mambo_jambo_SUITE ==> {tc_user_skip,check_not_implemented}
Detected test failure. Aborting

@ferd
Copy link
Collaborator Author

ferd commented Jan 8, 2019

It should have only failed on non-manual skips, so this might be a bug.

Can you open an issue and fill in the template? There are some variations across Erlang versions in how CT handles things so I want to be sure I reproduce in the right context.

@tsenturk
Copy link

tsenturk commented Jan 8, 2019

I didn't checked code but please do not hesitate contact me for support.
I've created #1992 for this.
Sevgiler.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants