Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dont fail error suites #540

Merged
merged 4 commits into from
Dec 1, 2022
Merged

Dont fail error suites #540

merged 4 commits into from
Dec 1, 2022

Conversation

ladisgin
Copy link
Member

@ladisgin ladisgin commented Nov 17, 2022

Before all generated error tests should be fail than run.

TEST(error, buggy_function2_test_2)
{
    write_to_file("../../../tests/lib/A", "\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0");
    write_to_file("../../../tests/lib/B", "\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0");
    write_to_file("../../../tests/lib/C", "\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0");

    // Construct input
    int a = 0;

    // Trigger the function
    buggy_function2(a);

    FAIL() << "Unreachable point or the function was supposed to fail, but "
        "actually completed successfully. See the SARIF report for details.";
}

In this PR adds the ability to choose whether the test will fail or pass.

image

This use EXPECT_ANY_THROW and ASSERT_DEATH gtest macros

TEST(error, buggy_function2_test_2)
{
    write_to_file("../../../tests/lib/A", "\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0");
    write_to_file("../../../tests/lib/B", "\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0");
    write_to_file("../../../tests/lib/C", "\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0""\0");

    // Construct input
    int a = 0;

    // Trigger the function
    ASSERT_DEATH(buggy_function2(a), ".*");
}

Add VSCode setting description at wiki

@ladisgin ladisgin marked this pull request as draft November 17, 2022 03:03
@ladisgin ladisgin requested a review from sava-cska November 17, 2022 08:40
@ladisgin ladisgin marked this pull request as ready for review November 21, 2022 02:30
@ladisgin ladisgin requested a review from Lana243 November 21, 2022 08:16
Copy link
Collaborator

@Lana243 Lana243 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to have some documentation about this PR. How does the behavior of running error test will change after this PR?
I would recommend to add this information to wiki or to the description of this PR and also comment new entities like ErrorMode and ErrorType in code.

Do not forget to squash commits before merge.

Comment on lines +93 to +97
enum ErrorMode {
FAILING = 0;
PASSING_IN_TARGET_ONLY = 1;
PASSING = 2;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add some docs about error mode? Probably, comments or doc at wiki?

server/src/KleeGenerator.h Outdated Show resolved Hide resolved
Comment on lines +7 to +11
enum ErrorType {
NO_ERROR,
ASSERTION_FAILURE,
EXCEPTION_THROWN
};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add some docs about error mode? Probably, comments or doc at wiki?

server/src/utils/TraceUtils.cpp Outdated Show resolved Hide resolved
@ladisgin ladisgin force-pushed the ladisgin/dont_fail_error_suites branch from 737185e to cc4ee35 Compare November 25, 2022 06:43
@ladisgin ladisgin merged commit b17516c into main Dec 1, 2022
@sava-cska sava-cska deleted the ladisgin/dont_fail_error_suites branch December 12, 2022 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Don't fail error suite tests (different ways to report founded assert fails: as error or as predicted and ok)
3 participants