Skip to content
This repository has been archived by the owner on Nov 1, 2023. It is now read-only.

add regression testing tasks #664

Merged
merged 111 commits into from
Mar 18, 2021
Merged

Conversation

bmc-msft
Copy link
Contributor

@bmc-msft bmc-msft commented Mar 11, 2021

This PR adds regression tasks to identify if new builds demonstrate or fix previously identified crashes.

The intent is to support two models:

  1. A task that run as part of every template, to enable automatic updating of ADO Work items or GitHub issues. While the notification integration does not work yet, consider automatic marking bugs found via fuzzing as "no longer crashes" or re-opening bugs that were re-introduced.
  2. A template that enables easy git bisect.

To demonstrate this idea, I added libfuzzer-regression test binaries of broken.exe and fixed.exe (not yet included in integration testing). As you would expect by the name, broken.exe has a bug and fixed.exe does not.

We can demonstrate this by doing the following:

PROJ=regress
NAME=regress
onefuzz template libfuzzer basic --vm_count 1 --duration 1 --target_exe ./broken.exe $PROJ $NAME 1 linux --wait_for_files unique_reports
onefuzz template stop ${PROJ} ${NAME}
onefuzz template libfuzzer basic --vm_count 1 --duration 1 --target_exe ./fixed.exe $PROJ $NAME 2 linux --wait_for_files regression_reports 
onefuzz template stop ${PROJ} ${NAME}
onefuzz template libfuzzer basic --vm_count 1 --duration 1 --target_exe ./broken.exe $PROJ $NAME 3 linux --wait_for_files regression_reports

During the first job, we'll see a single unique crash report created and no regression reports.

During the second job, we'll still have that unique one crash report. In the job specific regression reports container, now we'll have a regression report. Upon pulling that regression report, we'll see that the crash_test_result.no_repro exists.

During the third job, we'll still have that unique one crash report. In the job specific regression reports container, now we'll have a regression report. Upon pulling that regression report, we'll see that the crash_test_result.crash_result exists.

Addresses #308.

Note, this is in large part built from the work of #433.

@bmc-msft bmc-msft requested review from ranweiler and chkeita March 11, 2021 07:53
@bmc-msft bmc-msft changed the title add regression testing add regression testing tasks Mar 11, 2021
src/agent/onefuzz-agent/src/tasks/regression/common.rs Outdated Show resolved Hide resolved
src/agent/onefuzz-agent/src/tasks/regression/common.rs Outdated Show resolved Hide resolved
src/agent/onefuzz/src/blob/url.rs Outdated Show resolved Hide resolved
src/cli/onefuzz/templates/regression.py Outdated Show resolved Hide resolved
src/cli/onefuzz/templates/regression.py Outdated Show resolved Hide resolved
@bmc-msft bmc-msft requested a review from ranweiler March 13, 2021 01:55
src/cli/onefuzz/templates/regression.py Outdated Show resolved Hide resolved
src/cli/onefuzz/templates/regression.py Outdated Show resolved Hide resolved
@bmc-msft bmc-msft merged commit 6e60a8c into microsoft:main Mar 18, 2021
@bmc-msft bmc-msft deleted the regression-suggestions branch March 18, 2021 19:37
@ghost ghost locked as resolved and limited conversation to collaborators Apr 18, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a regression/crash repro task
4 participants