For Mac & Linux
Step 1: Set up Defects4J
Refer https://github.com/rjust/defects4j
Clone the repository
Follow documentation steps
Error:Can't Locate DBI.pm
Resolution for Mac: perl -MCPAN -e 'install DBI'
Install Postgres if required.
Resolution for Linux: sudo apt install libdbi-perl
Step 2: Clone the repo https://bitbucket.org/rjust/fault-localization-data/overview
Step 3: Download and install JDK 1.6 and JDK 1.8.
Step 4: Set evnironment variables
The path of defects4j installed in step 1
export D4J_HOME=/Users/{username}/Downloads/defects4j
The path to this root directory of the cloned repo fault-localization-data and append 'gzoltar/gzoltar.jar'
export GZOLTAR_JAR=/Users/{username}/Downloads/fault-localization-data/gzoltar/gzoltar.jar
Set JAVA_HOME to point to JDK1.6 Home if you have a different Java default version
export JAVA_HOME=/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Add to PATH variable
export PATH=$PATH:$D4J_HOME/framework/bin
Step 6: Replace the run_gzoltar.sh provided in this repository in fault-localization-data/gzoltar/gzoltar
Step 5: Test if set up works
Test defects4j
defects4j info -p Lang
Test Gzoltar
`bash run_gzoltar.sh Lang 37 . developer`
Step 7: Install sloccount
For Mac: brew install sloccount
For Mac: export SLOC_HOME=/usr/local/bin/sloccount
For Ubuntu: sudo apt-get install sloccount
For Ubuntu: export SLOC_HOME=/usr/bin/sloccount
Step 8: Run the script to get the buggy code
bash get_fixed_lines.sh Lang 37 .
This creates a file `Lang-37.fixed.lines` which contains the line fixed in the human patch for the corresponding bug.
This can be used to evaluate the suspiciousness score generated by the Fault localization technique.
To manually run junit tests in the defects4j projects
Step 1: checkout defects4j project(example: Lang)
defects4j checkout -p Lang -v 37b -w /tmp/Lang37
Step 2: Download JUNIT Jar
Download the junit jar from https://github.com/downloads/junit-team/junit/junit-4.10.jar
Step 3: Downgrade Maven to mvn 3.2 so as to run with Java 1.6
Check maven version by running.
mvn --version
If this is 3.2 or below then skip to Step 4.
If not follow these steps to downgrade maven
brew install maven@3.2
brew unline maven
brew link --force --overwrite maven@3.2
Step 4: Compile Lang Project
Go to the checked out Lang project folder in two session on Terminal
Session 1:
export JAVA_HOME=/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
mvn compile (This will fail)
Session 2:
mvn compile
rm -rf target/*
Session 1:
mvn compile (This should succeed)
Step 5: Run all of the Junit tests in Lang Project using maven
Session 1:
mvn -Dmaven.test.failure.ignore=true install (This will fail)
Session 2:
mvn -Dmaven.test.failure.ignore=true install
rm -rf target/commons-lang-3.0-SNAPSHOT.jar
Session 1:
mvn -Dmaven.test.failure.ignore=true install (This should succeed)
mvn test
Step 6: Run one of the Junit tests in Lang Project from command line without maven
cd target
cp {Path to junit jar}/junit-4.10.jar .
java -cp .:/tmp/Lang37/target/test-classes/:junit-4.10.jar:commons-lang-3.0-SNAPSHOT.jar org.junit.runner.JUnitCore org.apache.commons.lang3.ArrayUtilsAddTest
Step 7a: Setting up javaslicer Clone the repository for javaslicer (https://github.com/hammacher/javaslicer) cd javaslicer ./assemble.sh (Note: There might be failures when you run this the first time. Run ./assemble.sh to fix it.)
Step 7b: Run one of the junit test classes with tracer as a javaagent attached.
Session 2:
java -cp .:/tmp/Lang37/target/test-classes/:junit-4.10.jar:commons-lang-3.0-SNAPSHOT.jar -javaagent:/Users/jithinjohn/Downloads/CSC591/javaslicer/assembly/tracer.jar=tracefile:test.trace org.junit.runner.JUnitCore org.apache.commons.lang3.ArrayUtilsAddTest
Step 8: Run slicer to produce slicing results
Session 2:
java -Xmx2g -jar /Users/jithinjohn/Downloads/CSC591/javaslicer/assembly/slicer.jar -p test.trace org.apache.commons.lang3.ArrayUtils.addAll:2962:* > output.txt
Change directory to automation/src/
COMPILE:
javac -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests.java
RUN:
- Running a test file
java -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests /tmp/Lang37 runTestFile org.apache.commons.lang3.ArrayUtilsAddTest
- Running a test specific test case
java -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests /tmp/Lang37 runTestCase org.apache.commons.lang3.ArrayUtilsAddTest testJira567
- Getting test cases of a test file
java -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests /tmp/Lang37 getTestCases org.apache.commons.lang3.ArrayUtilsAddTest
- Getting line numbers of assert statements in a test case
java -cp .:/tmp/Lang37/target/:/tmp/Lang37/target/test-classes/:/tmp/Lang37/target/junit-4.10.jar:/tmp/Lang37/target/commons-lang-3.0-SNAPSHOT.jar InvokeTests /tmp/Lang37 getAssertLines org.apache.commons.lang3.ArrayUtilsAddTest testJira567
This repository contains data files, data-collection scripts, and data-analysis scripts of the "Evaluating and Improving Fault Localization Techniques" project. Before exploring this repository, please read the technical report that describes the results.
The experiments evaluate various fault localization techniques on artificial faults and on real faults.
At a high level, here's how it all works:
- The real and artificial faults come from the Defects4J Project.
- For each D4J fault, the scripts in
d4j_integration/
determine which lines are faulty. The resultant files are "buggy-lines" files, and live inanalysis/pipeline-scripts/buggy-lines/
. - Many fault localization techniques require coverage information. We use GZoltar to gather coverage information. The resultant files are called "matrix" and "spectra".
- Mutation-based fault localization (MBFL) techniques require mutation analysis. Our Killmap project (which lives in
killmap/
) does mutation analysis on all faults. The resultant files are called "killmaps," and specify how each test behaves on each mutant. (Each killmap also has an associated "mutants-log" file, which describes all the mutants that were analyzed.) - Our scripts enable you to compute all the mutation and coverage information, but doing so takes a great deal of computation. The resulting mutation/coverage information is available at http://fault-localization.cs.washington.edu.
- The "scoring pipeline" (which lives in
analysis/pipeline-scripts/
) determines how well each FL technique does on each fault -- that is, where the real buggy lines appear in the FL technique's ranking of the line of the program. The results appear indata/
.
Before doing anything else, run ./setup.sh
. This:
- clones the appropriate Defects4J fork (unless you've already exported a
D4J_HOME
directory); - updates your
.bashrc
to export some environment variables:D4J_HOME
andDEFECTS4J_HOME
, pointing to the newdefects4j
repository, if it neededFL_DATA_HOME
, pointing hereKILLMAP_HOME
, pointing at./killmap/
GZOLTAR_JAR
, pointing to./gzoltar/gzoltar.jar
The workflow to score a set of FL techniques on a given fault looks like this:
-
Various pieces of fault information were generated by the tools in
./d4j_integration/
and then checked in. You don't need to generate them yourself, but if you want to, see theREADME.md
in that directory. -
To run GZoltar, use
gzoltar/run_gzoltar.sh
.Example invocation:
bash run_gzoltar.sh Lang 37 . developer
Creates the files
./matrix
and./spectra
. -
To run Killmap, use
killmap/scripts/generate-matrix
.Example invocation:
killmap/scripts/generate-matrix \ Lang 37 \ /tmp/Lang-37 \ Lang-37.mutants.log \ | gzip > Lang-37.killmap.csv.gz
Creates the files
Lang-37.killmap.csv.gz
andLang-37.mutants.log
. -
To run the scoring pipeline, use
analysis/pipeline-scripts/do-full-analysis
.Example invocation:
analysis/pipeline-scripts/do-full-analysis \ Lang 37 'developer' \ ./matrix ./spectra \ Lang-37.killmap.csv.gz Lang-37.mutants.log \ /tmp/Lang-37-scoring \ Lang-37.scores.csv`
Creates the file
Lang-37.scores.csv
.
For more details on any of these scripts, see the README.md
in the script's directory.
If you want to skip running GZoltar and Killmap (which can be very computationally expensive), you can download the resulting files from http://fault-localization.cs.washington.edu.
-
analysis/
: Tools for analyzing the output of coverage/mutation analyses. -
aws/
: Scripts for computing killmaps on AWS. -
cluster_scripts/
: Scripts for computing killmaps on a Sun Grid cluster. -
d4j_integration/
: Scripts that build upon or extend Defects4J to populate or query its database. -
data/
: Data files for the final results and corresponding support scripts. -
gzoltar/
: Scripts for running the GZoltar tool to collect line coverage information. -
killmap/
: Mutation-analysis tool whose output is used for the MBFL techniques we study. -
stats/
: R scripts that crunch the data to produce numbers for the paper. -
utils/
: Utility programs and libraries for running/analyzing tests and parsing data files. ======= =======