Skip to content

Commit

Permalink
Add Q&A from Example 2
Browse files Browse the repository at this point in the history
  • Loading branch information
@espertus
espertus committed Mar 9, 2024
1 parent 038a8ea commit 3fd52c1
Showing 1 changed file with 105 additions and 37 deletions.
142 changes: 105 additions & 37 deletions doc/FAQ.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,12 @@
* [How do I use Checkstyle?](https://github.com/jacquard-autograder/jacquard-examples/blob/main/README.md#how-do-i-use-checkstyle)
* [What's PMD? How do I use it?](https://github.com/jacquard-autograder/jacquard-examples/blob/main/README.md#whats-pmd-how-do-i-use-it)
* [How do I set test result visibility?](https://github.com/jacquard-autograder/jacquard-examples/blob/main/README.md#how-do-i-set-test-result-visibility)
* [How is code coverage measured?](https://github.com/jacquard-autograder/jacquard-examples/blob/main/README.md#how-is-code-coverage-measured)
* [What is cross-testing?](https://github.com/jacquard-autograder/jacquard-examples/blob/main/README.md#what-is-cross-testing)
* [Why was the name "Jacquard" chosen?](https://github.com/jacquard-autograder/jacquard-examples/blob/main/README.md#why-was-the-name-jacquard-chosen)
* [Where can I view the Javadoc?](https://github.com/jacquard-autograder/jacquard-examples/blob/main/README.md#where-can-i-view-the-javadoc)
* [Where can I get support?](https://github.com/jacquard-autograder/jacquard-examples/blob/main/README.md#where-can-i-get-support)


## How do I make sure I have the latest version of the Jacquard library?

Expand Down Expand Up @@ -51,7 +56,7 @@ Autograder.Builder.getInstance()
.build();
```

See also the [Autograder configuration chapter](https://northeastern.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=ba36573d-dd4a-493d-8b3d-b06a0181c9ff&start=15) (0:15-2:06) from [Taking a first look at Homework 1](https://northeastern.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=ba36573d-dd4a-493d-8b3d-b06a0181c9ff).
See also the [Autograder configuration chapter](https://northeastern.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=6b00de8f-4abe-49a6-a348-b12e0012f37b&start=15) (0:15-2:06) from [Example 2.1: Going through a more complicated AutograderMain](https://northeastern.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=6b00de8f-4abe-49a6-a348-b12e0012f37b).

## How do I use Checkstyle?

Expand All @@ -67,7 +72,7 @@ CheckstyleGrader checkstyleGrader = new CheckstyleGrader(
5.0); // maximum penalty/points
```

See also the [`CheckstyleGrader` javadoc](http://jacquard.ellenspertus.com/com/spertus/jacquard/checkstylegrader/CheckstyleGrader.html).
See also the [`CheckstyleGrader` javadoc](https://jacquard.ellenspertus.com/com/spertus/jacquard/checkstylegrader/CheckstyleGrader.html).

We recommend putting your configuration file in your project's `config/`
directory so it is copied to Gradescope. We also recommend sharing it with
Expand All @@ -93,10 +98,10 @@ built in to PMD, such as [`category/java/bestpractices.xml`](https://github.com/

Jacquard's [PMDGrader](https://jacquard.ellenspertus.com/com/spertus/jacquard/pmdgrader/PmdGrader.html)
has two static factory methods:
* [`createFromRuleSetPaths()`](http://jacquard.ellenspertus.com/com/spertus/jacquard/pmdgrader/PmdGrader.html#createFromRuleSetPaths(double,double,java.lang.String...)),
* [`createFromRuleSetPaths()`](https://jacquard.ellenspertus.com/com/spertus/jacquard/pmdgrader/PmdGrader.html#createFromRuleSetPaths(double,double,java.lang.String...)),
which lets you specify one or more rulesets to be used in their entirety [used in [Jacquard Example 0](https://github.com/jacquard-autograder/jacquard-example0)]
* [`createFromRules()`](http://jacquard.ellenspertus.com/com/spertus/jacquard/pmdgrader/PmdGrader.html#createFromRules(double,double,java.lang.String,java.lang.String,java.lang.String...)),
which lets you specify one ruleset and one or more rules from that ruleset [used in Jacquard Example 2]
* [`createFromRules()`](https://jacquard.ellenspertus.com/com/spertus/jacquard/pmdgrader/PmdGrader.html#createFromRules(double,double,java.lang.String,java.lang.String,java.lang.String...)),
which lets you specify one ruleset and one or more rules from that ruleset [used in [Jacquard Example 2](https://github.com/jacquard-autograder/jacquard-example2)]

There are PMD plugins for [IntelliJ](https://plugins.jetbrains.com/plugin/1137-pmd) and [Eclipse](https://marketplace.eclipse.org/category/free-tagging/pmd).

Expand All @@ -119,15 +124,15 @@ opposed to the cross-tester) must be annotated with [`@GradedTest`](https://jacq
attribute `visibility` has the default value [`Visibility.VISIBLE`](https://jacquard.ellenspertus.com/com/spertus/jacquard/common/Visibility.html#VISIBLE) but
can be set to any other visibility. This code is from [Jacquard Example 0](https://github.com/jacquard-autograder/jacquard-example0):
```java
@Test
@GradedTest(name = "works for empty list", points = 5.0, visibility = Visibility.AFTER_PUBLISHED)
public void iteratorOverEmptyList() {
FavoritesIterator<String> iterator = new FavoritesIterator<>(favoriteHotSauces0);

// No items should be returned.
assertFalse(iterator.hasNext());
assertThrows(NoSuchElementException.class, () -> iterator.next());
}
@Test
@GradedTest(name = "works for empty list", points = 5.0, visibility = Visibility.AFTER_PUBLISHED)
public void iteratorOverEmptyList() {
FavoritesIterator<String> iterator = new FavoritesIterator<>(favoriteHotSauces0);

// No items should be returned.
assertFalse(iterator.hasNext());
assertThrows(NoSuchElementException.class, () -> iterator.next());
}
```

### Other results
Expand All @@ -137,38 +142,101 @@ The visibility level can be set for all other types of autograder results throug
The visibility level of a generated [`Result`](https://jacquard.ellenspertus.com/com/spertus/jacquard/common/Result.html) can be mutated by calling the [`changeVisibility(Visibility visibility)` instance method](https://jacquard.ellenspertus.com/com/spertus/jacquard/common/Result.html#changeVisibility(com.spertus.jacquard.common.Visibility)) or [`Result.changeVisibility(List<Result> results, Visibility visibility)`](https://jacquard.ellenspertus.com/com/spertus/jacquard/common/Result.html#changeVisibility(java.util.List,com.spertus.jacquard.common.Visibility)), as shown:

```java
// Use the default configuration, which includes full visibility.
Autograder.init();
final Target target = Target.fromClass(FavoritesIterator.class);
List<Result> results = new ArrayList();

// Checkstyle results should be fully visible.
CheckstyleGrader checkstyleGrader = new CheckstyleGrader(
"config/checkstyle-rules.xml",
1.0,
5.0);
results.addAll(checkstyleGrader.grade(target));

// PMD results should be visible only after the due date.
PmdGrader pmdGrader = PmdGrader.createFromRules(
1.0,
5.0,
"category/java/bestpractices.xml",
"MissingOverride");
List<Result> pmdResults = pmdGrader.grade(target);
// Change visibility before adding to results.
Result.changeVisibility(pmdResults, Visibility.AFTER_DUE_DATE);
results.addAll(pmdResults);
// Use the default configuration, which includes full visibility.
Autograder.init();
final Target target = Target.fromClass(FavoritesIterator.class);
List<Result> results = new ArrayList();

// PMD results should be visible only after the due date.
PmdGrader pmdGrader = PmdGrader.createFromRules(
1.0,
5.0,
"category/java/bestpractices.xml",
"MissingOverride");
List<Result> pmdResults = pmdGrader.grade(target);
// Change visibility before adding to results.
Result.changeVisibility(pmdResults, Visibility.AFTER_DUE_DATE);
results.addAll(pmdResults);
```

## How is code coverage measured?

Code coverage is measured using [JaCoCo](https://www.jacoco.org/jacoco/index.html). We recommend
having students run JaCoCo inside IntelliJ or Eclipse, because the plugins show which lines of
code are exercised by the tests.

When creating a [`CodeCoverageTester`](https://jacquard.ellenspertus.com/com/spertus/jacquard/coverage/CodeCoverageTester.html),
a [`Scorer`](https://jacquard.ellenspertus.com/com/spertus/jacquard/coverage/Scorer.html) must be
provided to convert the line and branch coverage percentages into points. The concrete scorers are
provided:
* [`LinearScorer`](https://jacquard.ellenspertus.com/com/spertus/jacquard/coverage/LinearScorer.html),
which uses a linear function of the line and branch coverage percentages
* [`LinearBranchScorer`](https://jacquard.ellenspertus.com/com/spertus/jacquard/coverage/LinearBranchScorer.html),
which uses a linear function of the branch coverage percentage (ignoring line coverage)
* [`LinearLineScorer`](https://jacquard.ellenspertus.com/com/spertus/jacquard/coverage/LinearLineScorer.html),
which uses a linear function of the line coverage percentage (ignoring branch coverage)
If you want to write your own scorer, we suggest viewing [`LinearScorer.java`](https://github.com/jacquard-autograder/jacquard/blob/main/src/main/java/com/spertus/jacquard/coverage/LinearScorer.java).

## What is cross-testing?
Cross-testing is my term for running multiple sets of tests against multiple implementations.
Most autograders only run instructor tests against student code. Jacquard also supports running
student tests against multiple versions of instructor code.

Cross-testing using submitted code is specified by a CSV file, such as
[Example 2's `student-tests.csv`](https://github.com/jacquard-autograder/jacquard-example2/blob/main/src/main/resources/student-tests.csv):

| | student | correct | buggy |
|---: | :--: | :--: | :--: |
| size | 10 | 5 | -5 |
| concat | 20 | 10 | -10 |

The header and first row mean:
* If the tests do not report any errors on the implementation of the `size()` method in the `student` package, 10 points are earned.
* If the tests do not report any errors on the implementation of the `size()` method in the `correct` package, 5 points are earned.
* If the tests do report an errors on the implementation of the `size()` method in the `buggy` package, 5 points are earned.

The negative signs in the "buggy" column indicate that the tests are inverted (i.e., points are earned if they fail).

Test names must start with the name of the method under test, such as "sizeWorksForEmptyList()".

This excerpt from [Example 2's `main()` method](https://github.com/jacquard-autograder/jacquard-example2/blob/main/src/main/java/student/AutograderMain.java#L79)
shows how the cross-tester is created and run:
```java
// Create CrossTester to run student tests on:
// * student code (20 points)
// * hidden correct implementation (15 points)
// * hidden buggy implementation (15 points)
// Grading detail is in student-tests.csv.
CrossTester crossTester = new CrossTester(
student.ILOSTest.class, // the test to run
"student-tests.csv" // the name of the CSV file
);
results.addAll(crossTester.run());
```

See also the [Example 2 cross-tester video](https://northeastern.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=165ca9fa-98eb-4f0f-8841-b069013430c5).

## Why was the name "Jacquard" chosen?

The CSV files used for cross-testing made me think of looms, such as the [looms created by
Joseph Marie Jacquard](https://en.wikipedia.org/wiki/Jacquard_machine), which were
controlled by punched cards so play an important role in computing history. Also, the
starting letters correspond to Java or Java Autograder. Claude.ai suggested
this backronym: _Java Assignment Checking with Quality Unit-testing, Analysis, Reporting, and Diagnostics_.
this backronym:

* **J**ava
* **A**ssignment
* **C**hecking with
* **Q**uality
* **U**nit-testing,
* **A**nalysis,
* **R**eporting, and
* **D**iagnostics

## Where can I view the Javadoc?

The Javadoc is available at [https://jacquard.ellenspertus.com/](https://jacquard.ellenspertus.com/).
There are also linked badges at the bottom of Markdown pages, such as this one.

## Where can I get support?

Expand Down

0 comments on commit 3fd52c1

Please sign in to comment.