High number of unique failures for Rhino CompilerTest

[ihayet]

I was running JQF+Zest on the Rhino CompilerTest example. I noticed that within 10m JQF reported 60 unique failures. The final status screen is as follows:

Semantic Fuzzing with Zest
--------------------------

Test name:            com.rhinoCompilerApp.rhinoCompilerApp.CompilerTest#testWithGenerator
Instrumentation:      Janala
Results directory:    /Users/...
Elapsed time:         10m 0s (max 10m 0s)
Number of executions: 28,755 (no trial limit)
Valid inputs:         17,675 (61.47%)
Cycles completed:     0
Unique failures:      60
Queue size:           717 (0 favored last cycle)
Current parent input: 316 (favored) {455/640 mutations}
Execution speed:      40/sec now | 47/sec overall
Total coverage:       5,728 branches (8.74% of map)
Valid coverage:       5,458 branches (8.33% of map)

The majority of the unique failures were associated with the following exception message:

Exception:
java.lang.IllegalStateException: FAILED ASSERTION: unexpected token: LP

I further checked one example input that caused this exception. The example input is as follows:

Example of Generated Input:
throw (([(556)]) -= (((989)[(s_0)]) - (new ({s_0: (k_1)}))))

There seems to be a SyntaxError in the generated input since the left-hand side of the assignment statement is not an identifier but an expression. Should the testWithString test method in the CompilerTest class catch the IllegalStateException under an assumption? Is a high number of unique failures expected for the Rhino CompilerTest? Any help will be highly appreciated. Thanks in advance.

[rohanpadhye]

Hello. Thanks for this note. Looking at the JS generator, I see that the assignment operators are treated as binary tokens similar to any other binary operator. This leads to syntactically incorrect JS in some cases. I don't recall the exact reason why the generator is designed this way (it was written ~6 years ago), but it should be possible to fix it pretty easily.

The main thing will be to add a new generateAssignmentStatement and insert it in the generateStatement logic. The assignment generator should only generate identifiers or array/field expressions on the LHS.

Feel free to open a PR if you want to give this a go.

[rohanpadhye]

Fixed by #250.