Consider fuzzing syntax changes

[brianrourkeboll]

Proposal

I wonder whether it might be worth considering setting up some kind of process to fuzz (along the lines of this) any changes to the compiler that could affect either:

1. The treatment of existing syntax
2. The treatment of new syntax

I've noticed a pattern of longstanding compiler oddities or inconsistencies as well as of regressions and unintended features accidentally introduced alongside new syntax, both of which might have been caught earlier by fuzzing—that is, generating random programs (FsCheck? Hedgehog?) and asserting whether and how they should compile.

Even some kind of tool that could be run by hand on demand feels like it might be better than nothing.

Problem examples

Here is a quick, non-exhaustive list of examples that fuzzing might have helped catch sooner:

• [Regression] Compiler is unable to compile specific flavor of type check in patterns #16447
• [regression] static member can be defined without member keyword, in unexpected way #16342
• Single tilde ~ in expressions compiles, but shouldn't #16135
• Expressions other than properties/method calls/indexing are allowed after _. in dot-lambdas #16136
• Inconsistent parenthesization requirements for single-argument additional constructors (with new) #16257
• Parenthesized/double unit literal () → (()) required in certain scenarios #16254
• Inconsistent handling of arguments for disposable types #15622
• Compiler does not infer string type for nameof pattern #13377
• No compiler error for instance members on static class; produces invalid IL & causes runtime error #13147
• No compiler error when declaring multiple parameters with the same name in abstract methods #13148
• Compiler allows what looks like a module nested inside of a type, but isn't #10066
• Missing argument compiles without error and throws CLR error during runtime #15955
• Collection of findings related to string interpolation #9893
• Regression: DU case at pattern match site no longer requires parameter #9420
• Regression: indexing with type annotation fails to parse #8844
• Compiler allows non-functions to be bound to active pattern names #17190
• …

A language like F# is perhaps more prone to this kind of problem than many other programming languages, since most of F#'s features are designed to be composable—the language's expression and pattern-based nature, and the fact that expressions and patterns and types can be nested inside of themselves and each other, means that the language's syntagmatic surface area is already quite vast, and every new or updated construct makes it bigger still.

I think it's inevitable that we humans will fail to consider all of the potential syntactic implications of any given change to the parser or addition of a new syntactic construct.

Some questions

1. How do other languages handle this kind of problem?
2. Is it feasible to build and maintain such a tool, or is it known not to be a worthwhile undertaking?