Traversal over streams is unergonomic

[samhh]

Context: #39

Worth exploring ways to simplify and condense it.

[samhh]

Really we want a way to generically traverse over Stream in our recursive datatypes. Unfortunately, because our datatypes are mutually recursive, I think the recursion schemes approach becomes substantially more complex. Not that I've necessarily fully wrapped my head around ordinary recursion schemes. 🙈

Lenses are something else to look at. If we go this route we should probably prefer optics over lens.

[samhh]

This came up again in linting for getStream. It's traversal more generally that's a challenge.

[samhh]

Came up again in mapTokens which could at least reuse some of the stuff defined for flattening.

[samhh]

For better linting output we need to retain source span information
in the AST (#142). A naive implementation might look like this:

type Stream = [(Token, SourceSpan)]

This makes for a relatively small lib/ diff, and all else looks rosy until we contemplate how we write tests. Where now we might write Plaintext "foo", we'd now need to provide source information. And we'd have to do this everywhere in our tests, and it'd be nonsense that we never use anyway.

A similar problem presents when flattening. It's a lossless procedure in which any source information should be thrown away as it's really no longer valid.

This makes the idea of making our AST parameterised, recursion schemes or not, very tempting. In that scenario our tests can continue to target [Token], but our parser output and linting input could be [(Token, SourceSpan)]. Flattening would transform the latter to the former.

Edit: This describes what I want to do: https://www.reddit.com/r/haskell/comments/sxsj4m/what_are_strategies_for_tracking_type_checking/hxu2tsj/