add #_ discard syntax

[gilch]

Closes #1058.

This still needs docs, a comment macro, and lots of tests. [Edit: and a news mention!] But I'd like to verify my approach first.

The #_ syntax needs to be at the "reader" level. It can't be a simple macro. The lexer is too low-level for this. A regex is enough for ; comments, but the #_ syntax needs to be able to identify Hy forms properly, even when they contain nested brackets. So I added an "empty" production to the grammar in the parser to make it work.

I had to add the "empty" production in a lot of places to avoid shift/reduce warnings. This does complicate the parser a bit. If someone has a more elegant approach, I'd like to see it.

[Kodiologist]

I think the simplest way to do this is with postprocessing. Allow HASHSCORE to parse as a kind of term that leaves a special object, say DeleteWithNextToken, behind. When you're done parsing but before compiling, run through the token stream and delete each instance of DeleteWithNextToken and the following token.

N.B. I'd recommend against changing HASHSTARS to r'#\*\*?', because then Hy will parse something like #**** as #** ** without complaint.

[gilch]

Here's another approach that doesn't change the grammar so much. All term productions now produce a list, and a discarded term produces an empty list.

With this approach (so far), there are places where #_ is allowed in Clojure that aren't allowed here, that is, in between other reader syntax and the discarded form, e.g. '#_() x will cause an error, but is just 'x in Clojure. I could probably make it work with this approach, but I don't know that we need it. [Edit: maybe not. And the errors crash the whole repl. Not good.]

[gilch]

I found a simpler grammar change. This time it only adds a few productions and leaves the existing ones alone.

[Kodiologist]

Seems pretty cool to me.

[gilch]

I'll start rounding it out then. Tests, comment, docs, news. What order would you commit those in?

[Kodiologist]

It doesn't matter very much so long as you commit documentation, NEWS, and tests after (or at the same time as) the corresponding changes to the code. Everything in one big commit is also a reasonable option.

As for the order in which you actually write stuff, the order you listed seems like the most natural one, with "tests for comment" added after comment itself.

[Kodiologist]

I thought that comment was to have the same effect as #_, but it results in a None rather than discarding itself entirely. Is that on purpose? If you wanted to not evaluate a form and return None, I'd expect you to write (when False FORM).

[gilch]

I made this to be Hy's equivalent of Clojure's comment macro, which ignores its arguments and returns nil.

(when False BODY) or similarly (and None FORM) would work, but static analyzers (like Hydiomatic) would be justified in raising a warning that you've probably made a mistake. With comment the intent is clear. It's also easier to type.

That said, why not just use #_ FORM instead? (Usually you would.) I'm not sure, but Clojure hasn't deprecated comment, so it probably has uses. If I had to guess, sometimes you want to "comment out" code, but still need a None place holder. In that case FORM -> #_FORM None could also work. But you'd have to wrap BODY in something anyway, because #_ would only get the first term. Or you might still want the contents to show up after parsing for some reason (maybe for some other tool), even though it's semantically a just a comment that should be a no-op in the Python AST. Maybe something that renders docs from comments in HTML or ReST or Markdown, would want to find them with the help of Hy's parser. Normally we'd prefer docstrings for this kind of thing, but in the literate style, you might want markup comments interspersed with code in places that docstrings aren't allowed.

[Kodiologist]

¯\_(ツ)_/¯ Okay, it's got a dead-simple implementation in any case.

Otherwise, everything looks good to me. You might want tests for #_ in a function call and a dictionary display.

[gilch]

I added more tests.

[Kodiologist]

Please include an example, and write out "Extensible Data Notation". I didn't know what "EDN" stands for, and I bet that most other readers won't, either.

[Kodiologist]

An example of comment would be good, too.