nom 5.0 roadmap

[Geal]

now that I'm done with the experimental phase, it's time to plan the nom 5.0 release.
Here's the rough plan:

• nom 5.0 internal design #878: core redesign: write parsers and combinators as functions, rewrite the macros to use the functions instead. This keeps code backward compatible, improves performance, and the function style is very nice to use.
• removing the verbose-errors feature #887: remove the verbose-errors feature. Instead combinators expect an error type that implements the ParseError trait. The calling site can decide which error type to use (there's a default one of (Input, ErrorKind)). This fixes the issue of additive cargo features, and we'll get much better error handling if needed. See https://github.com/Geal/nom/blob/5.0/examples/json.rs for an example
• use modules to have different versions of some combinators, some that work in streaming mode, some that assume we have all the data. Remove CompleteByteSlice and CompleteStr, since those types are hard to manipulate and don't fully solve the issue
• add float and double parsers that construct the number as it is parsed, instead of the current solution of applying recognize_float and then calling parse::<f64>() on it

I've advanced a lot on the first two points, but it's still going to be a lot of work, so I'm adding issues with small tasks that could be contributed:

• core redesign:
  • architecture:
    • should function combinators return a FnMut? #898: should combinators return a FnMut
  • missing combinators:
    • port bit stream parsers to the new design #892 bit level parsers
    • port byte level parsers to the new design #893 byte level parsers
    • port branch combinators to the new design #894 branch combinators
    • evaluate which combinators to keep in src/macros.rs #895 general purpose combinators
    • move methods and white space combinators in their own crates #896 extract methods and whitespace combinators
    • port multi combinators to the new design #897 multi combinators
    • port sequence combinators to the new design #899 sequence combinators
• error management
  • blanket implementations of ParseError #900 blanket implementations of ParseError
  • Verbose error type implementation #901 verbose error type implementation
• streaming/complete specific versions of combinators
  • list which combinators and parsers require it
• proper implementation of float and double parsers from text representation #902 proper implementation of float and double parsers from text representation
• documentation rewrite: most of the code examples and markdown docs (in doc/) are obsolete, they refer to older versions of nom. Also we might have better ways to write most of those now. The new documentation should also show the different patterns: macros or functions
  • nom 5 code level documentation #914 doc layout
  • Undocumented nom 5.0 functions #915 list of functions to document
  • rewrite the error management guide #933 error management guide
  • rewrite the "making a new parser from scratch" guide #934 parser writing guide

Anything else to add here?

[jamwaffles]

documentation rewrite ... Also we might have better ways to write most of those now.

One pain point I keep hitting with nom < 5 is the lacking docs for the macros list. A lot of the time, I just want to see a quick copypasta of how to apply a combinator. I think it'd be really useful to have most of the usage-level docs alongside the functions/macros themselves in the generated docs, with perhaps guide-level docs as separate documents so they can be longer and more in-depth. I don't know if this is already the plan, but that's my two pence 🙂

[Geal]

the macros are missing a lot of examples, I plan to add a lot more with this version

[jothan]

FWIW, I wrote a PDF parser for lopdf using nom5 and found the experience quite pleasant. The resulting parser is very fast and maintainable.

See https://github.com/J-F-Liu/lopdf/blob/master/src/nom_parser.rs

[MarkMcCaskey]

Following up on what @jothan said, I also just wrote an s-expression parser for wasm contracts using nom5 and had a good experience too!

The last version I had seriously used prior to this was version 2.2 and compared to that, the error messages have improved a ton, which was my biggest pain point. I found that actually using nom was very pleasant; I was able to get things done quickly and easily.

I didn't use any of nom's macros this time because they don't seem to work with the function parsers at all. The biggest issues I ran into were:

• Creating a generic function: I made a higher order function which parses s-expressions (so the same as delimited but allowing for whitespace and comments). The error messages from Rust were pretty intimidating and they guide you to covering your functions with tons of trait-bounds. I ended up with something simple that I'm happy with though.
• Returning errors on bad input, return_error! seems to be exactly what I want, but I haven't been able to find a non-macro version.

I think both of the above can be fixed purely through additional examples.

Anyways, just stopping by to give some feedback and thank you all your work on this project over the years! It's been very useful for me multiple times and seems to be getting even better!

[Geal]

it's good to see that nom 5 is easier to use than previous versions :)
For error management, there's cut that implements one part of return_error's behaviour, which is preventing backtracking (combinators like alt will not try other branches). And you can use context with VerboseError like the JSON example, or make your own error type that implements ParseError

[MarkMcCaskey]

@Geal Thanks! That's exactly what I was looking for.

Unfortunately, when I tried to use cut just now, I ran into issues with the type_length_limit. I think Rust will be fixing this rust-lang/rust#54540

[stavinsky]

Hey! ) I'm very new to nom and started with nom4 but after suggest in gitter I switched to nom5. Everything looks much better. I miss only two things: dbg! and ws! macro. I didn't find how to use it with functions.

And thanks. You are doing great job. I spent not more that few hours to start writing productive code with nom. It's awesome.

[miedzinski]

I also wrote a parser for regular expressions with nom 5 without any prior experience with parser combinators. IME functions seem way easier than macros. :) I didn't look into error handling though, mainly because it wasn't covered in docs.

https://github.com/miedzinski/regex/blob/master/src/ast.rs

[Geal]

nom 5 is now released :)
http://unhandledexpression.com/general/2019/06/17/nom-5-is-here.html
https://www.reddit.com/r/rust/comments/c4nbdu/nom_parser_combinators_50_release_replace_macros/