[RFC007] Migrate the parser to the new AST #2083

Instead of elaborating piecewise definitions (such as `{foo.bar = 1, foo.baz = 2}`) directly at the parsing stage, this commit makes the new AST closer to the source language by making record a list of field definition, where the field "name" can be a sequence of identifiers and strings. This representation is used internally by the parser; we now make it the default in the AST, such that the migration of the parser won't have to do this elaboration at all. The elaboration is offloaded to the conversion to `RichTerm`, which happens in the `ast::compat` module. This makes the AST closer to the source language. The first motivation is that it'll be better for the LSP, where some open issues on the tracker are caused by the inability to trace what the LSP get back to the original piecewise definitions. The second reason is that we can't actually elaborate a piecewise definition while staying in the new AST correctly as of today: the new AST only has one record variant, which is recursive by default, but this doesn't match the way recursion and scoping work for piecewise definition. For example, `{foo.bar = 1, baz.foo = foo + 1}` works fine in today's Nickel (evaluate to `{foo = {bar = 1}, baz {foo = 2}}`), but if we elaborate it in the new AST naively, we'll get an infinite recursion: `{foo = {bar = 1}, baz = {foo = foo + 1}}`. Mailine Nickel currently uses a non recursive `Record` for that, but we don't want to introduce such "runtime dictionary" in the new AST as they can't be expressed in the source language. Instead, we rather keep record as defined piecewise and will do further elaboration when needed, during typechecking, future compilation, or in the meantime when converting the new AST representation to mainline Nickel.

First stab at making the parser compatible with the new AST representation (`bytecode::ast::Ast`). This is a heavy refactoring which required to update most of `parser::uniterm` and `parser::utils` as well as `grammar.lalrpop`. The current version is far from compiling; fixing compiler errors is planned in follow-up work.

As we move toward a bytecode compiler and a bytecode virtual machine, we are replacing the left part of the pipeline with the new AST representation. The bytecode module was previously gated by an experimental feature, thea idea being that this feature would enable the whole bytcode compiler pipeline. However, for now, we only have a new AST representation, and it's being used in the mainline Nickel parser (and soon, in the typechecker, etc.). Thus we need access to the new AST representation by default, and it doesn't make much sense to gate it behind a feature. We'll reintroduce the feature once we have a prototype compiler and a bytecode virtual machine, when it will then make sense to use the feature to toggle between the legacy tree-walking interpreter and the new bytecode compiler.

…esolution for RepeatSep1)

…cords)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC007] Migrate the parser to the new AST #2083

[RFC007] Migrate the parser to the new AST #2083

Commits on Nov 21, 2024

[RFC007] Migrate the parser to the new AST #2083

Are you sure you want to change the base?

[RFC007] Migrate the parser to the new AST #2083

Commits on Nov 21, 2024