Add support for "dialects"

[nojb]

Following-up on #2373 and @diml's comment, we propose the introduction of a (dialect ...) stanza. Full form is:

(dialect
  (name <name>)
  (interface
    (extension <string>)
    (preprocess <action>)
    (format <action>))
  (implementation
    (extension <string>)
    (preprocess <action>)
    (format <action>)))

• Fields preprocess and format are optional, in which case it is assumed that the files in question are already valid OCaml (and/or do not need special formatting).
• Dialects are uniquely identified by their name on a given project
• Each dialect has exactly two associated extensions: one for interfaces and one for implementations. The extensions are unique among all dialects of a project.

[nojb]

In the first few commits I added internal plumbing to reimplement the existing logic for OCaml/Reason (not yet the formatting part).

[ghost]

I like the name dialect BTW :)

[emillon]

If I understand it correctly, the interface expected for (format <action>) is using stdin/stdout. Unfortunately, some formatters like ocamlformat don't support this. Do you think there's a way to support that?

[nojb]

If I understand it correctly, the interface expected for (format <action>) is using stdin/stdout. Unfortunately, some formatters like ocamlformat don't support this. Do you think there's a way to support that?

• dune preprocessing actions write to stdout so it makes sense to do the same here
• if you are formatting ocaml code (ie. you are not specifying a preprocess action for the dialect) you do not need to actually write the formatting action and dune knows how to invoke ocamlformat
• why doesn't ocamlformat support writing on stdout ?? 😄

[ghost]

@emillon it's only printing to stdout that's expected. The input file is passed via a variable %{input-file}. For instance:

(format (run ocamlformat %{input-file}))

[nojb]

Ah, sorry for the confusion, I thought the question was about printing to stdout rather than reading from stdin.

[nojb]

This is ready for review.

[nojb]

Using __POS__ here is wrong because in case of error the location points to dune's source code. I guess we want to pass Loc.none somehow.

[rgrinberg]

If you don't want this, I think you can just use make_var and make_text. You can pass any location you want this way.

[nojb]

Thanks, I did this change.

[nojb]

We do not validate that the language names used in the (enabled_for ...) field actually exists (i.e. are either dune or a dialect name). This validation has to happen after the dune-project file has been parsed, but the "extensions" API would need to be extended to allow doing such a thing.

[nojb]

The string list is a list of "extra dependencies" for the formatting rule. This is only used to record the dependency of the ocamlformat rule on .ocamlformat and .ocamlformat-ignore, and is not exposed to the user.

[nojb]

Thanks!

[nojb]

I added some doc for the manual. I didn't know where to put it so I added a new top-level section (called Dialects). But the functionality is pretty minor, so it would probably be better to insert it inside some existing section.

[nojb]

I added a few more tests. I will merge after the CI passes unless someone objects.

[ghost]

I forgot to comment about this, but what do you think about calling this module DB? This is the name we used in other places for this kind of things:

• Lib.DB
• Scope.DB

[nojb]

Good point. I will include this change (and anything else that comes up) in #2411

[ghost]

Thanks, I had a quick look through the code. Apart from the comment about loc, it all looks good to me!

[ghost]

It's better to make loc required here. We can always pass Loc.none as a dummy value, but at least we have to think about what we are doing and whether it is justified.

[nojb]

Yes, will do.

[bikallem]

This is interesting. I am wondering if this could be used to implement a JSX syntax in ocaml without the use of ppx, i.e. JSX syntax which is supported as part of ocaml syntax (like in ReasonML). Here the scope of the dialect is just to convert JSX into function application and nothing more.

More information can be found in ocaml/ocaml#8802.

How would one go about doing it using the dialect support in ocaml? Is there a simple example repo that I can study/look?

Could the following be perhaps my starting point?

1. Fork just the parsing bit from https://github.com/ocaml/ocaml/tree/trunk/parsing.
2. Implement JSX syntax on top of it
3. Implement AST transformation from JSX to ocaml AST.

[nojb]

1. Fork just the parsing bit from https://github.com/ocaml/ocaml/tree/trunk/parsing.
2. Implement JSX syntax on top of it
3. Implement AST transformation from JSX to ocaml AST.

That sounds like it should work; but maintaining your fork of the OCaml parser is probably going to be quite a bit of work.

[bikallem]

That sounds like it should work; but maintaining your fork of the OCaml parser is probably going to be quite a bit of work.

Yep, that is my main worry too. It would have been nice if menhir supported these type of scenarios, i.e. inheriting the ocaml compiler parser without modifying it and only extending certain sections of it in separate .mly(?) file. Much like in OO how child classes can further specialize parent function/methods without having to maintain/modify the parent classes.

This would make the maintenance burden of small dialects like I am proposing much palatable. Any ideas suggestions if this is possible perhaps?

[ghost]

Note that this is exactly what Camlp4 allowed to do; it allowed to extend the syntax of OCaml without forking the parser. Camlp4 is the predecessor of ppx. One issue with this approach is that we ended up with many different dialects and it was impossible for editors and tools to know them all. That's why we introduced the extension point and attribute syntax: to provide a uniform syntax for extending the language.

Given that, I wouldn't recommend extending the syntax in this way. Instead, you could design a syntax based on plain extension points, or embed the foreign syntax in a string: [%jsx{| <div> ... </div> |}].