Implement AST.py

[fubuloubu]

@jacqueswww and I are writing a compiler in Python using the AST module for the front end. We were hoping to evaluate using brython to release the compiler in a JavaScript-compatible format for use online. We have very few dependancies, so it's looking positive for us to adopt brython, but the AST module is one of the requirements that is not implemented yet (we saw it on the roadmap!).

We were wondering how we could help implement this functionality?

[PierreQuentel]

@fubuloubu
If it's on the roadmap, it's a bug ;-)
The ast module in CPython stdlib relies on a C module ; to support it we should develop a Javascript program to parse Python source and generate the AST. It's a lot of work ! It would be very interesting to do, but I put it very far in the todo list.

[jacqueswww]

@PierreQuentel Being able to run vyper in the browser directly would be a huge benefit for vyper, with help of a bounty from Gitcoin and the Ethereum Foundation, we will be putting a bounty on implementing the ast python module in brython (thanks Scott! https://gitcoin.co/profile/ceresstation) soon.

I hope this will in be order? Will be interesting to see if someone picks it up 😸

[fubuloubu]

@PierreQuentel how can we best support work on the AST and integrate with the rest of the work on brython so it is not too disruptive.

It seems to me that work on the ast module could very well streamline the front-end aspects of brython's development, but also cause a lot of heartache if it is implemented poorly.

[PierreQuentel]

@jacqueswww As much as I appreciate the intention, our contributing guide specifies that we don't accept any financial support, and that we won't merge pull requests from contributors who get paid for it. This was discussed in threads on the Google Group (https://groups.google.com/forum/?fromgroups=#!topic/brython/mXyPBP9ZEdQ, https://groups.google.com/forum/?fromgroups=#!topic/brython/eby7xv5o9lo).

@fubuloubu If I started Brython now, it's very likely that I would implement an AST parser and generate the Javascript code based on the resulting tree. For the moment, adding an ast module to the library would be nice since we try to be as close to the Language Reference as possible, but using it to generate the translation to Javascript would mean rewriting py2js.js from the ground up, and it scares me...

[fubuloubu]

Definitely appreciate the immensity of that work. Perhaps we will try separately to construct an appropriate py-ast.js module and loop around with you later during a major revision of brython to see if inclusion could be an option.

Thank you for your time!

[acbart]

Hello, was there any follow up to this development? I am very interested in an implementation of ast.py for Brython. We're you able to create it, Fubuloubu?

[fubuloubu]

@acbart we were able to find a workaround using a different project. Check out https://github.com/jacqueswww/vyper-in-browser

[acbart]

I see, I don't think that direction will work for what I am interested in. For reference, I am a developer for the BlockPy project, which has historically used Skulpt. For various reasons, we are looking into whether Brython would be a suitable replacement for Skulpt. We have a few requirements, some of which we had to hack into Skulpt (e.g., minimal support for MatPlotLib, Requests). This includes an implementation of the ast module. I'm not opposed to building the same thing for Brython, though it would be nice if we could just use someone else's version.

Our Skulpt version of ast uses Skulpt's generated parse tree and maps it back to a Skulpt Python representation (fairly straightforward). It seems to me that the major issue is that py2js.js doesn't generate a true AST in JS, but does something fuzzier? Is that accurate? If so, it seems to me that a straightforward solution might be to use ANTLR with a JS target to create a proper parser that we can then convert to the appropriate Brython representation. I actually happen to have some students working on an ANTLR JS Python parser, so I might have that for free. From there, it would just be a matter of converting the ANTLR representation to Brython's, which doesn't strike me as bad.

We're still exploring whether Skulpt could be swapped out for Brython, but does the above sound reasonable or interesting to the community? I'm happy to share whatever we produce, and I'll point out that we make no money from what we do.

[fubuloubu]

I think the brython maintainers arent interested in adding AST support in the short-term, and it may complicate their development cycle. I am sure that you're more than welcome to build it as a fork, and if others see the value in it, it could be upstreamed.

I think we are happy with pyodide at the moment though, it meets our requirements.

[PierreQuentel]

I can only confirm what I wrote in my last comment : developing an ast module is definitely not on the roadmap, and converting an AST tree to the tree Brython uses to generate Javascript code would be a nightmare.