Add Rollup.js tree-shaking and terser minification

[opichals]

The functionality is off by default so is should not affect users unless they set ROLLUP to true or set (MODULE_)MINIFICATION_LEVEL to TERSER in the job file.

Uses https://github.com/opichals/rollup-plugin-espruino-modules

• Add Config.ROLLUP which replaces loadModules
  The core/modules.js transformForEspruino directly use the
  espruinoRollup.loadModulesRollup instead of loadModules

• Separately add TERSER minification option

• The minified libs/rollup/espruino-rollup.browser.js bundle currently commited is 907K. The libs/rollup is built itself by npm run build command (which by itself uses rollup to create the browser suitable bundle).

Perhaps this PR could be deemed superior to the #68 as it is considerably smaller in size and by an order of magnitude better in performance.

Fixes the #64 by providing offline minifier method which gives us a possibility to deprecate and eventually decommission the Esprima-based minification.

Using rollup fixes #27

[gfwilliams]

Thanks for this! If I do:

...$ cat flash.js
var  on = false;
setInterval(function() {
  on = !on;
  LED1.write(on);
}, 500);
...$ espruino -p /dev/ttyACM0 --config MINIFICATION_LEVEL=TERSER -w flash.js --verbose

Everything works great - it seems to minify fine to var t=!1;setInterval(function(){t=!t,LED1.write(t)},500);.

However adding --config ROLLUP=1 as well, I get var n=!1;setInterval(function(){n=!n,LED1.write(n)},500), - so there's a comma stuck on the end instead, which breaks the upload.

Any thoughts?

How do you feel about just using espruino-rollup.browser.js on node.js as well, given it'd be in GitHub and probably NPM anyway? We could then stick rollup and rollup-plugin-espruino-modules as dev dependencies. It feels like it'd be easier from the point of view of testing as then we're testing the exact same code. If one works we could be more confident the other does as well.

I know we don't do that for esprima at the moment but it seems like this would be a great opportunity to ditch that and clean things up.

Looking at https://github.com/opichals/rollup-plugin-espruino-modules it looks like you implement your own code to get modules, so (for instance) if you're using the project sandbox on the chrome web app and you turn on rollup then it looks like wouldn't find modules any more. Or if you stick a module in a modules directory and use the CLI then it works until you turn on rollup, then it can't find the module.

There seems to be quite a lot of rollup-plugin-espruino-modules (espruino-modules-plugin.js and github-modules-plugin.js) that's actually dedicated to doing stuff that EspruinoTools does already (like loading modules) - I guess it's so that rollup-plugin-espruino-modules can run on its own without needing EspruinoTools.

How difficult do you think it'd be to pull just the main bit of rollup-plugin-espruino-modules into EspruinoTools itself and to then use the internal EspruinoTools functionality to get the modules? It seems like it's not that painful.

To get a module it's just Espruino.callProcessor("getModule", { moduleName:fullModuleName, moduleCode:undefined }, callback), and isBuiltIn could also be exported from Espruino.Core.Modules so that built-in modules didn't have to be loaded as well.

[opichals]

there's a comma stuck on the end instead, which breaks the upload

Just fixed by opichals/rollup-plugin-espruino-modules@4d00e52 .

How do you feel about just using espruino-rollup.browser.js on node.js as well

This is not possible. Normally the internally used rollup-plugin-commonjs as well as the rollup default resolver use the node.js fs modules directly. The .browser.js bundle replaces the node native fs with memfs and completely emulates all filesystem access in memory. So rollup thinks it is running on top of fs but in fact it is running inside the espruino-rollup.browser.js.

if you're using the project sandbox on the chrome web app and you turn on rollup then it looks like wouldn't find modules any more

You are right here. I didn't know about the sandbox feature at all till now.

if you stick a module in a modules directory and use the CLI then it works until you turn on rollup, then it can't find the module.

This should work just fine. It does check the modules folder first and if not found there it downloads the module file from the www.espruino.com/modules URL.

[opichals]

I implemented the module resolution logic as well as the built-in module check because my thinking was rather in reverse to how you approach it.

I thought we could provide a complete Rollup.js suite for Espruino to be able to separate the JavaScript processing from the rest of the EspruinoTools' internals. The EspurinoTools would rather plugin into the rollup eventually. If you'd find it the idea suitable I am ready to contribute the whole repo to live under the Espruino organization on github for you to have it under control.

That said it should not be too much work to make a rollup resolver based on the Espruino.callProcessor("getModule"...) to get it all working with the IDE sandbox feature and also for the offline mode to work I think. It's just I'd rather like to see this inside the rollup-plugin-espruino-modules so that it would all work when used with just the rollup CLI itself of which the resulting bundle I'd simply pipe over to the board.

What do you think?

[opichals]

Drafted the getModule/getURL integration in another branch for now: opichals/EspruinoTools@add-rollup...opichals:add-rollup-getModule

[gfwilliams]

Thanks! Since you're using memfs I guess it'd run fine on Node using getModule/etc?

I guess the real benefit of using the module would be to be able to use NPM packages when running on Node, and you wouldn't get that with espruino-rollup.browser.js though.

Still, It's great at the moment that the CLI and Web IDE on all platforms re-use all the same code - and that makes testing a bug finding much easier. It'd suck to go backwards and suddenly start having issues that occurred in the IDE but not the CLI (and vice-versa).

The EspruinoTools would rather plugin into the rollup eventually.

I'm really not convinced about that at the moment. What is the benefit of making everything a Rollup plugin, rather than just including Rollup as part of Espruino's processing pipeline?

EspruinoWebIDE is basically built on top of EspruinoTools, using the same plugin system. Then there's a lot of hidden behind-the-scenes stuff, like figuring out which board is connected and loading the JSON (and getting the list of modules which comes from the board itself), then formatting the output code so it can be parsed correctly by Espruino's REPL and tagging it all with line numbers so the debugger can work. Not to mention things like the JS compiler and assembler.

What about writing the code to the board itself, and providing a REPL into it? How would Rollup provide those?

I imagine it's going to take me a few weeks of learning/porting/bugfixing/support to get sorted, and then there's going to be a bunch of ongoing effort from having to support maybe 10 or so extra NPM packages and GitHub repositories for all the extra Espruino plugins - so from my point of view there's got to be a big benefit for Espruino users.

[opichals]

Since you're using memfs I guess it'd run fine on Node using getModule/etc?

Ah, yes, it should. Will try.

re-use all the same code - and that makes testing a bug finding much easier.

Not absolutely sure about that though with a 900K minified bundle.

The EspruinoTools would rather plugin into the rollup eventually.

I'm really not convinced about that at the moment. What is the benefit of making everything a Rollup plugin, rather than just including Rollup as part of Espruino's processing pipeline?

Well it would make life easier for JavaScript newcomers. They know or heard of rollup already. They would be able to inspect the default rollup configuration and change it at any point to e.g. include babel or any other necessary stuff. All the things you are writing about, compiler and assembler, are IMO possible in Rollup (either through an API call or building it all in using WASM or asm.js). From my point of view EspruinoTools could only contain the board inspection and communication parts. All the rest is being taken care of by transpilers and bundlers in the common JS world. I think it would be beneficial to be as close to that world as possible.

Having said all the above I very much understand your maintenance concern. I am going to get getModule/getURL parts into this PR and let's work on the remaining details to get it in.

As for the rollup-plugin-espruino-modules I'd leave and integrate with the code at github and after we squash any initial issues I'd expose it as an npm.

[gfwilliams]

That sounds great - thanks!

I think you're right about the bundle though - hopefully the two bits of code should be identical? Just one is minified/bundled, one isn't? You're right, debugging anything in rollup when it's minified will be a nightmare - can we leave it as-is?

[opichals]

debugging anything in rollup when it's minified will be a nightmare - can we leave it as-is?

Do you mean leaving the browser bundle unminified? That would balloon its size quite a bit (I think around 3MB) which I am not sure is a good thing for the Web IDE.

[gfwilliams]

Do you mean leaving the browser bundle unminified?

No, just what you have already - dependencies on rollup in package.json, and we can .npmignore the browser bundle for the NPM package.

[opichals]

the two bits of code should be identical?

They are bundled by rollup itself with debug and fs replacements.

It is only necessary to not to forget to do 'npm run build' after any libs/rollup/* change.

[opichals]

@gfwilliams Pushed the changes, please review.

[opichals]

@gfwilliams What do you think about it as it is now? Do you see any issue?

[gfwilliams]

Thanks for the prod. I guess I'd prefer that rollup-plugin-espruino-modules was actually part of EspruinoTools, but I can see why you want to make it standalone in its own right.

Let's merge and see what happens :)

[opichals]

@gfwilliams Yes, I like the fact it is possible to use rollup CLI and then only send the results to the board via the EspruinoTools as a separate step.

But as I suggested earlier I would be for moving the rollup-plugin-espruino-modules under the Espruino github organization if you find any value in such an arrangement.