feat: add option for building both a legacy and modern browser build

[bryceosterhaus]

Heres a general idea and POC for https://issues.liferay.com/browse/LPS-118803 to enable a "differential build" for both legacy and modern browsers.

One tricky part of this is that we also need to add some sort of filter on the DXP side for serving the correct bundle. Right now, every module seems to serve the js a little differently and wasn't sure how we best want to do this. Any ideas?

Also, note that this is only for webpack right now and so we may want to look into doing the same for just for if someone just uses our build script without webpack.

[jbalsas]

Could you tell if this makes any noticeable difference in our current DXP output?

[bryceosterhaus]

Looking at just frontend-js-web

Current Bundle:
global.bundle.js 138 KiB

With differential build:
global.bundle.js 138 KiB
modern_global.bundle.js 129 KiB

So really only a 9kb difference...

I also tested with lighthouse and the results didn't seem noticeable. Although it could be because I am only testing the frontend-js-web module.

[jbalsas]

Well... that's a bit disappointing, isn't it? 😂

[bryceosterhaus]

I did notice that for some reason the modern build isn't being minified, so that may have part to do with its size. Im looking into why it doesnt compress like the other build.

[bryceosterhaus]

@jbalsas do you know if there would be a particular avenue in portal for filtering the js requests depending the browser? Otherwise we would need to add support on a module-by-module basis and add a dynamic filter for checking how to include the js

[jbalsas]

@jbalsas do you know if there would be a particular avenue in portal for filtering the js requests depending the browser? Otherwise we would need to add support on a module-by-module basis and add a dynamic filter for checking how to include the js

Yeah, this should be no problem at all... now the question is... is it worth it?

With IE11 going away pretty soon (🤞 ) and defaults providing such a small improvement... wondering what else is out there that could help us... maybe our overhead is coming from something else, like duplicated babel helpers and so on... 🤔

[wincent]

One tricky part of this is that we also need to add some sort of filter on the DXP side for serving the correct bundle. Right now, every module seems to serve the js a little differently and wasn't sure how we best want to do this. Any ideas?

In an ideal world, I'd think that we would divide the build into:

• (A) Browsers that natively support real ES modules.
• (B) Everything else.

That seems to be the most future-proof cut line, because it basically corresponds to "evergreen browsers" vs "the rest". Then we'd make them available via the appropriate script tags:

<script type="module" src="some-bundle.mjs" /></script>
<script nomodule src="some-bundle.js" /></script>

Browsers in group "A" would download the ".mjs" bundle and ignore the nomodule script, and browsers in group "B" would do the opposite. Browsers in group "A" support all sorts of delicious stuff, like async/await with no transpilation.

Obviously, in the presence of our bundler, loader, and Liferay.Loader.require API etc we'd need to translate the above into "Liferay"-speak (and we would transpile dynamic import('thing').then(...) into Liferay.Loader.require() or whatever we needed under the hood too, so that people wouldn't have to think so explicitly about being in Liferay-world), but the core idea is the current "standard"; some references:

• https://jakearchibald.com/2017/es-modules-in-browsers/
• https://web.dev/serve-modern-code-to-modern-browsers/
• https://philipwalton.com/articles/deploying-es2015-code-in-production-today/

(An example webpack config for where I am doing this on my blog).

[bryceosterhaus]

Did some tests with a few different modules to get an idea of what sort of sizes we would get with esmodules.

Simply just added a config with

{
	presets: [
		[
			'@babel/preset-env',
			{
				targets: {
					esmodules: true,
				},
			},
		],
	],
}

ESModules vs The Rest™️

frontend-js-web
ESModules: 2.2mb
The Rest: 2.3mb

segments-web
ESModules: 588kb
The Rest: 668kb

app-builder-web
ESModules: 1.5mb
The Rest: 1.7mb

Overall it seems to help cut some costs and if it was applied across portal, we could see some benefits. However, I'm not sure what the overall cost would be to apply this. Throughout portal we tend to add JS in a variety of ways, any idea where we might investigate adding the two script tags? Or would this potentially just be a giant filter for ever js file?

[wincent]

Throughout portal we tend to add JS in a variety of ways, any idea where we might investigate adding the two script tags? Or would this potentially just be a giant filter for ever js file?

Haven't analyzed this at all, but I don't imagine this being a filter. I think we'd have to take a good look at the specific places we're emitting script tags and look at what would take for each of them to become "bundler/ESModule-aware" (by which I mean, that our build process would emit the two formats, and the places where we emit script tags would need to be aware when the alternative formats were available and inject both accordingly).

Given that most of the JS actually gets loaded by the loader and not by script simple script tags rendered into the HTML by the server though, the solution would likely include changes to make the loader "aware" as well.

cc @izaera: who basically built everything related to JS loading at Liferay.

[jbalsas]

Haven't analyzed this at all, but I don't imagine this being a filter. I think we'd have to take a good look at the specific places we're emitting script tags and look at what would take for each of them to become "bundler/ESModule-aware" (by which I mean, that our build process would emit the two formats, and the places where we emit script tags would need to be aware when the alternative formats were available and inject both accordingly).

Definitely not a filter. We usually do this through the Servlet that serves the files. The most obvious example being RTLServlet which will serve a different main_rtl.css file when main.css is requested for an RTL language.

That being said... IE11 might be going away in 7.4 so we might not even need a dual-build?

[wincent]

ESModules vs The Rest™️

frontend-js-web
ESModules: 2.2mb
The Rest: 2.3mb

One other thing about this; it's not only about size. There's also a benefit in terms of the compiled output being:

• Easier to read for humans.
• More likely to be optimized in the JS engine (because the engine is optimized to deal with "idiomatic" code).

Example input:

async function thing() {
  await something();
}

Will lead Babel to produce this horrible thing:

"use strict";

function asyncGeneratorStep(gen, resolve, reject, _next, _throw, key, arg) { try { var info = gen[key](arg); var value = info.value; } catch (error) { reject(error); return; } if (info.done) { resolve(value); } else { Promise.resolve(value).then(_next, _throw); } }

function _asyncToGenerator(fn) { return function () { var self = this, args = arguments; return new Promise(function (resolve, reject) { var gen = fn.apply(self, args); function _next(value) { asyncGeneratorStep(gen, resolve, reject, _next, _throw, "next", value); } function _throw(err) { asyncGeneratorStep(gen, resolve, reject, _next, _throw, "throw", err); } _next(undefined); }); }; }

function thing() {
  return _thing.apply(this, arguments);
}

function _thing() {
  _thing = _asyncToGenerator( /*#__PURE__*/regeneratorRuntime.mark(function _callee() {
    return regeneratorRuntime.wrap(function _callee$(_context) {
      while (1) {
        switch (_context.prev = _context.next) {
          case 0:
            _context.next = 2;
            return something();

          case 2:
          case "end":
            return _context.stop();
        }
      }
    }, _callee);
  }));
  return _thing.apply(this, arguments);
}

But given an ESModule's build it will make something like:

"use strict";

async function thing() {
  await something();
}

Which is night and day...

[wincent]

Definitely not a filter. We usually do this through the Servlet that serves the files. The most obvious example being RTLServlet which will serve a different main_rtl.css file when main.css is requested for an RTL language.

I don't think it's only about what the servlet returns; it's that the tag itself should be different, so that the browser knows to interpret it as a module (ie. the type in <script type="module" ... matters).

That being said... IE11 might be going away in 7.4 so we might not even need a dual-build?

A tantalizing possibility, but the mobile browsers in the compatibility matrix (currently Chrome and Safari "latest") might have different capabilities (not sure).

[jbalsas]

I don't think it's only about what the servlet returns; it's that the tag itself should be different, so that the browser knows to interpret it as a module (ie. the type in <script type="module" ... matters).

Wouldn't that require us to stop AMD'zing the things? I don't think that's on the table right now, we'll still be wrapping modules in an AMD wrapper, so things should still be loaded via a regular <script> tag?

[wincent]

Wouldn't that require us to stop AMD'zing the things? I don't think that's on the table right now, we'll still be wrapping modules in an AMD wrapper, so things should still be loaded via a regular <script> tag?

Yeah, that's why I said:

In an ideal world...

This really would require a top-to-bottom look at how we bundle and deliver JS to the client. We may not be able to do this now, but I imagine that in 5 years when then entire internet is delivering everything as ESModules over HTTP/3 and we're still using AMD modules plus a bunch of very proprietary code for bundling/serving/loading (even if we're delegating a lot of the underlying work to webpack/Babel), the pressure will be on us to "modernize".

[jbalsas]

In an ideal world...

[...] but I imagine that in 5 years when then entire internet is delivering everything as ESModules over HTTP/3 [...]

That's a possibility... we made ourselves this same question 5 years ago, but it was then HTTP/2 what we thought internet would be using instead 😂

Definitely need to keep an eye on this and try to take a holistic approach to modernize our platform. 👍

[wincent]

we made ourselves this same question 5 years ago, but it was then HTTP/2 what we thought internet would be using instead

And it kind of is, isn't it? HTTP/2 is used by 47.9% of websites (and 100% of sites that care about performance).

[jbalsas]

And it kind of is, isn't it? HTTP/2 is used by 47.9% of websites (and 100% of sites that care about performance).

Fair point!

[izaera]

Funny: I was asking myself too yesterday when would we support HTTP2...

Regarding <script module> we have thought about it (like @jbalsas says) several times, but one of the biggest problems with the current approach is that we would need thousands of those tags if we wanted to have one JS file per module. That would be similar to the current development setup which would be quite slow for a production site (unless HTTP2-3-whatever comes to the rescue).

However, we should definitely give it a try, though I suspect the majority of npm packages are not yet prepared for that :-( (based on the assumption that everyone seems to optimize their packages for webpack, as far as we've seen until now).

To finish with, I remember having read that there were some subtleties, when importing browser modules, compared to node, webpack and our AMD setup. Subtleties like module names, dynamic requires, and/or exotic operations which I don't remember quite well, but will lead to terrible headaches for sure 😅 .

[jbalsas]

Just for the LOLs... you can see we planned to do this back in DXP Frontend Infrastructure - Vision and Lines of Work for 2016 - 2017

Maybe the HTTP/2 time has come!!

Also for reference, we decided to:

I think we should consider AMP before HTTP2. While both are important, HTTP2 requires that the clients also upgrade their IT infrastructure which is often a very slow process before they are able to fully appreciate the feature. AMP though can be fully implemented by us and then their dev teams and content teams can then leverage the feature.

Needless to say, we did neither :D

[matuzalemsteles]

We probably won't see big leaps in performance between HTTP2 and HTTP3, not yet with recent drafts, when compared to HTTP2. In my humble opinion even if IE will be deleted from the earth phase 🙂, I think it is still worth working on delivering files without babel transformation for more modern browsers, in addition to the benefits of minimally reducing the size of packages, we will still have better performance, the browser will handle modern syntaxes better than babel transformations and we can gain something here.

Even if 5 years from now, it's HTTP3 and it's just downloaded by module, I still see that the use of minification and the maybe single bundle will still be an important factor, entering as an optimization phase or say packaging. That could still decrease the amount of tags, right? if that's what I understand from this conversation 🙂

[wincent]

Even if 5 years from now, it's HTTP3 and it's just downloaded by module, I still see that the use of minification and the maybe single bundle will still be an important factor, entering as an optimization phase or say packaging. That could still decrease the amount of tags, right? if that's what I understand from this conversation 🙂

The thing is, unlike a lot of "conventional websites", a Liferay DXP instance can't possibly build optimal bundles ahead-of-time, because we don't have ahead-of-time knowledge about what portlets are going to appear on a page. So that's why we build our own bundler/loader/registry to do some of the work ahead-of-time and the rest on demand, and still deliver aggregate packages via the "combo" servlet.

So I think that for us, more than most websites, the promise of being able to do this kind of on-the-fly multiplexing over a shared connection is pretty interesting, because it would enable us to jettison a bunch of custom infrastructure and tooling in favor of something that just leveraged standard transport protocols and standard language features.

That's what the advertising campaigns say, anyway. 😂

Footnote: About building "optimal bundles" ahead of time, we've talked about doing some statistical analysis based on access patterns to compute a likely optimal bundle and then refine it based on actual usage, but it would be tricky to pull off. So v2 of the bundler does very little actually "bundling" ahead of time (it's really just preparing individual modules for being bundled later at runtime). v3 of the bundler does a bit more work ahead of time, but it basically leaves it in the users' hands to draw the boundaries between modules.

[bryceosterhaus]

Definitely not a filter. We usually do this through the Servlet that serves the files.

my bad on the confusion, I was using filter in the generic sense of the word and not the java/liferay sense...

Circling back around to the initial idea of modern vs legacy builds and from reading through the responses. It seems like its probably not worth going the route of dual builds since we anticipate(🤞 ) non-evergreen browsers to no longer be supported in future liferay versions? It seems like the question has morphed more into "how will we distribute JS in the future?"

[matuzalemsteles]

The thing is, unlike a lot of "conventional websites", a Liferay DXP instance can't possibly build optimal bundles ahead-of-time, because we don't have ahead-of-time knowledge about what portlets are going to appear on a page. So that's why we build our own bundler/loader/registry to do some of the work ahead-of-time and the rest on demand, and still deliver aggregate packages via the "combo" servlet.

Yes I agree, I understand that we had to do this due to our peculiarity, I already had several discussions with some people here to explain about it 🤪 but considering that we have the modules already included in the Portal and we have more control over this, we could create specific bundles for our main "critical modules", this would at least help the admin part, for portlets things really get more complicated.

So I think that for us, more than most websites, the promise of being able to do this kind of on-the-fly multiplexing over a shared connection is pretty interesting, because it would enable us to jettison a bunch of custom infrastructure and tooling in favor of something that just leveraged standard transport protocols and standard language features.

Yeah, we can really earn a lot here since we send a lot of files.

Footnote: About building "optimal bundles" ahead of time, we've talked about doing some statistical analysis based on access patterns to compute a likely optimal bundle and then refine it based on actual usage, but it would be tricky to pull off. So v2 of the bundler does very little actually "bundling" ahead of time (it's really just preparing individual modules for being bundled later at runtime). v3 of the bundler does a bit more work ahead of time, but it basically leaves it in the users' hands to draw the boundaries between modules.

This seems to me guessjs adjusting at build time instead of doing it at runtime as you were thinking.

[diegonvs]

Just leaving a thought about our performance here:

When having a conversation with @matuzalemsteles, We guessed to can use service workers for initially caching our foundational JS/CSS assets. It may include ClayCSS, the selected theme(unstyled, classic, styled) assets, frontend-js things. I know this feature isn't supported on IE11 but it's widely supported around the internet(exactly 93.85%).

I'm not a fan of the ServiceWorker API. So, I think we could try using WorkBox

[wincent]

We guessed to can use service workers

Shall we discuss this in a separate ticket? (There are a few trade-offs involved, so probably don't want the conversation buried in an unrelated ticket.)

[matuzalemsteles]

Shall we discuss this in a separate ticket? (There are a few trade-offs involved, so probably don't want the conversation buried in an unrelated ticket.)

Yeah, that would be better.