New Assets Build Tool (Assets Manager)

[Skrypt]

Some extra work for me on Christmas holidays.🎅🏼

Assets Manager

Created by @jptissot

Based on Concurrently and Parcel but can also run Vite and Webpack too. This is a non-opiniated build tool which allows to be extended for any asset compiler/bundler that someone may require.

This build tool essentially uses Concurrently instead of Gulp.
Concurrently, is a concurrent shell runner which allows to trigger any possible shell command. It allows to trigger Parcel, Vite, Gulp or any other command that we need for building assets. Everything is written as ES6 modules (.mjs files).

Kind of needed for Vue 3 migration because it needs to use either Vite or Parcel to build as ES6 modules.

Old assets are not compiled as ES6 modules so they don't need these bundlers. For that matter I kept the old gulpfile.js which will be now triggered by Concurrently.

What needs to be done over time is to migrate these javascript files to ES6 modules (.mjs files or .ts files) so that we can compile them as modules. But that's a migration that will happen over time.

Features

• Build everything: yarn build
• Build module by name: yarn build -n module-name
• Watch module by name: yarn watch -n module-name.
• Build with gulp: yarn build -g
• Rebuild with gulp: yarn build -gr
• Host with dev server: yarn host -n module-name.
• Clean folders with yarn clean. Will also clean parcel-cache folder.
• Makes uses of latest yarn version 4.6.0. OC is currently on version 1.something
• Makes use of yarn workspaces which allows to import files from different locations in the app for sharing ES6 modules.
• VS Code launcher debug option added as "Asset Bundler Tool Debug"
• Build assets by tag: yarn build -t tagname
• Gulp pipeline still using GulpAssets.json file
• New Assets.json file definitions for building with new tool.
• Concurrently will retry building up to 3 times making CI build less prone to fail.
• Moving package.json files to Assets folder of modules and themes. This will remove these files from the compiled C# .dll files.
• Removes unnecessary package-lock.json files as we now rely on a single yarn.lock file.
• When using watch sass action it will recursively watch imports in files.

Update

Finally, I'm keeping the old gulp pipeline because there is nothing wrong with it. Also, for backward compatibility with older modules that requires it. I'm using GulpAssets.json for the Gulp build tool and using Assets.json for the new one. This way, no need to migrate to the new build tool. I'm simply triggering gulp rebuild with Concurrently in the new asset build tool. Because that's what it is essentially, a concurrent shell runner. This way, it is going to be a softer migration for those who already have modules that are built with the old gulp pipeline.

Of course, this PR needs to be accepted by community first.

TODO

• More documentation.

Modules to migrate:

• OrchardCore.Resources (Almost everything worked except (Monaco and Trumbowyg RTL)
• OrchardCore.Media (Not a great idea to waste time on migrating this as we are refactoring to Vue 3)
• OrchardCore.Liquid (Monaco Editor Intellisense)

@jptissot @deanmarcussen @aderderian

Fixes #14169. Related to #15740.

[Piedone]

You've been sneakily working on this since April ;).

Fixes #15740 #14169 #6762?

[Skrypt]

Alright, I think I pushed enough efforts on that one. Time to approve it and get it merged.
I want to move to File App as soon as possible. That was the original goal.

@deanmarcussen @Piedone @sebastienros

Please redo review

[Piedone]

Will check this out soon.

[Skrypt]

@sebastienros I can see only around 30 messages on this one (after expanding everything).
Nevermind. I forgot to expand resolved discussions. There is probably 71 messages.

[Piedone]

I'm not competent to comment on the asset manager's code, so I focused on trying it out and using it as any contributor, checking if I find any issues, and reviewing the rest.

What do you mean by?

I want to move to File App as soon as possible. That was the original goal.

Please:

• Apply suggested changes directly so the reviewer doesn't have to eyeball the changes. These resolve themselves after applying them, and that's fine.
• Don't resolve other conversations so it's easier to track for the reviewer. Then, the reviewer will resolve them. There will be a lot here, so this is important, otherwise we won't be able to follow the conversation.
• Feel free to mark conversations that you addressed to keep track of them with an emoji or otherwise, just don't resolve them.

[Piedone]

Are these still necessary?

I'd guess that touching Media is not really important with the Vue 3 migration you are working on, but the other are actually using the new pipeline.

[Skrypt]

Yes, can't migrate them all yet unless refactor them entirely.

[Piedone]

Can you explain why a refactoring is necessary? This is what I see in the Assets.json files (again not counting Media):

• AdminMenu: the remain Gulp pipeline does bundling of two JS files. As we discussed in a meeting, this is not even necessary with HTTP 2 today, so it could be swapped out with a ResourceManagementOptionsConfiguration for the two scripts and a dependency declaration there, but if not, the new Assets Manager supports bundling. Why not just use that? Seems like a 1-1 replacement to me.
• Liquid: I suppose this kind of TS compilation is different than the one you migrated in some other projects?
• Resources: SCSS compilation and JS bundling, which are supported by the new Assets Manager 1-1 too.

[Skrypt]

Feel free to try doing what you just suggested there. On my side, I tried refactoring them using Typescript and compiling them as ES6 assets. Did not work for me so far. We could maybe try compiling these as CommonJS with Parcel, I did not try.

AdminMenu: The remaining asset is the fontawesome iconpicker component and I tried converting it to typescript but it is just not written for doing imports. It uses a JQuery extension method which the TS compiler doesn't understand. Needs to be refactored without JQuery. Else, we could try compiling it as CommonJS with Parcel but I'd rather keep the Gulp pipeline and not waste more time on something that needs a refactor.

Resources: Monaco and Trumbowyg editors mainly which I did not have time to work on trying to make them work. I started and I saw that there was a ton off merged files together. Why wasting time on migrating that stuff to the new Asset Manager if we don't refactor it using typescript? That's waste of time and there's nothing wrong with the gulp pipeline... it still works ... it is just old.

[Skrypt]

And also, I think we will still need to keep the Gulp pipeline up untill we figure out how to compile the Monaco, Liquid and Trumbowyg editors. They all have customizations for OC that I don't know how to recode because I don't know the required specs for them yet. This will take time.

[Skrypt]

Else, when we have single .js files I'm simply using "min" action which doesn't involve any compiler.
But when we "merge" files together then the compiler doesn't know which file should be used first unless I would send an array of files. The current Asset Manager doesn't support array of files because there is no need when using proper "imports" with ES6 scripts.

[Skrypt]

Ok, so distinction to make. Parcel, Vite, Webpack, Rollup are all compilers. Which means they will compile everything that is Ecmascript. They are also bundlers but they are mainly compilers/transpilers. So, this is why I'm refactoring to Typescript.

[Skrypt]

Ok Parcel maybe needs some tweaks to handle better CommonJS use cases.

Other options: Webpack seems to allow to only bundle files... not sure yet.

Either I'm pushing time on refactoring these assets or adding more tweaks to the Asset Manager.
I don't think we will need to support merging files together in the future if we refactor these scripts ...

---

The issue is that merging files together has never really been a compiler thing.
CommonJS modules normally uses "require"

const circle = require('./circle.js');`

Then the compiler will merge this file with the one where it is written. This is how "bundling" works.

Ecmascript will use "import"

import { modal } from 'boostrap'
import * from 'vue'

But here there is no path involved in this. The compiler will resolve the dependency from the package.json file(s) of your app. And probably also do some tree shaking on the dependency graph.

So, @Piedone here we are making a mistake of thinking that merging 2 files together like gulp does is bundling. It is not bundling at all. What we did with gulp is simply opening a writer and creating a new files out of multiple files without involving any compiler. Parcel, Vite and Webpack may fail to compile these files properly because of missing imports or require.

[Skrypt]

Using the "bundleEntrypoint" with Parcel will not work because it will produce 2 CommonJS modules out of these 2 files.
Modules are scoped meaning that you can't access their method without defining their exports.

https://dev.to/lico/how-to-import-and-export-in-commonjs-and-es-modules-43m1

[Skrypt]

So, this is why if we use these compilers that we need to refactor the code.

[Piedone]

What is this Frontend folder for?

[Skrypt]

I was tempted to name it "Bloom" javascript framework for Orchard Core. That's a folder for shared .ts components/modules. This is where we we will build it.

[Skrypt]

@Piedone Let me know for this one. I think that's the most important one in all of your comments.
If "Frontend" is not appropriate I would rename it to some more evident like "OrchardCore.Scripts" or "OrchardCore.Bloom" ... I don't know. "Frontend" seems a bit bad. Never found anything more meaningfull.

[Piedone]

I wasn't commenting on the name, but inquiring what the purpose of the folder and its contents are, I see, so this is intended as a component library. Then I'd rather have them in an .esproj and added to the solution.

[Skrypt]

.esproj ? I was asking about the name and if we would prefer something else because it is time now to decide before merging.

[Piedone]

The name is not an issue, for me at least. But that this folder is disjointed from the rest of the codebase.

[Piedone]

Why is this file necessary?

[Skrypt]

It is a renamed assets so that it compiles as admin-menu.js file in the wwwroot folder. The actions no longer allows to rename files from source to destination.

[Piedone]

What was it renamed from? I don't see it. It's strange because it seems to be only a copy of a jQuery plugin.

[Skrypt]

Right it was a jQuery plugin file which I needed to rename so that it works. I believe the licence is kept in that file so there should be no issue.

[Piedone]

I see: originally, there was an Assets/js/jquery.mjs.nestedSortable.js, and one also in OrchardCore.Menu. The diff only shows the former being renamed to the latter, hence why this shows as a new file, while it actually isn't.

But now I see that this has 4 copies in the codebase, one in OrchardCore.Resources, so please use that instead.

[Skrypt]

@Piedone based on all your comments, I think we will need to have a call instead of me answering on each questions here. Also, I explained on last standups meeting some of your questions already. Maybe you were not there.

[Piedone]

I wasn't there last week. Sure, we can have a call, but please focus on the harder ones, we shouldn't just go through each comment in a call. I've written you on Discord.

[Skrypt]

What do you mean by?

I want to move to File App as soon as possible. That was the original goal.

I meant Media Gallery refactor to Vue 3 and typescript. The goal here with this PR was to create a Build tool that would work for building the new Media Gallery in Orchard Core. And that I would be able to use with CI. I did not intent to start refactoring the entire javascript base code of Orchard Core for that matter.

So, this is why I did it partially and pushing work on those remaining module for later on.

[Piedone]

I see, I didn't know you were referring to the Media module, OK.