What if Gutenberg was released as a single npm package?

[adamziel]

What is this issue about?

Gutenberg could be released as a single @wordpress/gutenberg package instead of 40+ packages. It’s an idea @dmsnell had and shared with me recently.

What are the upsides?

Here’s what it would do:

• WordPress consumers would have significantly easier jobs managing @wordpress dependencies. This includes Gutenberg releases and WordPress core merges. It would be impossible to run into problems with mismatched packages versions.
• Private experimental APIs would no longer be an issue – they would never get publicly exported in the first place
• The project setup would get drastically simpler. We’d get rid of:
  • Lerna, multi-package management flows, reverting accidental version bumps
  • 39 tsconfig.json files and problematic cross-package dependencies
  • 91 package.json files and
  • Multiple build steps with per-package build, build-module, and build-style setup
  • Many lines of build scripts and GitHub automations across Gutenberg and WordPress repositories
  • ...and that's just off the top of my head – I'm confident there's more.

Anecdotally, maintaining wordpress-playground got significantly easier after I migrated the monorepo from four internal packages to a single repository-wide package.

What are the downsides?

First, let me flip the question – what value is added by having 40+ packages? Or, as @dmsnell phrased it:

For all the slowdown, complexity, obstruction, confusion, incompatible-looking version numbers, hard-to-figure-out dependencies, undocumented relationships, and institutional knowledge required to keep the system moving, what are we getting?

I've heard the following concerns so far:

Switching to a single package will break the existing plugins relying on separate packages

The existing packages wouldn’t disappear from npm. They would get deprecated. Every package would ship one final release where every relevant member would get re-exported from @wordpress/gutenberg.

Separate packages enable surgically using specific parts of Gutenberg

A single package would make it even easier. Figuring out which of the 40+ packages are needed is a hit-and-miss trial-and-error process.

How would e.g. the site editor not have JavaScript to do with the widgets editor? A lot of first and third pages rely on smaller bits like “components”, so we need a way to bundle wp-components or wp-data based on that single package.

The bundler would take care of that automatically with tree shaking. The package consumer would import the bits they are interested in, and the bundler would remove all the irrelevant code. If two entrypoints require the same package, the bundler would move it to a separate chunk to avoid downloading the same code twice.

In WordPress, we talk about scripts and global variables and not about packages. These things shouldn’t change

A single package could still produce the same set of JavaScript files as the existing 40+ packages.

Some of the modules have side effects. For instance, if you import the block-editor package, it will register the store automatically.

This can continue to work in the same way. If necessary, a webpack plugin would take care of that. There could also be a new and explicit registerStore() utility.

Packages allow a logical code split and force the developers to think about what should depend on what

The logical split and the directory structure would remain exactly the same as it is now. Only the package.json files would be gone. As a side note, Gutenberg's separation of concerns isn't nearly perfect despite the use of packages.

What are the alternatives?

• Reducing the number of packages. For example, instead of 40+ there would be 10.
• Add one more "proxy" package (like @wordpress/bootstrap) that would re-export everything from the existing 40+ packages. That would be a win for package consumers.

Both would improve some things, but would not address the complexity related to Lerna, version management, and __experimental exports discussed earlier.

What do you think?

CC @youknowriad @gziolo @mcsf @noisysocks @priethor @aristath @peterwilsoncc @carolinan @tellthemachines @talldan @georgeh @artemiomorales @kevin940726 @ramonjd @andrewserong @ellatrix

[peterwilsoncc]

Adding @azaozz @desrosj @hellofromtonya and @SergeyBiryukov to the CC list.

I'll ponder this a little more before committing any notes to the ticket.

[youknowriad]

Hi @adamziel Thanks for creating this issue and starting this discussion, it's always good to reassess past decision and see whether they're still good decision for what has to come.

Before doing so though, it's important that we decontextualize how the architecture of Gutenberg came to be, the history and the reasoning behind it as the issue right now seem to go directly into a "solution" by omitting a lot of context.

First, it's important to note that Gutenberg has been built from the start as packages, in fact, originally it was organized as a single monolith, using folders, very close to the suggestion here. You can check old tags like here.

Quickly though, some use-cases started to emerge:

• We wanted to be able to share scripts with other Core features and use some core scripts (api-request which became api-fetch for instance, a11y script which became a11y package...) => this led the creation of the @wordpress/packages repository, a repository of reusable packages that can be used both in Gutenberg and WordPress. Ultimately, the maintenance of two separate repositories both updating related scripts (Gutenberg and packages repository) become too difficult which led to the decision of merging all packages/scripts in a single repository (Gutenberg repository).

• We wanted to be able to use components and UI from gutenberg in other third-party pages, settings pages... without necessarily having to load the whole editor scripts

• We wanted to offer a way to build several kind of block editors: comments, post editor, third-party editors... (later, widgets editor, navigator editor, site editor).

This led us to the necessity of modularizing our code into semantically valid units which increased by an order of magnitude the separation of concerns in the project. You mention that it's not perfect, and I agree but I think it's decent and in fact the big issues we might have right now pre-dates the introduction of packages (block-library, editor package).

I've seen a number of times that the packages force people to think where to put components, hooks, where's the right place for things. At start, people get frustrated about not being able to just put the code wherever they want and ship but that's a good thing to force people to think about the modularization and separation of concerns. In fact, with the initial folder based architecture, we were supposed to have folders with dedicated roles but people fell too easily in the trap of just mixing stuff together. (For instance using post related data in block editor specific components...)

• We wanted to be able to expose extensibility APIs to third-party developers within WordPress, this led to the wp globals, which was also used as a communication layer between core packages, similarly to how third-party developers use it.

Next, came the integration with Core, and how to move the code into Core. Packages were not the only option explored, in fact, we also tried git submodules but npm packages ended up being the best option there.

Now, later third-party npm users benefited from all of these packages to build integrations in Tumblr, Drupal, Laravel and way more...

There's also the merge of the Gutenberg packages with the mobile packages which I know has helped the folks working on the mobile applications a lot. I'm not entirely familiar with this area, so we'd need their input here as well.

Each one of these steps were discussed in #core-js meetings between several folks, at the time these meetings were very lively and productive.

Requirements

This is to say, that any change we make here, we should be mindful of all these requirements that accumulated over time:

1- Enforced awareness and listing rules to guarantee separation of concerns (and modularity) as much as we can.
2- Shared scripts between the plugin and WordPress Core.
3- The possibility to load a small chunk (for instance just the components, or just the a11y utils) in a particular wp-admin page, be it an editor page or not.
4- Keeping the existing wp globals largely untouched: Any major change would impact backward compatibility way too much.
5- Being able to use our modules using npm for third-party consumers (in fact whether it's one module or multiple here doesn't matter)

Proposed solution

The proposed solution is basically to just merge all the packages inside one using folders, so:

• @wordpress/components becomes @wordpress/gutenberg/components in fact, it can even be just wordpress/components (components folder inside a wordpress package, there's no need for a "gutenberg package", since these are not editor specific and gutenberg is just a code name)
• @wordpress/x becomes wordpress/x

If I'm understand properly, we just publish a single package to npm instead of publishing multiple ones.

Promises

So now, let's analyze the proposed solution and the promises above:

Separate packages enable surgically using specific parts of Gutenberg
A single package would make it even easier. Figuring out which of the 40+ packages are needed is a hit-and-miss trial-and-error process.

There's an issue right now, which I'd call a "bootstrapping" issue, in the sense that if you want to build a block editor, you need a provider, to load blocks, to load formats, and have some providers... and these can come from several packages. Most of these are optional because they are exchangeable: you can decide to not load blocks and bring your own, bring your own formats, load shortcuts or not... You guessed it, it's not easy to bootstrap an editor. Moving to a single package though won't change anything to the fact that you still need to figure out what you need to load and what you don't need to load, the only thing that changes is that you'll do npm install only once which will bring all the packages in one go (including the ones you don't need: think all the WP specific code, all the mobile code, all the dev tools you might not need...)

The bundler would take care of that automatically with tree shaking. The package consumer would import the bits they are interested in, and the bundler would remove all the irrelevant code. If two entrypoints require the same package, the bundler would move it to a separate chunk to avoid downloading the same code twice.
A single package could still produce the same set of JavaScript files as the existing 40+ packages.

In the current approach, we do this by marking all @wordpress/* packages as externals, so every time the bundler finds @wordpress/x it's transformed automatically to wp.x global variable and a dependency is added in php and each one of these entry points (packages now, but I guess top level folders in the proposed solution) creates a new wp.x global (a webpack config).

So basically, with the new solution, we'd have to find a way to recreate this behavior using webpack or any other bundler. I think it's probably the exact same config we have now, expect the name of the externals to change (assuming folders work as externals too which I'm not sure about and in which case we'd need to build a custom bundler plugin)

Private experimental APIs would no longer be an issue – they would never get publicly exported in the first place

Unfortunately, I don't think this solves that at all. Here's an example. I want to introduce a new Hello component inside the components folder (since it's a reusable UI component for SoC) and I want to use the Button component but I want Hello to be internal and only used inside block-editor folder for now.

The block editor code would do something like import Hello from '/components/hello':
And we did see in the previous point that if we are to generate the same chunks (and globals) we have now, every time we find /components/hello we'll turn it into wp.components.Hello.

The bundler doesn't have any knowledge that allow him to differentiate "internal" components from "public" ones, both are imported in the same way.

Pros & cons

• Yes, third-party devs would only do a single npm install but it's also a con, it would bring the whole Gutenberg repository and I think you're underestimating all the non block-editor code in the repo (react native packages, all edit-* packages, all the dev tool packages...) So we'd be downloading way more code than needed.
• Yes, we'll be removing some code (Lerna, package.jsons...) I think that's a good thing
• We'll be trading some of these scripts with more complexity in the bundlers (both core and gutenberg plugin bundlers)

---

So far, for me, the only advantage that is clear is the removal of complex lerna publishing and scripts... (assuming we keep a single package of course). I also feel this is largely underestimating the effort it takes to accomplish what it is proposed. At this point at least, I feel it's a giant project for a small outcome unfortunately.

The "bootstrapping" issue is a real pain point though and a less impactful and way more straightforward solution is the "proxy" package that is proposed in the alternatives above. In fact, it's an approach that has already been taken by some third-party developers including the Drupal folks that built their own "proxy" package here or the isolated block editor by a8c. An official "proxy/bootstrap" package (so a single package to import and bootstrap with settings or something like that) would be a good addition anyway.

[adamziel]

Private experimental APIs would no longer be an issue – they would never get publicly exported in the first place
Unfortunately, I don't think this solves that at all. (...) Every time we find /components/hello we'll turn it into wp.components.Hello.

Oh my, you're right. I'm now realizing that, regardless of the repo structure, the .js generated for the browser are effectively packages themselves. Merging the repo into a single package does not solve the problem of sharing private code between the output packages.

Hypothetically, the bundler could move it into another file like chunk-W6GS52X6.js and then, we'd turn /component/hello into require("chunk-W6GS52X6.js").componentHello. Unfortunately, it's not as watertight as a real private property. It's also technically the same as @gziolo's secret key proposal that was ultimately rejected.

Well, at least the single-package research avenue for private experimental APIs can now be closed. cc @peterwilsoncc

[adamziel]

The "bootstrapping" issue is a real pain point though. (...) An official "proxy/bootstrap" package would be a good addition anyway.

Agreed!

[adamziel]

We'll be trading some of these scripts with more complexity in the bundlers (both core and gutenberg plugin bundlers)

I can't prove it without a PR, but I think we'd trade a large amount of complexity for a small amount of new complexity which is a net positive.

[adamziel]

So far, for me, the only advantage that is clear is the removal of complex lerna publishing and scripts

@youknowriad I still see more upsides than that, as I mentioned above, but I agree they're smaller than what I originally suggested. I'm now leaning towards the alternatives, so a "proxy bootstrap" package and a reduced number of npm packages. Thank you for taking the time to write up your excellent reply!

[dmsnell]

packages are a much more effective forcing function

this probably takes us back to the general practice of code review and design review. there are likely ways we can enforce more constrained imports if that's really what we want to do, such as write a linter that specifies that if you import a module from another package directory, then you can only import from the top of that directory instead of importing deep inside of it.

packages do enforce separation, and that's a valid point. personally I have not felt the value this brings to outweigh the costs they impose on us for reasons you iterated at the top.

a single package wouldn't help you figure out all the functions, components, and variable names to import and how to wire them together.

this is something I don't agree with because a single package gives us a single place to start exploring what's available. I can always look at the list of top-level exports or the wp global variable in a debug build and explore what I can do. I don't have to know what I'm looking for or know where to look for it.

it doesn't solve the problem, but bringing everything together in one place definitely aids in building a picture of the entire system.

think all the WP specific code, all the mobile code, all the dev tools you might not need

I also think this fear might be overblown, particularly because we still download most of that if we want to build a new bare-minimum editor. the difference is in how much hassle we have to deal with before we figure out the minimum required set of packages, all with different version numbers, all which have to correspond to each other lest they break at runtime.

it's not like we only get one package when we npm install a Gutenberg package. those inter-package dependencies are sprawling.

npm install @wordpress/components pulls in 20 of the 92 total packages.
npm install @wordpress/data pulls in 12 of them.
npm install @wordpress/block-library pulls in 39.

my point here is that any project with a functioning editor is already grabbing hundreds of MBs of dependencies and near half of the project's packages - and all the big important ones.

the difference between installing only @wordpress/block-library and installing every possible package is the difference between 187 MB of dependencies and 497 MB. That's a lot of difference, but both are overwhelmingly bloated and in the grand scheme of things I don't see much of a practical difference here.

Every time we find /components/hello we'll turn it into wp.components.Hello.

This could be related to async-loading code/packages. If we had a solid mechanism for doing that I don't think these would be as big of a deal? Because we could start with the minimum core and load in extensions/packages/modules as we need them. Not sure personally how this works with React Native.

the removal of complex lerna publishing and scripts

This is no small advantage! These scripts have practically eliminated my ability to interactively develop Gutenberg in the context of a third-party integration. I have spent multiple hours trying to figure out how to do the equivalent of npm link and nothing works. This has meant that in order to work with Gutenberg outside of WordPress I have to guess on what I need, get those changes merged into the project, wait for a release, and wait until the integration can upgrade to the latest release (which usually involves a lot of breaking changes requiring significant rework), and only then see if I guessed right.

It may be that I'm still just ignorant of how to do this but I haven't found any answers online or in person other than "don't do that."

beyond that it's a lot of maintenance work and confusion dealing with the variegated package versions. there's no good way to correlate the individual package versions with each other or with a release, and if you get one wrong your project breaks.

when I created a new blank project and ran npm install for every package in the project I ended up with 86 packages listed in package.json with 41 distinct version numbers, and they range from 0.5.0 to 25.0.0. I don't understand how anyone consuming these packages is supposed to make any sense of this; when producing them I don't think it matters as much because we don't care about the versions we send out into the world.

[mtias]

Could this be moved to a discussion thread instead? I personally don't think this is worth the effort and see very few upsides (if any?), but pros and cons can be discussed more granularly there in comment threads.

[peterwilsoncc]

WordPress consumers would have significantly easier jobs managing @wordpress dependencies. This includes Gutenberg releases and WordPress core merges. It would be impossible to run into problems with mismatched packages versions.

I see this as the biggest single advantage of this proposal -- it's certainly something I've run in to in a previous role -- but I agree with @youknowriad that the separation of packages enforces an increased awareness of sound architectural decisions.

As I mainly work in wordpress-develop, I have more experience as a consumer and super packages such as @wordpress/wordpress-core and @wordpress/gutenberg-plugin would provide an advantage to avoid the conflicts but is counter to the goal of fewer packages :)

As long as the wp.? namespaces are maintained in their current form, I'm happy to defer to those that work in the gutenberg repo here.

[dmsnell]

but is counter to the goal of fewer packages :)

I don't see it. in fact, given that we are stuck here in this legacy of our packages, I had initially thought the idea of a super package was a potential way to resolve the worst problems without touching the framework side, or at least in giving the framework time to gradually transition.

having 92 instead of 93 packages doesn't seem as big of a problem to as having to import a growing list of unknown packages vs. importing one. if that super package can operate outside of lerna and we can use the platform tools, such as npm link, or if we could even lean on symbolic links to that super-package and be able to do dev work on Gutenberg outside of it, then that would be great, even it appears to make the problem marginally worse for a time.

[noahtallen]

The big benefit is that it'd be simple to update. The main reason it's complex right now is that @wordpress/ packages really must come from the "same release" to be compatible with each other. But neither the packages (nor dependency management systems) enforce that, so it's easy to get out-of-sync if you aren't careful.

You must update every @wordpress/ package at the same time, but you also have to be very careful that you don't accidentally update one @wordpress/ package in between updates.

For example, it's not uncommon for someone to add a new WP dependency to our monorepo, or to update a dep individually if they're looking for a new feature. But this pretty much always creates unexpected issues in the repo, since that new version is out of sync with the rest of the repo. And this new version would now pull along updates from all the transitive dependencies -- often including undocumented breaking changes -- making things very complex.

Recently, I updated a dependency that uses WP packages, and the package manager started resolving the new versions, whereas the rest repo is somewhat out of date. (Due to updates being challenging and time-consuming.) The new versions pulled in tons of new changes, including the React 18 update, which then pulled in a lot of other transitive dependencies, like newer React 18 types. This of course created lots of issues, and I had to go in and manually enforce dependency resolution to stay at what the rest of the repository was using. All of this, just because I was bumping the version of one package which happened to use WordPress packages internally.

I definitely think this problem (compatibility among package versions) needs to be better solved. One Package To Rule Them All definitely does that, but I suppose there are other ways. For example, the bi-weekly package release could get a new major version every time, and that version could be the same across all the packages. Maybe difficult to make the change at first, but that would prevent package managers from doing unexpected updates to the latest semver-compatible versions.

TLDR: package versions from different releases aren't typically fully compatible with each other, but this is not enforced at any level.

[gziolo]

To some degree, it's possible to update all WordPress packages in one go to the latest version or to the exact snapshot used by a major WordPress version. It's possible with the @wordpress/scripts package and the packages-update. We use that tool to backport changes applied in the Gutenberg repository to WordPress core.

--dist-tag – allows specifying a custom dist-tag when updating npm packages. Defaults to latest. This is especially useful when using @wordpress/dependency-extraction-webpack-plugin. It lets installing the npm dependencies at versions used by the given WordPress major version for local testing, etc. Example: wp-scripts packages-update --dist-tag=wp-6.0.

Major WordPress releases from 5.7 are tagged this way. In theory, we could apply the same strategy for every Gutenberg plugin release if that would be useful. It probably would be simpler to enforce a major version bump on every Gutenberg plugin release as we do minors today:

gutenberg/.github/workflows/build-plugin-zip.yml

Line 339 in 906e5a7

./bin/plugin/cli.js npm-latest --semver minor --ci --repository-path ../publish

That would require only updating minor with major in the line above.

Another option discussed to was to use a single version for all packages, and that is also a change that requires a single line change here:

gutenberg/lerna.json

Line 14 in 906e5a7

"version": "independent"

In addition, we would probably need to change all version numbers for existing packages to something like 100.0.0-prerelease, where the number would have to be the maximum number found in all existing packages.

[jsnajdr]

WordPress consumers would have significantly easier jobs managing @wordpress dependencies.

I agree with @noahtallen that this goal could be very well achieved with another, much less drastic change: unify the versioning of all @wordpress/* packages and always release them with one single version number. This is, for example, what the Jest project does. There are 50+ subpackages like @jest/core or @jest/fake-timers, but they are released with the same version, say 29.3.1, and if package @jest/x depends on @jest/y, the versions of x and y are always the same.

This makes upgrading the monorepo much easier, as you can quickly detect if your packages are out of sync, just by scanning the lockfile or the node_modules folder.

On the other hand, if my project depends on wordpress/block-editor@11.1.0 and wordpress/data@8.1.0 -- is that OK, are the packages compatible? Nobody really knows, these version number are rather random and unrelated, they don't have much meaning for humans. The dependency tree between the 90 packages is intractable.

Additionally, we are very bad at maintaining changelogs. If you want to know what changes are there between wordpress/data 7.0.0 and 7.6.0, you'll have a very hard time figuring that out. That makes all the version numbers even less meaningful.

So, the basic suggestion would be to adopt the Jest convention and release Gutenberg packages in one version line.

There's also a more radical version of the suggestion: screw semver and use the same version for the Gutenberg plugin and the NPM packages. When releasing Gutenberg 14.9.2, release also the NPM packages with version 14.9.2. I'm not sure if that's even technically possible, but as a consumer, I would love to see it.

[gziolo]

My comment #46693 (reply in thread) is relevant also here.

We could also revisit #26954 where you proposed that we change many WordPress packages to peer dependencies. We still need to switch to Node 18 and npm 9 in WordPress projects to make it possible, but at least it's going to be the default behavior in the JS community to have peer dependencies installed by default from May, when Node 14 will stop getting official support.

We bump a minor version with every npm publishing tied to the Gutenberg plugin release, so we could use that to also develop a custom logic to automatically maintain changes to the listed peer dependencies that aren't covered by Lerna.

[mrmurphy]

Hi, folks! Thanks for doing the hard work of figuring out what to do here. I don't have much opinion about which direction should be taken, but I wanted to let you know the current situation has been quite painful for us on the Day One Web team as consumers of @WordPress packages. Specifically, package version mismatches have wasted hours of developer debugging time. We'd vote for any way to better know what versions to install when upgrading packages so we don't get multiple copies of dependencies that cause subtle logical bugs (like using two different copies of @wordpress/hooks, for example. This was the latest case I debugged. Our project's package.json listed a different version from the one that @wordpress/block-editor specified)

[johnhooks]

Continuing from #49140

I am also experiencing similar issues with the current versioning scheme. Of the options I like @jsnajdr's suggestion of unified versioning the best. It's simple, at a glance block authors can see what version of Gutenberg they are targeting. Thank you @gziolo for pointing out package-update, I will definitely use this. Though while that is helpful, it does not resolve the issue for someone working on multiple projects that are all in different states. Without creating additional documentation in the projects, it's difficult to identify what version of Gutenberg a project is trying to develop for, because its version number is no where in the dependencies.

I can see how this isn't an important issue outside of the monorepo, because it doesn't matter what the versions are. It's almost arbitrary because of the liberal use of "file:./*" paths in the packages.json files. The issue of managing these dependencies in Gutenberg is a non issue, though it is something keenly felt by third-party block developers.

[johnhooks]

@gziolo package-update was actually not helpful in my situation. I'm using pnpm and that script calls npm install which was installing all the packages outside of the system pnpm uses. I created this utility last night, if anyone is interested in a half measure solution, check it out let me know how it works for you. https://www.npmjs.com/package/wp-deps

Edit: I actually think wp-deps turned out better than a "half measure solution"!

[gziolo]

Yes, we should fix that in the script to detect what calls it and construct a corresponding command.

[noahtallen]

Another (potentially easy to implement) idea is to use ~ instead of ^ when publishing the packages. While a patch release is hopefully unlikely to break anything, minor releases can!

For example, say a new function is added to @wordpress/compose and then @wordpress/data imports that new function. Both are published as new minor versions. However, data doesn't require compose to update when this happens! So if I only try to update the data dependency without compose, my application will crash when that new function is unavailable. This is because the carat doesn't require packages to update unless the major version changes, and this was only a minor version bump.

[anomiex]

This is because the carat doesn't require packages to update unless the major version changes, and this was only a minor version bump.

This is not true. If data depends on something that was added in compose version 12.5.0, then its dep should be ^12.5.0 and any lower version (even 12.4.999) won't satisfy that.

If it's depending on ^12.0.0 or the like and expecting to get 12.5.0 or later, that dependency is wrong in the first place. That's not a problem with the use of caret, it's with the dependency being incorrect in allowing too low of a version to pass.

Ignoring 0.x versions, a dep on ^x.y.z is just saying >= x.y.z < (x+1).0.0. A dep on ~x.y.z is similar, meaning >= x.y.z < x.(y+1).0. In either case if the x.y.z is in fact too low then you'll have a problem.