Add `find_additional_modules` option to support partial bundling with externals #3726

petebacondarwin · 2023-08-09T16:56:56Z

~~Disclaimer - WIP and untested!~~

~~This fairly chunky branch contains code that should allow us to search for rules matching modules when also bundling Workers.~~

Hey! 👋 This should be good to go now, see #3726 (comment) for details.

changeset-bot · 2023-08-09T16:57:00Z

🦋 Changeset detected

Latest commit: cce3d79

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
wrangler	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2023-08-09T16:59:46Z

A wrangler prerelease is available for testing. You can install this latest build in your project with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/6418496031/npm-package-wrangler-3726

You can reference the automatically updated head of this PR with:

npm install --save-dev https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/prs/6418496031/npm-package-wrangler-3726

Or you can use npx with this latest build directly:

npx https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/6418496031/npm-package-wrangler-3726 dev path/to/script.js

Additional artifacts:

npm install https://prerelease-registry.devprod.cloudflare.dev/workers-sdk/runs/6418496031/npm-package-cloudflare-pages-shared-3726

Note that these links will no longer work once the GitHub Actions artifact expires.

wrangler@3.10.1 includes the following runtime dependencies:

Package	Constraint	Resolved
`miniflare`	3.20231002.0	3.20231002.0
`workerd`	1.20231002.0	1.20231002.0
`workerd --version`	1.20231002.0	2023-10-02

|

Please ensure constraints are pinned, and miniflare/workerd minor versions match.

codecov · 2023-08-09T17:34:14Z

Codecov Report

Merging #3726 (cce3d79) into main (5fc7a88) will increase coverage by 0.23%.
The diff coverage is 86.61%.

@@            Coverage Diff             @@
##             main    #3726      +/-   ##
==========================================
+ Coverage   75.09%   75.32%   +0.23%     
==========================================
  Files         217      222       +5     
  Lines       12077    12179     +102     
  Branches     3128     3149      +21     
==========================================
+ Hits         9069     9174     +105     
+ Misses       3008     3005       -3

Files	Coverage Δ
packages/wrangler/src/api/pages/deploy.tsx	`86.15% <ø> (ø)`
packages/wrangler/src/config/validation.ts	`90.03% <ø> (ø)`
packages/wrangler/src/deploy/deploy.ts	`87.90% <100.00%> (+0.31%)`	⬆️
...wrangler/src/deployment-bundle/apply-middleware.ts	`100.00% <100.00%> (ø)`
...s/wrangler/src/deployment-bundle/build-failures.ts	`100.00% <100.00%> (ø)`
...ages/wrangler/src/deployment-bundle/bundle-type.ts	`100.00% <100.00%> (ø)`
...src/deployment-bundle/create-worker-upload-form.ts	`89.85% <100.00%> (+1.05%)`	⬆️
...s/wrangler/src/deployment-bundle/dedupe-modules.ts	`100.00% <100.00%> (ø)`
packages/wrangler/src/deployment-bundle/rules.ts	`100.00% <100.00%> (ø)`
packages/wrangler/src/dev.tsx	`83.53% <ø> (ø)`
... and 17 more

... and 3 files with indirect coverage changes

packages/wrangler/src/deployment-bundle/find-additional-modules.ts

mrbbot · 2023-08-18T17:33:27Z

Hey! 👋 I think this PR is ready to go now (minus some docs). You can now add something like...

find_additional_modules = true
rules = [
  { type = "ESModule", globs = ["**/*.js"]},
]

...to your wrangler.toml file, and have Wrangler bundle your entrypoint as usual, but upload all *.js files as is, marking them external. Appropriate files are watched in wrangler dev to trigger rebuilds when files are changed/created.

This gives us a slightly nicer "no bundle" mode, with the ability to use middleware whilst maintaining lazy loading/dynamic imports of --no-bundle mode.

Note, find_additional_modules is disabled when using service workers as it relies on the ability to pass multiple modules to the runtime, which isn't supported with service worker scripts.

Closes #3074

mrbbot · 2023-08-21T13:41:03Z

Blocked by cloudflare/miniflare#631

dario-piotrowicz · 2023-08-22T15:41:56Z

Confirming that this works and it's making the new lazy loading implementation of the workers nuxt adapter work 😃 🚀 🎉

See https://nuxt-split-workers-test.devdash.workers.dev/ (source)

dario-piotrowicz

Look good to me 😃

I gave the PR a somewhat thorough read and it looks all good to me, but I am very very unfamiliar with the codebase so this for sure still needs to be reviewed by someone much much more familiar with it than me

Note
This requires #3799 in order to work

packages/wrangler/src/deployment-bundle/build-failures.ts

…ole is.

… passed in

…dules flag is on

The no-bundle-import tests would actually pass if Wrangler actually bundled the code, since it would identify and inline all the dynamic imports that were being tested. This change adds a test that would not pass if we are not capturing the additional modules. It also moves the declaration of no-bundle to the wrangler.toml to ensure that normal use of `wrangler dev` will get this behavior.

This has a few of benefits: - we don't risk creating too many file reads in a single go (using up all the available file handles); previously file reads were all done in parallel - we don't read the files over and over for each rule - the code is a bit easier to follow as we don't have to create `Promise.all()` objects

When using `find_additional_modules` (or `no_bundle`) we find files that will be uploaded to be deployed alongside the Worker. Previously, if an `outDir` was specified, only the Worker code was output to this directory. Now all additional modules are also output there too.

irvinebroque · 2024-10-11T19:09:03Z

Is there context on why we added this but didn't put in dev docs?

petebacondarwin · 2024-10-14T18:57:40Z

Is there context on why we added this but didn't put in dev docs?

An oversight on my part, or at least I got pulled onto other stuff and it fell off my radar; on top of that it was mainly designed for advanced users who kind of know the code already. But I think it is worth documenting this now.

petebacondarwin · 2024-10-14T20:58:28Z

cloudflare/cloudflare-docs#17524

RamIdeas reviewed Aug 9, 2023

View reviewed changes

packages/wrangler/src/deployment-bundle/find-additional-modules.ts Outdated Show resolved Hide resolved

IgorMinar mentioned this pull request Aug 14, 2023

🚀 Feature Request: Add support for shims when no-bundling is applied #3074

Closed

mrbbot force-pushed the bundle-refactoring branch 2 times, most recently from bbd88b2 to d2f8f05 Compare August 18, 2023 17:32

mrbbot marked this pull request as ready for review August 18, 2023 17:34

mrbbot requested review from a team as code owners August 18, 2023 17:34

mrbbot changed the title ~~WIP: Bundle refactoring~~ Add find_additional_modules option to support partial bundling with externals Aug 18, 2023

mrbbot requested a review from RamIdeas August 18, 2023 17:35

mrbbot force-pushed the bundle-refactoring branch from d2f8f05 to d9bccf7 Compare August 21, 2023 12:35

mrbbot mentioned this pull request Aug 22, 2023

chore: bump miniflare to 3.20230821.0 #3799

Closed

2 tasks

mrbbot force-pushed the bundle-refactoring branch from d9bccf7 to 2f5c272 Compare August 22, 2023 14:25

mrbbot requested a review from a team as a code owner August 22, 2023 14:25

mrbbot force-pushed the bundle-refactoring branch from 2f5c272 to d27eaad Compare August 22, 2023 14:27

dario-piotrowicz approved these changes Aug 22, 2023

View reviewed changes

packages/wrangler/src/deployment-bundle/build-failures.ts Outdated Show resolved Hide resolved

packages/wrangler/src/deployment-bundle/build-failures.ts Outdated Show resolved Hide resolved

mrbbot force-pushed the bundle-refactoring branch from d27eaad to c419f5d Compare August 30, 2023 11:17

petebacondarwin force-pushed the bundle-refactoring branch 4 times, most recently from 73f2b7d to 00a8590 Compare August 31, 2023 18:54

mrbbot mentioned this pull request Sep 5, 2023

chore: bump miniflare to 3.20230904.0 #3895

Merged

2 tasks

petebacondarwin force-pushed the bundle-refactoring branch 2 times, most recently from 07919ca to d095eff Compare September 6, 2023 10:19

petebacondarwin and others added 22 commits October 5, 2023 12:36

move stuff

f73331c

remove unnecessary typing

3660aed

remove unnecessary active property from middleware loader config

662fabc

more refactorings

1738368

rename traverseModuleGraph as this is not what it does

bc2c1ff

refactor to simplify findAdditionalModules()

e686026

Rename the functions build helpers to make it more clear what their r…

a297503

…ole is.

invert module collector config - no longer need to return it as it is…

6cd5eee

… passed in

implement module finding in normal bundling if the find_additional_mo…

cb56381

…dules flag is on

allow __STATIC_CONTENT_MANIFEST to be imported from any module

6bfb60c

pnpm fixups

4c13313

test: ignore failure to remove tmp dir on Windows

407eaa6

test: do not show output in d1 time-travel tests

ee6e162

refactor: consolidate writing additional modules

bc11492

test: add CommonJS lazy import to additional-modules fixture

1deec5e

test: rename spec to test

83372be

Add debug logging when writing additional modules

a593ca0

Test additional module failure case and fix message typo

57eacce

Display build warnings before updating the bundle

cce3d79

petebacondarwin force-pushed the bundle-refactoring branch from 3dfad3f to cce3d79 Compare October 5, 2023 11:37

petebacondarwin merged commit 7d20bdb into main Oct 5, 2023
23 checks passed

petebacondarwin deleted the bundle-refactoring branch October 5, 2023 13:25

workers-devprod mentioned this pull request Oct 5, 2023

Version Packages #4061

Merged

Skye-31 mentioned this pull request Oct 24, 2023

🐛 BUG: Wrangler is injecting development code in production builds #4269

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `find_additional_modules` option to support partial bundling with externals #3726

Add `find_additional_modules` option to support partial bundling with externals #3726

petebacondarwin commented Aug 9, 2023 •

edited by mrbbot

Loading

changeset-bot bot commented Aug 9, 2023 •

edited

Loading

github-actions bot commented Aug 9, 2023 •

edited

Loading

codecov bot commented Aug 9, 2023 •

edited

Loading

mrbbot commented Aug 18, 2023 •

edited

Loading

mrbbot commented Aug 21, 2023

dario-piotrowicz commented Aug 22, 2023

dario-piotrowicz left a comment

irvinebroque commented Oct 11, 2024

petebacondarwin commented Oct 14, 2024

petebacondarwin commented Oct 14, 2024

Add find_additional_modules option to support partial bundling with externals #3726

Add find_additional_modules option to support partial bundling with externals #3726

Conversation

petebacondarwin commented Aug 9, 2023 • edited by mrbbot Loading

changeset-bot bot commented Aug 9, 2023 • edited Loading

🦋 Changeset detected

github-actions bot commented Aug 9, 2023 • edited Loading

codecov bot commented Aug 9, 2023 • edited Loading

Codecov Report

mrbbot commented Aug 18, 2023 • edited Loading

mrbbot commented Aug 21, 2023

dario-piotrowicz commented Aug 22, 2023

dario-piotrowicz left a comment

Choose a reason for hiding this comment

irvinebroque commented Oct 11, 2024

petebacondarwin commented Oct 14, 2024

petebacondarwin commented Oct 14, 2024

Add `find_additional_modules` option to support partial bundling with externals #3726

Add `find_additional_modules` option to support partial bundling with externals #3726

petebacondarwin commented Aug 9, 2023 •

edited by mrbbot

Loading

changeset-bot bot commented Aug 9, 2023 •

edited

Loading

github-actions bot commented Aug 9, 2023 •

edited

Loading

codecov bot commented Aug 9, 2023 •

edited

Loading

mrbbot commented Aug 18, 2023 •

edited

Loading