[DO NOT MERGE] chore(precompile): reduce the size of precompiled dependencies #58038

SukkaW · 2023-11-04T21:42:50Z

The PR addresses https://twitter.com/hazae41/status/1720557445678858248.

Currently, The installation size of the next package is at 73.6 MiB. Among 53 MiB of them are precompiled dependencies.

The PR attempts to reduce that size by removing unused ones:

~~Remove find-cache-dir~~ chore(precompile): remove obsolete precompiled assets #60316
- The usage of find-cache-dir was removed 4 years ago (Switch to using .next/cache for flying shuttle and terser #7013) but the dist and the build script were never removed.
~~Remove @segment/ajv-human-errors~~ chore(precompile): remove obsolete precompiled assets #60316
- The usage of @segment/ajv-human-errors was removed in refactor: rewrite config schema in zod #56383 by me. Though the build script was also removed in that PR, the dist never got removed.
~~Remove Re-add precompile watchpack~~ chore(precompile): re-add watchpack to the precompile #60309
- Next.js has stopped bundling the watchpack since Improve compile time on large application #50792. Currently, watchpack is a direct dependency of Next.js, ~~but the type never got updated and the pre-compiled dist never got removed.~~ the PR adds the pre-compile back.
~~Enable minification of @next/font~~ chore(font): enable minification #60319
- Currently, the un-minified @next/font is bundled with Next.js.
~~Remove chalk inside @next/react-dev-overlay~~ refactor(dev-overlay): remove chalk #60317
- @next/react-dev-overlay is bundled with Next.js. The usage of chalk inside @next/react-dev-overlay was not removed in chore: remove chalk in favor of picocolors #55992, thus the chalk is still being shipped.
Remove schema-utils2 and webpack-sources1
- schema-utils@2 and webpack-sources@1 were only used in webpack 4. After webpack 4 was dropped, they were no longer necessary.
Remove loader-utils2
- saas-loader no longer uses loader-utils while resolve-url-loader only uses loader-utils.isUrlRequest. So it is safe to remove loader-utils2 and replace its usage with loader-utils3.

ijjk · 2023-11-04T21:47:24Z

Failing test suites

Commit: 7ddfe33

pnpm test test/integration/scss/test/scss-loader-handling.test.js

SCSS Support loader handling > production mode > Data Urls > should compile successfully
SCSS Support loader handling > production mode > Data Urls > should've emitted expected files

Expand output

● SCSS Support loader handling › production mode › Data Urls › should compile successfully

expect(received).toBe(expected) // Object.is equality

Expected: 0
Received: 1

  210 |         })
  211 |         // eslint-disable-next-line
> 212 |         expect(code).toBe(0)
      |                      ^
  213 |         // eslint-disable-next-line
  214 |         expect(stdout).toMatch(/Compiled successfully/)
  215 |       })

  at Object.toBe (integration/scss/test/scss-loader-handling.test.js:212:22)

● SCSS Support loader handling › production mode › Data Urls › should've emitted expected files

ENOENT: no such file or directory, scandir '/root/actions-runner/_work/next.js/next.js/test/integration/scss-fixtures/data-url/.next/static/css'

Read more about building and testing Next.js in contributing.md.

ijjk · 2023-11-04T21:49:06Z

Stats from current PR

Default Build

General

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
buildDuration	12.9s	12.8s	N/A
buildDurationCached	7.2s	6.2s	N/A
nodeModulesSize	200 MB	198 MB	N/A
nextStartRea..uration (ms)	428ms	421ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
193.HASH.js gzip	181 B	182 B	N/A
3f784ff6-HASH.js gzip	53.3 kB	53.3 kB	✓
433-HASH.js gzip	28.6 kB	28.6 kB	N/A
framework-HASH.js gzip	45.2 kB	45.2 kB	✓
main-app-HASH.js gzip	240 B	242 B	N/A
main-HASH.js gzip	31.7 kB	31.8 kB	N/A
webpack-HASH.js gzip	1.7 kB	1.7 kB	N/A
Overall change	98.5 kB	98.5 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
_app-HASH.js gzip	194 B	195 B	N/A
_error-HASH.js gzip	183 B	181 B	N/A
amp-HASH.js gzip	504 B	502 B	N/A
css-HASH.js gzip	321 B	321 B	✓
dynamic-HASH.js gzip	2.5 kB	2.5 kB	N/A
edge-ssr-HASH.js gzip	255 B	253 B	N/A
head-HASH.js gzip	350 B	349 B	N/A
hooks-HASH.js gzip	369 B	369 B	✓
image-HASH.js gzip	4.28 kB	4.28 kB	N/A
index-HASH.js gzip	255 B	256 B	N/A
link-HASH.js gzip	2.61 kB	2.61 kB	✓
routerDirect..HASH.js gzip	312 B	311 B	N/A
script-HASH.js gzip	385 B	383 B	N/A
withRouter-HASH.js gzip	307 B	308 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	3.4 kB	3.4 kB	✓

Client Build Manifests

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
_buildManifest.js gzip	483 B	484 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
index.html gzip	527 B	529 B	N/A
link.html gzip	540 B	541 B	N/A
withRouter.html gzip	523 B	524 B	N/A
Overall change	0 B	0 B	✓

Edge SSR bundle Size

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
edge-ssr.js gzip	93.8 kB	93.8 kB	N/A
page.js gzip	147 kB	147 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
middleware-b..fest.js gzip	625 B	622 B	N/A
middleware-r..fest.js gzip	151 B	151 B	✓
middleware.js gzip	37.4 kB	37.4 kB	N/A
edge-runtime..pack.js gzip	1.92 kB	1.92 kB	✓
Overall change	2.07 kB	2.07 kB	✓

Next Runtimes

	vercel/next.js canary	SukkaW/next.js remove-unused-precompile	Change
app-page-exp...dev.js gzip	169 kB	169 kB	✓
app-page-exp..prod.js gzip	94.2 kB	94.2 kB	✓
app-page-tur..prod.js gzip	94.9 kB	94.9 kB	✓
app-page-tur..prod.js gzip	89.5 kB	89.5 kB	✓
app-page.run...dev.js gzip	139 kB	139 kB	✓
app-page.run..prod.js gzip	88.8 kB	88.8 kB	✓
app-route-ex...dev.js gzip	24.1 kB	24.1 kB	✓
app-route-ex..prod.js gzip	16.7 kB	16.7 kB	✓
app-route-tu..prod.js gzip	16.7 kB	16.7 kB	✓
app-route-tu..prod.js gzip	16.3 kB	16.3 kB	✓
app-route.ru...dev.js gzip	23.5 kB	23.5 kB	✓
app-route.ru..prod.js gzip	16.3 kB	16.3 kB	✓
pages-api-tu..prod.js gzip	9.38 kB	9.38 kB	✓
pages-api.ru...dev.js gzip	9.65 kB	9.65 kB	✓
pages-api.ru..prod.js gzip	9.37 kB	9.37 kB	✓
pages-turbo...prod.js gzip	21.9 kB	21.9 kB	✓
pages.runtim...dev.js gzip	22.6 kB	22.6 kB	✓
pages.runtim..prod.js gzip	21.9 kB	21.9 kB	✓
server.runti..prod.js gzip	49.5 kB	49.5 kB	✓
Overall change	932 kB	932 kB	✓

Commit: 7ddfe33

feedthejim · 2023-11-05T09:57:27Z

What's the after size?

I think the watch pack one might be an oversight and we should re use the pre compiled one

SukkaW · 2023-11-05T16:20:09Z

What's the after size?

The final installation size reduction can be seen from the PR stats, nodeModulesSize. Currently, I've only reduced it from 175 MiB to 172 MiB.

I think the watch pack one might be an oversight and we should re-use the pre-compiled one

I can bring it up in the PR. This should help reduce the installation size further.

Edit: The pre-compiled watchpack is now back.

SukkaW · 2024-01-05T17:56:49Z

It seems that somehow sass-loader data url tests failed after the changes. I will look into this when I get time.

styfle · 2024-01-05T19:29:29Z

@SukkaW If its easier - you could split this up into a few PRs:

minify font
remove chalk
remove unused precompiled deps
compile watchpack

@styfle

Per @styfle suggestion, #58038 has been split into multiple PRs for easier review. - Enable minification of `@next/font` - Currently, the `@next/font` is built separately (without minification) and then copied to the `dist` folder of the Next.js.

@styfle

Per @styfle suggestion, #58038 has been split into multiple PRs for easier review. - Remove `find-cache-dir` - The usage of `find-cache-dir` was removed **4 years ago** (#7013) but the dist and the build script were never removed. - Remove `@segment/ajv-human-errors` - The usage of `@segment/ajv-human-errors` was removed in #56383 by me. Though the build script was also removed in that PR, the dist never got removed.

@styfle

Per @styfle suggestion, #58038 has been split into multiple PRs for easier review. - Re-add precompile `watchpack` - Next.js has stopped bundling the `watchpack` since #50792. Currently, `watchpack` is a direct dependency of Next.js. The PR adds the pre-compile back. Co-authored-by: Steven <steven@ceriously.com>

@styfle

Per @styfle suggestion, #58038 has been split into multiple PRs for easier review. - Remove `chalk` inside `@next/react-dev-overlay` - `@next/react-dev-overlay` is bundled with Next.js. The usage of `chalk` inside `@next/react-dev-overlay` was not removed in #55992, thus the `chalk` is still being shipped. --------- Co-authored-by: JJ Kasper <jj@jjsweb.site>

SukkaW · 2024-01-13T15:45:28Z

All the subsequent PRs have been merged. The PR can be closed.

The PR continues from #58038 aiming to reduce the installation size further, and **improve performance in doing so**. Most of `micromatch`'s APIs are essentially wrappers/alias of `picomatch`, supplemented with additional features facilitated by a transitive dependency `braces`. However, Next.js doesn't use those "extra features". Switching from `micromatch` to `picomatch` can avoid introducing the transitive dependency `braces`. The PR also improves some globs' performance by hoisting. - `micromatch.makeRe` is just an alias of `picomatch.makeRe` - https://github.com/micromatch/micromatch/blob/90dc0cd3e186b8ab4dd05bfc41e2b98f5067a49b/index.js#L387C30-L387C30 - `micromatch.isMatch(str, patterns, options)` equals `picomatch(patterns, options)(str)` - https://github.com/micromatch/micromatch/blob/90dc0cd3e186b8ab4dd05bfc41e2b98f5067a49b/index.js#L123 - `micromatch.matcher` is just an alias of `picomatch` - https://github.com/micromatch/micromatch/blob/90dc0cd3e186b8ab4dd05bfc41e2b98f5067a49b/index.js#L104C1-L104C40 --------- Co-authored-by: Sam Ko <sam@vercel.com>

ijjk added the type: next label Nov 4, 2023

chore(precompile): remove find-cache-dir

fd3da2f

SukkaW added 2 commits November 5, 2023 05:53

chore(precompile): remove files

3abde6d

chore: improve watchpack types

04eb4d5

SukkaW changed the title ~~chore(precompile): remove find-cache-dir~~ chore(precompile): remove unused precompiled dependencies Nov 4, 2023

SukkaW force-pushed the remove-unused-precompile branch from 476df3d to 04eb4d5 Compare November 4, 2023 21:59

chore(font): enable ncc minify

a67d36b

SukkaW changed the title ~~chore(precompile): remove unused precompiled dependencies~~ chore(precompile): reduce the size of precompiled dependencies Nov 4, 2023

SukkaW added 5 commits November 5, 2023 06:32

chore(precompile): remove watchpack

fa51c17

chore: replace chalk within @next/react-dev-overlay

f1c0684

chore: fix react-dev-overlay build

471e8be

chore(precompile): remove schema-utils2

66c44e6

chore: update precompiled files

04785e7

SukkaW force-pushed the remove-unused-precompile branch from a3caaed to 04785e7 Compare November 4, 2023 23:32

SukkaW marked this pull request as ready for review November 5, 2023 00:01

SukkaW requested review from timneutkens, ijjk, shuding, huozhi, feedthejim, ztanner, wyattjoh and a team as code owners November 5, 2023 00:01

Merge branch 'canary' into remove-unused-precompile

b4395a7

Merge branch 'canary' into remove-unused-precompile

b0ede68

SukkaW and others added 8 commits November 6, 2023 00:22

chore(precompile): remove loader-utils2

177b891

chore(precompile): re-add watchpack

97e0c86

Merge branch 'canary' into remove-unused-precompile

7e28efb

Merge branch 'canary' into remove-unused-precompile

1384094

chore: fixup lockfile

e6262eb

Merge branch 'canary' into remove-unused-precompile

f57b71b

Merge branch 'canary' into remove-unused-precompile

9bb8f02

Merge branch 'canary' into remove-unused-precompile

7ddfe33

SukkaW marked this pull request as draft January 6, 2024 11:10

SukkaW changed the title ~~chore(precompile): reduce the size of precompiled dependencies~~ [DO NOT MERGE] chore(precompile): reduce the size of precompiled dependencies Jan 6, 2024

This was referenced Jan 6, 2024

chore(precompile): re-add watchpack to the precompile #60309

Merged

chore(precompile): remove obsolete precompiled assets #60316

Merged

refactor(dev-overlay): remove chalk #60317

Merged

chore(font): enable minification #60319

Merged

SukkaW closed this Jan 13, 2024

SukkaW mentioned this pull request Jan 16, 2024

chore: replace micromatch w/ picomatch #60699

Merged

github-actions bot added the locked label Jan 28, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jan 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE] chore(precompile): reduce the size of precompiled dependencies #58038

[DO NOT MERGE] chore(precompile): reduce the size of precompiled dependencies #58038

SukkaW commented Nov 4, 2023 •

edited

Loading

ijjk commented Nov 4, 2023 •

edited

Loading

ijjk commented Nov 4, 2023 •

edited

Loading

feedthejim commented Nov 5, 2023

SukkaW commented Nov 5, 2023 •

edited

Loading

SukkaW commented Jan 5, 2024 •

edited

Loading

styfle commented Jan 5, 2024

SukkaW commented Jan 13, 2024 •

edited

Loading

[DO NOT MERGE] chore(precompile): reduce the size of precompiled dependencies #58038

[DO NOT MERGE] chore(precompile): reduce the size of precompiled dependencies #58038

Conversation

SukkaW commented Nov 4, 2023 • edited Loading

ijjk commented Nov 4, 2023 • edited Loading

Failing test suites

ijjk commented Nov 4, 2023 • edited Loading

Stats from current PR

feedthejim commented Nov 5, 2023

SukkaW commented Nov 5, 2023 • edited Loading

SukkaW commented Jan 5, 2024 • edited Loading

styfle commented Jan 5, 2024

SukkaW commented Jan 13, 2024 • edited Loading

SukkaW commented Nov 4, 2023 •

edited

Loading

ijjk commented Nov 4, 2023 •

edited

Loading

ijjk commented Nov 4, 2023 •

edited

Loading

SukkaW commented Nov 5, 2023 •

edited

Loading

SukkaW commented Jan 5, 2024 •

edited

Loading

SukkaW commented Jan 13, 2024 •

edited

Loading