Add a mode to next build to make it easier to debug memory issues

[mknichel]

This PR adds a --experimental-debug-memory-usage flag to next build to make it easier to debug memory performance. This mode does the following things:

• Periodically prints the current memory usage of the process
• Records garbage collection events and warns about long running GC events
• Kills the process if it detects GC thrashing near heap limit
• Automatically takes a heap snapshot if heap usage rises above 70% of the total heap
• Automatically takes a heap snapshot if the process is close to running out of memory
• Prints a report at the end of the build with information about peak memory usage and time spent in garbage collection

[ijjk]

Tests Passed

[ijjk]

Stats from current PR

Default Build

General Overall increase ⚠️

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
buildDuration	13.9s	13.9s	N/A
buildDurationCached	8.4s	6.2s	N/A
nodeModulesSize	199 MB	199 MB	⚠️ +65.8 kB
nextStartRea..uration (ms)	400ms	401ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
2453-HASH.js gzip	31.4 kB	31.4 kB	N/A
3304.HASH.js gzip	181 B	181 B	✓
3f784ff6-HASH.js gzip	53.7 kB	53.7 kB	✓
8299-HASH.js gzip	5.04 kB	5.04 kB	N/A
framework-HASH.js gzip	45.2 kB	45.2 kB	✓
main-app-HASH.js gzip	242 B	242 B	✓
main-HASH.js gzip	32.2 kB	32.2 kB	N/A
webpack-HASH.js gzip	1.68 kB	1.68 kB	N/A
Overall change	99.3 kB	99.3 kB	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
polyfills-HASH.js gzip	31 kB	31 kB	✓
Overall change	31 kB	31 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
_app-HASH.js gzip	196 B	197 B	N/A
_error-HASH.js gzip	184 B	184 B	✓
amp-HASH.js gzip	505 B	505 B	✓
css-HASH.js gzip	324 B	325 B	N/A
dynamic-HASH.js gzip	2.5 kB	2.5 kB	N/A
edge-ssr-HASH.js gzip	258 B	258 B	✓
head-HASH.js gzip	352 B	352 B	✓
hooks-HASH.js gzip	370 B	371 B	N/A
image-HASH.js gzip	4.21 kB	4.21 kB	✓
index-HASH.js gzip	259 B	259 B	✓
link-HASH.js gzip	2.67 kB	2.67 kB	N/A
routerDirect..HASH.js gzip	314 B	312 B	N/A
script-HASH.js gzip	386 B	386 B	✓
withRouter-HASH.js gzip	309 B	309 B	✓
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	6.57 kB	6.57 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
_buildManifest.js gzip	481 B	484 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
index.html gzip	529 B	529 B	✓
link.html gzip	542 B	541 B	N/A
withRouter.html gzip	524 B	523 B	N/A
Overall change	529 B	529 B	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
edge-ssr.js gzip	95.4 kB	95.4 kB	N/A
page.js gzip	3.06 kB	3.06 kB	✓
Overall change	3.06 kB	3.06 kB	✓

Middleware size

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
middleware-b..fest.js gzip	626 B	623 B	N/A
middleware-r..fest.js gzip	151 B	151 B	✓
middleware.js gzip	25.5 kB	25.5 kB	N/A
edge-runtime..pack.js gzip	839 B	839 B	✓
Overall change	990 B	990 B	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
app-page-exp...dev.js gzip	170 kB	170 kB	✓
app-page-exp..prod.js gzip	97 kB	97 kB	✓
app-page-tur..prod.js gzip	98.8 kB	98.8 kB	✓
app-page-tur..prod.js gzip	93 kB	93 kB	✓
app-page.run...dev.js gzip	144 kB	144 kB	✓
app-page.run..prod.js gzip	91.5 kB	91.5 kB	✓
app-route-ex...dev.js gzip	21.4 kB	21.4 kB	✓
app-route-ex..prod.js gzip	15.1 kB	15.1 kB	✓
app-route-tu..prod.js gzip	15.1 kB	15.1 kB	✓
app-route-tu..prod.js gzip	14.9 kB	14.9 kB	✓
app-route.ru...dev.js gzip	21.1 kB	21.1 kB	✓
app-route.ru..prod.js gzip	14.9 kB	14.9 kB	✓
pages-api-tu..prod.js gzip	9.55 kB	9.55 kB	✓
pages-api.ru...dev.js gzip	9.82 kB	9.82 kB	✓
pages-api.ru..prod.js gzip	9.55 kB	9.55 kB	✓
pages-turbo...prod.js gzip	22.5 kB	22.5 kB	✓
pages.runtim...dev.js gzip	23.1 kB	23.1 kB	✓
pages.runtim..prod.js gzip	22.4 kB	22.4 kB	✓
server.runti..prod.js gzip	51.1 kB	51.1 kB	✓
Overall change	945 kB	945 kB	✓

build cache

	vercel/next.js canary	vercel/next.js mknichel/memory-debugging-mode	Change
0.pack gzip	1.58 MB	1.58 MB	N/A
index.pack gzip	106 kB	106 kB	N/A
Overall change	0 B	0 B	✓

Diff details

Diff for middleware.js

Diff too large to display

Commit: 0065cce

[ijjk]

Do we want to grab the memory usage from inside of the worker when using the webpackBuildWorker mode?

[ijjk]

A small test to ensure the flag is properly honored would be solid to avoid regressing

[danielhjacobs]

After this PR, https://nextjs.org/ no longer works with PerformanceObserver undefined. See #64211.

[danielhjacobs]

https://github.com/vercel/next.js/blob/canary/packages/next/src/lib/memory/gc-observer.ts should be changed, see also #30387.

[danielhjacobs]

Suggestion:

const obs = window.PerformanceObserver && new PerformanceObserver((list) => {
  const entry = list.getEntries()[0]
  gcEvents.push(entry)

  if (entry.duration > LONG_RUNNING_GC_THRESHOLD_MS) {
    warn(bold(`Long running GC detected: ${entry.duration.toFixed(2)}ms`))
  }
})

/**
 * Starts recording garbage collection events in the process and warn on long
 * running GCs. To disable, call `stopObservingGc`.
 */
export function startObservingGc() {
  obs?.observe({ entryTypes: ['gc'] })
}

export function stopObservingGc() {
  obs?.disconnect()
}

[ijjk]

@danielhjacobs can you provide more information why you think this PR is related, this is only changing code that runs in the node environment and doesn't affect client side code.