Next.js development high memory usage

[timneutkens]

Before posting a comment on this issue please read this entire post.

Previous work

The past few weeks we've been investigating / optimizing various memory usage issues. Specifically geared towards production memory usage. In investigating these we were able to find there was one memory leak in Node.js itself when using fetch() in Node.js versions before 18.17.0 (you'll want to use 18.17.1 for security patches though).

Most of the reports related to memory usage turned out to be reports of "it's higher than the previous version" rather than a memory leak. This was expected because in order to run App Router and Pages Router at the same time with different React versions two separate processes were needed. This has been resolved by reducing the amount of processes to two, one for Routing and App Router rendering, and one for Pages Router rendering. So far we haven't received new reports since the latest release.

In some issues there were reports related to Image Optimization in production, however no reproduction was provided so it could not be investigated adequately, if you have a reproduction for that please refer to this issue: #54482

New

With the memory usage in production resolved we've started investigating reports of development memory usage spikes. Unfortunately these reports suffer from the same problem as the production memory usage issues people raised before, they're full of comments saying same issue or posting screenshots of monitoring tools saying "Look, same issue".

Unfortunately, as you can image, these replies are not enough to investigate / narrow down what causes the memory usage, for example in multiple cases that we did get a reproduction and could investigate the reason for the high memory usage was:

• A bug in the application code, causing infinite looping components
• Accidental import of ~11.000 modules through icon libraries -- Yes, many icon libraries ship massive amounts of re-exports and those have to be bundled before it can be tree-shaken. We've been working on an automated way to split these up in optimize_barrel SWC transform and new optimizePackageImports config #54572 that should help a bit to reduce the size (and compilation speed too).
• Webpack customization, e.g. adding external libraries that change webpack settings, for example to write additional sourcemaps

So far I've been able to make one small change to webpack's memory caching to make it garbage collect a bit more aggressively in #54397. I'm not expecting that change to have a big impact on the reported issues though.

We'd like to investigate these reports further, however we're unable to narrow these down if there is no code to run to collect heap snapshots and profiles for, hence this issue. If you are able to please provide runnable code of what you're experiencing.

Comments that don't include runnable code will be automatically hidden in order to keep this issue productive. This includes comments that only have a screenshot and applications that can't run.

I'm going to combine the other reports into this issue as separate comments.

I've made sure that we have 2-3 engineers on our team available to investigate when we get runnable reproductions to investigate.

Thanks in advance!

_NEXT-1569

[timneutkens]

@noetix Disagree that this had anything to do with bullying. All I did was explain that my posts are continuously being ignored, including in this issue. Then I explained what to do, which is to provide a runnable application. There is nothing we can do without a runnable example, which was already shared in the initial issue, I even made it bold to highlight that further.

Happy to explain it again, the reason we can't do anything without runnable code is that in order to narrow down the memory usage we need to change the Next.js core code in the application, for example to disable client components compilation and such in order to narrow down where the memory usage comes from. There is no way to do that based on screenshots / messages / information you can provide as it would require countless hours of your time and our time (think 2 weeks full time at least) in order to investigate this.

The emoji reactions not being shown for off-topic marked posts is a bug in GitHub. As mentioned in the initial issue any posts that don't include a reproduction will be automatically hidden.

Since you didn't like the earlier explanation I'll just remove it, don't feel strongly about keeping the comment. Definitely wasn't bullying, you were reading into that. Bullying would be the threats I've received recently from anonymous developers on Twitter that they'll come visit my house soon...

@weyert we haven't made changes to development memory usage besides the PR linked in the issue so really all I need is a reproduction, luckily @AhmedChabayta posted one, hopefully that is enough, fingers crossed.

@Thinkscape please open a separate issue, that bug would be separate from this issue 👍

[limeburst]

I've posted a reproduction here: https://github.com/limeburst/vercel-nextjs-54708

Start the development server, navigate from

• http://localhost:3000/1
• http://localhost:3000/2
• ...
• http://localhost:3000/20

And watch the memory usage grow, until the server restarts.

[rajat1saxena]

You can use this code to see the issue. The server is getting aborted silently without any errors.

https://github.com/codelitdev/courselit/tree/tailwindcss-2

Logs

rajat@rajat-laptop:~/projects/courselit$ yarn dev
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env.local
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env
- ready started server on [::]:3000, url: http://localhost:3000
- event compiled client and server successfully in 545 ms (18 modules)
- wait compiling...
- event compiled client and server successfully in 263 ms (18 modules)
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env.local
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env.local
- info Loaded env from /home/rajat/projects/courselit/apps/web/.env
- wait compiling /404 (client and server)...
- wait compiling / (client and server)...
rajat@rajat-laptop:~/projects/courselit$

[stigmartins1]

I'm getting 'server out of memory' after a while by letting the server run and writing/saving code that calls the following functions a few times.

import { google } from "googleapis";

export async function authSheets() {
  //Function for authentication object
  const auth = new google.auth.GoogleAuth({
    keyFile: "./auth/auth-sa-sptk.json",
    scopes: ["https://www.googleapis.com/auth/spreadsheets"],
  });

  //Create client instance for auth
  const authClient = await auth.getClient();

  //Instance of the Sheets API
  const sheets = google.sheets({ version: "v4", auth: authClient });

  return {
    auth,
    authClient,
    sheets,
  };
}

import { authSheets } from "./authSheets";

export async function clearSheetContents(sheetName) {
  console.log("sheet =", sheetName);
  const SHEET_ID = "123";
  const sheetId = SHEET_ID;
  const { sheets } = await authSheets();

  try {
    const result = await sheets.spreadsheets.values.clear({
      spreadsheetId: sheetId,
      range: sheetName,
    });
    console.log("result.data =", result.data);
  } catch (err) {
    // TODO (developer) - Handle exception
    throw err;
  }
}

import { authSheets } from "./authSheets";

// https://developers.google.com/sheets/api/guides/values
export async function setSheetValues(sheetName, input) {
  const SHEET_ID = "123";
  const sheetId = SHEET_ID;
  const values = [input];
  const resource = { values };
  // Updates require a valid ValueInputOption parameter
  const valueInputOption = "RAW"; // The input is not parsed and is inserted as a string.
  const { sheets } = await authSheets();

  try {
    const result = await sheets.spreadsheets.values.append({
      spreadsheetId: sheetId,
      range: sheetName,
      valueInputOption: valueInputOption,
      resource,
    });
    console.log("result.data =", result.data);
  } catch (err) {
    // TODO (developer) - Handle exception
    throw err;
  }
}

{
  "name": "test",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "autoprefixer": "10.4.15",
    "axios": "^1.5.0",
    "encoding": "^0.1.13",
    "eslint": "8.48.0",
    "eslint-config-next": "13.4.19",
    "googleapis": "^126.0.1",
    "next": "13.4.19",
    "postcss": "8.4.29",
    "react": "18.2.0",
    "react-dom": "18.2.0",
    "tailwindcss": "3.3.3"
  }
}

Node v. v18.17.1

[glennin-codes]

have the same issue ,when running npm run dev ,it freezes my browser i have to shutdown the laptop or close the vs code and the browser too.sometimes If I wanna test on the browser I have to kill one application i.e the vs code and view my changes in the browser when done kill the next server and continue coding.This is so borrowing . am using the following
Node v18.17.1 "next" "13.4.3", "typescript": "5.0.4",

[alex-statsig]

After doing some profiling, it seems like server modules are leaked across HMR in development, which in my company's case leads to rapid growth of memory due to a large backend (see #62217, auto-closed unfortunately). Curious if anyone else can repro / confirm the same thing, or confirm theirs don't leak.

[Hanskrogh]

This is not only an issue in development, but also production.......

We're using pages router with middleware and It seems like there is a memory leak in get cookies inside middleware.

[m-rphy]

I have pinned down the issue to next's global fetch implementation. I've created a repo that is very simple and re-creates the memory leak and uses the same logic that by passes it by using node's http module. There seems to be an issue surrounding PerformanceRequestTimers and fetchMetrics. More background can be found in issue #64212

[JClackett]

Was wondering why my laptop was slowly dying...

Tried using turbo but still got up to 8GB, any ideas? Am slowly trying to get a repro, starting with testing out removing the icon libraries, initial tests are still returning +10GB of memory used, it's a fairly large app thats slowly being migrated from pages to app router, so creating a repro is taking a lot of work.

Added note: I'm also curious about the rationale of the fix being "it must be your icon library". are we just now expected to vet every package we use in case it causes a memory leak in next? was this always a thing? I understand that we should be conscious of what packages we use, but im maintaining a load of next.js projects that have different icons libraries, ui libraries, etc do I just have to migrate off these now to move to the app router? is there no way to fix memory issues in next, for example with react-icons?

[rzzapp]

Similar to @JClackett I just hovering around 6GB memory usage. Also reloads takes like 30s for normal size pages.

[timneutkens]

Hey everyone, as part of Turbopack we've also implemented thorough memory tracing which allows for narrowing down why certain applications use a lot of memory. Can you follow these steps when using Turbopack and send us the relevant files:

If possible follow these steps which would give me the best picture to investigate:

• npm install next@canary (use the package manager you're using) -- We want to make sure you're using the very latest version of Next.js which includes the fixes mentioned earlier.
• rm -rf .next
• start development using the NEXT_CPU_PROF=1 and NEXT_TURBOPACK_TRACING=1 (regardless of if you're using Turbopack, it only affects when you do use Turbopack) environment variable. E.g.:
  • npm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 npm run dev
  • yarn: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 yarn dev
  • pnpm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 pnpm dev
• Wait a few seconds
• Open a page that you're working on
• Wait till it's fully loaded
• Wait a few seconds
• Make an edit to a file that holds a component that is on the page
• Wait for the edit to apply
• Wait a few seconds
• Make another edit to the same file
• Wait a few seconds
• Exit the dev command (ctrl+c)
• Upload the CPU traces put in the root of the application directory to https://gist.github.com
• Upload the .next/trace file to https://gist.github.com -- Please don't run trace-to-tree yourself, as I use some other tools (e.g. Jaeger) that require the actual trace file.
• If you're using Turbopack upload the .next/trace.log as well, if it's too large for GitHub gists you can upload it to Google Drive or Dropbox and share it through that.
• Upload next.config.js (if you have one) to https://gist.github.com
• Share it here

Added note: I'm also curious about the rationale of the fix being "it must be your icon library". are we just now expected to vet every package we use in case it causes a memory leak in next? was this always a thing? I understand that we should be conscious of what packages we use, but im maintaining a load of next.js projects that have different icons libraries, ui libraries, etc do I just have to migrate off these now to move to the app router? is there no way to fix memory issues in next, for example with react-icons?

It's important to note that this is not a memory leak, it's high memory usage, which is very different than leaking memory. Leaking memory means the memory is unused but can't be cleaned up. In this case the memory is used (i.e. memory caches of webpack and such).

It's also important to note that Turbopack memory usage is not fully optimized yet, right now it uses more memory than where we want the memory usage to be for Turbopack, at least a 50% reduction is what we're aiming for right now and that is being worked on currently.

Incorrect usage of icon libraries is just one cause of high memory usage, will try to explain why a bit:

There's libraries that ship 10.000 re-exported modules that all have to be analyzed before tree-shaking.

Next.js needs to compile your code (including node_modules) for multiple targets depending on your application:

• Node.js
  • Server Components (App Router)
  • Client Components server-side (App Router)
  • Client Component client-side (App Router)
  • Pages Router server-side
  • Pages Router client-side
• runtime = 'edge' (if you're using it) -- This is a separate compilation
  • Middleware
  • Server Components (App Router)
  • Client Components server-side (App Router)
  • Client Component client-side (App Router)
  • Pages Router server-side
  • Pages Router client-side

If you use all of those combined you're compiling not 10.000 modules but:

• 10.000 for pages router (server)
• 10.000 for pages router (client)
• 10.000 for app router (server components)
• 10.000 for app router (client components server)
• 10.000 for app router (client components client)

Obviously people use some combination of these, but it illustrates why misusing icon imports can be one of the problems.

[AryanJ-NYC]

Hey everyone, as part of Turbopack we've also implemented thorough memory tracing which allows for narrowing down why certain applications use a lot of memory. Can you follow these steps when using Turbopack and send us the relevant files:

If possible follow these steps which would give me the best picture to investigate:

• npm install next@canary (use the package manager you're using) -- We want to make sure you're using the very latest version of Next.js which includes the fixes mentioned earlier.

• rm -rf .next

• start development using the NEXT_CPU_PROF=1 and NEXT_TURBOPACK_TRACING=1 (regardless of if you're using Turbopack, it only affects when you do use Turbopack) environment variable. E.g.:

  • npm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 npm run dev
  • yarn: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 yarn dev
  • pnpm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 pnpm dev

• Wait a few seconds

• Open a page that you're working on

• Wait till it's fully loaded

• Wait a few seconds

• Make an edit to a file that holds a component that is on the page

• Wait for the edit to apply

• Wait a few seconds

• Make another edit to the same file

• Wait a few seconds

• Exit the dev command (ctrl+c)

• Upload the CPU traces put in the root of the application directory to https://gist.github.com

• Upload the .next/trace file to https://gist.github.com -- Please don't run trace-to-tree yourself, as I use some other tools (e.g. Jaeger) that require the actual trace file.

• If you're using Turbopack upload the .next/trace.log as well, if it's too large for GitHub gists you can upload it to Google Drive or Dropbox and share it through that.

• Upload next.config.js (if you have one) to https://gist.github.com

• Share it here

Added note: I'm also curious about the rationale of the fix being "it must be your icon library". are we just now expected to vet every package we use in case it causes a memory leak in next? was this always a thing? I understand that we should be conscious of what packages we use, but im maintaining a load of next.js projects that have different icons libraries, ui libraries, etc do I just have to migrate off these now to move to the app router? is there no way to fix memory issues in next, for example with react-icons?

It's important to note that this is not a memory leak, it's high memory usage, which is very different than leaking memory. Leaking memory means the memory is unused but can't be cleaned up. In this case the memory is used (i.e. memory caches of webpack and such).

It's also important to note that Turbopack memory usage is not fully optimized yet, right now it uses more memory than where we want the memory usage to be for Turbopack, at least a 50% reduction is what we're aiming for right now and that is being worked on currently.

Incorrect usage of icon libraries is just one cause of high memory usage, will try to explain why a bit:

There's libraries that ship 10.000 re-exported modules that all have to be analyzed before tree-shaking.

Next.js needs to compile your code (including node_modules) for multiple targets depending on your application:

• Node.js

  • Server Components (App Router)
  • Client Components server-side (App Router)
  • Client Component client-side (App Router)
  • Pages Router server-side
  • Pages Router client-side

• runtime = 'edge' (if you're using it) -- This is a separate compilation

  • Middleware
  • Server Components (App Router)
  • Client Components server-side (App Router)
  • Client Component client-side (App Router)
  • Pages Router server-side
  • Pages Router client-side

If you use all of those combined you're compiling not 10.000 modules but:

• 10.000 for pages router (server)
• 10.000 for pages router (client)
• 10.000 for app router (server components)
• 10.000 for app router (client components server)
• 10.000 for app router (client components client)

Obviously people use some combination of these, but it illustrates why misusing icon imports can be one of the problems.

What about users who don't use Turbopack?

[timneutkens]

@AryanJ-NYC You can try capturing heap dumps but there's not much we can do with those without also seeing the source code, so in essence providing the full source code is required to investigate when you are using webpack. See my earlier replies in this issue, there's very little we can do without code that can be profiled.

For Turbopack we're in control of the memory allocator as it's not running in JavaScript and that's why we can measure memory usage for Turbopack during tracing.

[alex-statsig]

@timneutkens could you take a quick look at the dump in #62217? Auto-closed but I shared the heap dumps there (hard to share a repro unfortunately). I'm mainly curious if its expected for old module source-map strings to be retained in general, or if it indicates something not cleaned up properly? If its due to something not properly cleaned up, should there be mechanisms for that (#46018)? And should we expect so many modules to need to be recompiled (#45204)?

[cryptoMavrik]

Hey everyone, as part of Turbopack we've also implemented thorough memory tracing which allows for narrowing down why certain applications use a lot of memory. Can you follow these steps when using Turbopack and send us the relevant files:

If possible follow these steps which would give me the best picture to investigate:

• npm install next@canary (use the package manager you're using) -- We want to make sure you're using the very latest version of Next.js which includes the fixes mentioned earlier.

• rm -rf .next

• start development using the NEXT_CPU_PROF=1 and NEXT_TURBOPACK_TRACING=1 (regardless of if you're using Turbopack, it only affects when you do use Turbopack) environment variable. E.g.:

  • npm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 npm run dev
  • yarn: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 yarn dev
  • pnpm: NEXT_TURBOPACK_TRACING=1 NEXT_CPU_PROF=1 pnpm dev

• Wait a few seconds

• Open a page that you're working on

• Wait till it's fully loaded

• Wait a few seconds

• Make an edit to a file that holds a component that is on the page

• Wait for the edit to apply

• Wait a few seconds

• Make another edit to the same file

• Wait a few seconds

• Exit the dev command (ctrl+c)

• Upload the CPU traces put in the root of the application directory to https://gist.github.com

• Upload the .next/trace file to https://gist.github.com -- Please don't run trace-to-tree yourself, as I use some other tools (e.g. Jaeger) that require the actual trace file.

• If you're using Turbopack upload the .next/trace.log as well, if it's too large for GitHub gists you can upload it to Google Drive or Dropbox and share it through that.

• Upload next.config.js (if you have one) to https://gist.github.com

• Share it here

We are experiencing very high memory usage, build times, compilation times(60s+), and have had no luck tracking down the issue. Hot refreshes can take 5s+. Also our app router migration turned 9000 modules -> 15000 modules. Any help would be greatly appreciated!

https://gist.github.com/cryptoMavrik/2697c797d720d99f32e5a93b6fb8c2b0

[rirh]

Similar to @JClackett I just hovering around 4GB memory usage. Also reloads takes like 30s for normal size pages.

[sorenhoyer]

Node.js v. 20.15 seems to be the last version, before the big memory leak got introduced in v. 20.16 (a very very minor leak can still be observed, but it's nothing in comparison).

[karlhorky]

Node.js v. 20.15 seems to be the last version, before the big memory leak got introduced in v. 20.16

There's a Node.js memory leak in one of the latest LTS releases? Do you have an official source for that? Is it an undici leak?

Oh, and did you try 20.17.0 to compare?

[rirh]

Node.js v. 20.15 seems to be the last version, before the big memory leak got introduced in v. 20.16 (a very very minor leak can still be observed, but it's nothing in comparison).

Check out my updated pictures

[sorenhoyer]

Node.js v. 20.15 seems to be the last version, before the big memory leak got introduced in v. 20.16 (a very very minor leak can still be observed, but it's nothing in comparison).

Check out my updated pictures

I'm saying you need to use exactly v. 20.15.x not a later major/minor (you're using 22.8 according to your pictures) as some leak was introduced in Node 20.16 - at least that's what we saw.

[rirh]

Node.js v. 20.15 seems to be the last version, before the big memory leak got introduced in v. 20.16 (a very very minor leak can still be observed, but it's nothing in comparison).

Check out my updated pictures

I'm saying you need to use exactly v. 20.15.x not a later major/minor (you're using 22.8 according to your pictures) as some leak was introduced in Node 20.16 - at least that's what we saw.

This node version seems way too new. Honestly, it feels like the React team is just throwing everything into one big mix, which doesn’t really match their usual style. I genuinely hope they can make it work, but right now, it feels like they’re more focused on flashy new features and buzz rather than practical memory management. sad :(

[GabenGar]

20.16 is a minor version of the latest LTS major version released almost a year ago. Not to mention NodeJS hickups has nothing to do with React, especially in NextJS thread.

[dalalRohit]

my next-server using 10.67GB memory on MacBook Air M1.