Memory leak in recent releases of Sentry (>= 8.6.0) #12317
Comments
github-actions
bot
added
the
Package: nextjs
|
Hi, thanks for writing in about this. We are currently taking a look what might cause this! A question and an ask:
Also, if you have any custom code around Sentry feel free to share it here! |
|
@lforst if it helps, we have the same issue (also using Next.js) and it was caused by upgrading from 8.4.0 to 8.5.0, so I'm fairly confident the issue lies in here. We are currently pinned to 8.4.0 which doesn't have this issue. Here is our `instrumentation.ts` if it helps// This file configures the initialization of Sentry on the server.
// The config you add here will be used whenever the server handles a request.
// https://docs.sentry.io/platforms/javascript/guides/nextjs/
import { checkIfKnownBot, cleanSentryEvent } from '@scripts/clean-sentry-events'
import * as Sentry from '@sentry/nextjs'
/* eslint no-process-env: "off" */
const SENTRY_DSN = process.env.SENTRY_DSN || process.env.NEXT_PUBLIC_SENTRY_DSN
export async function register() {
if (process.env.NEXT_RUNTIME === 'nodejs') {
Sentry.init({
dsn: SENTRY_DSN,
tracesSampleRate: 0.25,
environment: process.env.ENVIRONMENT,
beforeSend: (...args) => {
const [event, hint] = args
// Sentry now treats error information as `unknown` which means that the only way to handle this types is really treat them as any and check for existence on runtime
// https://github.com/getsentry/sentry-javascript/pull/7361
const originalException = hint?.originalException as any
const isKnownBot = checkIfKnownBot(event.request?.headers?.['user-agent'])
if (isKnownBot) {
return null
}
const mainAndMaybeCauseErrors = event.exception?.values?.slice(-2) ?? []
for (const error of mainAndMaybeCauseErrors) {
const { type } = error
const is404 = type === 'Not Found'
const is403 = type === 'Forbidden'
const is401 = type === 'Unauthorized'
const isStripeCardError = originalException?.data?.reason?.raw?.type === 'card_error'
if (is404 || is401 || is403 || isStripeCardError) {
return null
}
}
return cleanSentryEvent('server', ...args)
},
sendDefaultPii: false,
ignoreErrors: [
/**
* Error when the user is using a invalid JWT or an JWT that points to a different user.
* This is not actionable and is often caused by employee using multiple accounts at the same time.
*/
/Not Found/i,
/Forbidden/i,
/Unauthorized/i,
/**
* Error when we check an expired or malformed token. This is not actionable.
*/
/Received invalid bearer token/i,
],
integrations: [
Sentry.captureConsoleIntegration({
levels: ['error', 'warn'],
}),
],
})
}
if (process.env.NEXT_RUNTIME === 'edge') {
Sentry.init({
dsn: SENTRY_DSN,
tracesSampleRate: 0.25,
environment: process.env.ENVIRONMENT,
beforeSend: (...args) => {
const [event] = args
const isKnownBot = checkIfKnownBot(event.extra?.userAgent as string)
if (isKnownBot) {
return null
}
return cleanSentryEvent('edge', ...args)
},
sendDefaultPii: false,
ignoreErrors: [
/**
* Error when the user is using a invalid JWT or an JWT that points to a different user.
* This is not actionable and is often caused by employee using multiple accounts at the same time.
*/
/Not Found/i,
/Forbidden/i,
/Unauthorized/i,
/**
* Error when we check an expired or malformed token. This is not actionable.
*/
/Received invalid bearer token/i,
/**
* Validation of expired tokens are not actionable
*/
/"exp" claim timestamp check failed/i,
],
integrations: [],
})
}
} |
|
Also seeing this on a bump from 8.3.0 to 8.5.0 last week. |
|
@AlecRust that helps a lot narrowing it down. Thanks! I'll investigate further. |
|
One more question to the people following this issue. Does anybody experience this with other SDKs than |
|
Update: I am struggling to find the culprit. If anybody is able to share a memory profile with this happening that would be awesome. Also, any information on what kind of instrumentation/database ORM is used is super useful. |
|
Another note - if you remove We've also heard memory concerns with people using Postgres - is anyone using that? |
|
I am like 90% confident I found the leak and I hope never having to touch a memory profiler in my life again: #12335 Why it leaks, no idea, but it leaks. |
|
I figured out the issue now but it's not entirely clear to me why it was triggered yet. On node calling However only the So to get a The memory dump that @lforst shared with me indicates that we have timers leaked in The repro case is trivial: // leaks
for (i = 0; i < 500000; i++) {
+setTimeout(() => {}, 0);
}This will create 500000 un-collectable > x = {}
{}
> x[setTimeout(() => {}, 0)] = 42;
42
> x
{ '119': 42 }So there might be some patterns in either our codebase or in opentelemetry that do that. Independent of that we should open an issue against node as there is clearly a bug there Timer is removed here from the list but not from Compare this to how |
|
Always love it when SDK bugs actually reveal bugs with node.js or the browser 😂 We spent some time trying to reproduce why the timeout becomes a primitive. With just initializing the SDK neither Next we looked at So this means the problem is with Next.js for sure. @lforst ran some tests and apparently sometimes Next.js patches
This functionality was introduced in Next to fix another Node.js bug 😓 vercel/next.js#57235 I'll let @lforst say it best 
|
|
Hey, we've just released 8.8.0 which should hopefully fix this issue! Let us know if you still notice any problems. |
|
Experiencing huge memory leaks on @sentry/node@8.9.2, will downgrade to 8.3.0 |
|
@AnthonyDugarte memory leaks are p0 for us to fix, could you open a new GH issue with details about your setup? We'll get someone on that asap! |
|
Also getting memory leaks on 8.28.0. From my crude analysis, it does appear to be the same Timeout issue:
|
|
We experience memory leak in (at least) @sentry/bun@8.34.0. |
have you resolved this? I also get this ! :(( |
|
Hey, I fix this by use node lts 20.18.0 nodejs/node#53337 |
|
I get this error too. My VPS has upto 8/12GB Ram available... here is my log. |
|
@IRediTOTO this seems like a bug in Bun from looking at the logs. I recommend you follow up with the Bun team to fix this. Switching to Node.js in the mean time will probably unblock you. |
No, I tried many ways
|
|
If you disable sentry, but enable generating sourcemaps does the error still occur? (basically is the problem with uploading or with actual sourcemap generation) You can check this with If the problem exists just with sourcemap generation, this is a next.js problem. It's nextjs/webpack which is generating the sourcemap. |
Is there an existing issue for this?
How do you use Sentry?
Sentry Saas (sentry.io)
Which SDK are you using?
@sentry/nextjs
SDK Version
8.7.0
Framework Version
No response
Link to Sentry event
No response
SDK Setup
Steps to Reproduce
Next.js application using
"@sentry/nextjs": "8.7.0",(here is a full list of dependencies)Service is deployed via AWS ECS + Fargate
We noticed that our first deploy following an upgrade from
8.3.0to8.6.0started causing our containers to hit their memory limits and crash + restart. We noticed this behavior happening across two separate Next.js applications / containers that were upgraded to8.6.0at the same time.Expected Result
Containers stay under memory limit.
Actual Result
Here is a memory usage graph from one of our containers. Version 8.3.0 does not appear to contain an issue, version > 8.6.0 does, we did not check versions 8.3.0 or 8.4.0
Here are some logs observed at time of crash:
The text was updated successfully, but these errors were encountered: