Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CPU usage 100%, app frozen #42

Closed
dandv opened this issue Aug 5, 2014 · 11 comments
Closed

CPU usage 100%, app frozen #42

dandv opened this issue Aug 5, 2014 · 11 comments

Comments

@dandv
Copy link
Member

dandv commented Aug 5, 2014

It looks like within the past few hours, Kadira has been blocking our app. htop showed 100% or more CPU usage, and http://stockbase.com:3000 would never complete loading.

After removing kadira from .meteor/packages and the Kadira.connect(Meteor.settings.Kadira.appId, Meteor.settings.Kadira.appSecret); server code, the app runs normally (<5% CPU usage).

What made me suspect Kadira was the culprit is that I saw the Kadira: successfully authenticated message, but not the Kadira: completed instrumenting the app one despite waiting for ~15 minutes.

@arunoda
Copy link
Member

arunoda commented Aug 5, 2014

Is that only with past couple of hours?
what are the changes you've done to the app recently?
Does it occurred with users or without?
If that's with users, is there any change in user access patterns recently.

If possible, we can schedule a Dtrace profiling session for your app. So we might can findout the issue.

@dandv
Copy link
Member Author

dandv commented Aug 5, 2014

I've just re-enabled Kadira and used the app with only one user. CPU got to 100% in 4 minutes. The console says Connection timeout. No DDP heartbeat received.

The problem has been happening only for the past hour actually, since around 21:30 Pacific.

There have been changes in user access patterns before 21:30 today, because we launched officially, but right now it's past market hours so there are no users on the site.

Unfortunately we need to keep this production instance up so I'll be disabling Kadira for the moment.

@arunoda
Copy link
Member

arunoda commented Aug 5, 2014

Okay. It seems fair to remove kadira.
Are you using a custom meteor fork? If so send me that.

@arunoda arunoda closed this as completed Aug 5, 2014
@arunoda arunoda reopened this Aug 5, 2014
@arunoda
Copy link
Member

arunoda commented Aug 6, 2014

What's the node version you are running? I kind a see high CPU usage under node 0.10.30 on Telescope. But didn't check after removing Kadira.

@arunoda
Copy link
Member

arunoda commented Aug 6, 2014

I can confirm this is with node version 0.10.30 I'll find a fix.

@dandv
Copy link
Member Author

dandv commented Aug 6, 2014

Standard Meteor 0.8.3, node v0.10.30. Kadira also caused a ~4x memory leak (150MB -> ~600-700MB app size in RAM) in our backend app, which has no clients connecting to the server. That app does a lot of Mongo inserts.

@arunoda
Copy link
Member

arunoda commented Aug 6, 2014

Thanks dan. We are on the way to fix it. Let's see.

On Wednesday, August 6, 2014, Dan Dascalescu notifications@github.com
wrote:

Standard Meteor 0.8.3, node v0.10.30. Kadira also caused a ~4x memory leak
(150MB -> ~600-700MB app size in RAM) in our backend app, which has no
clients connecting to the server. That app does a lot of Mongo inserts.


Reply to this email directly or view it on GitHub
#42 (comment).


Arunoda Susiripala

@arunoda http://twitter.com/arunoda
http://gplus.to/arunodahttps://github.com/arunoda
http://www.linkedin.com/in/arunoda

@arunoda
Copy link
Member

arunoda commented Aug 7, 2014

dtrace shows it as a timers issue. So here's the bug: nodejs/node-v0.x-archive#8065

I'll try to give it a fix anyway from our side.

@dandv
Copy link
Member Author

dandv commented Aug 7, 2014

Awesome, thanks for the work! Looking forward to re-enable Kadira.

arunoda added a commit that referenced this issue Aug 7, 2014
node 0.10.30 introduced a bug which caused 100% CPU usage
if do a timeout with a non integer value. see: #8065
And it causes #42
This will fix that issue on kadira
@arunoda
Copy link
Member

arunoda commented Aug 7, 2014

Dan, update kadira and I've done a fix. Can you please check.

Note:

  • Meteor 0.8.3 uses node version 0.10.29
  • Always use the bundled node version comes with the meteor release

@arunoda
Copy link
Member

arunoda commented Oct 23, 2014

This bug is not anymore and only node version 0.10.29 had this issue.

@arunoda arunoda closed this as completed Oct 23, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants