Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http: free parser on keepalive #33167

Closed
wants to merge 1 commit into from

Conversation

ronag
Copy link
Member

@ronag ronag commented Apr 30, 2020

Free parser when putting socket back in the agent keep alive free list.

Fixes: #29394

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines

Free parser when putting socket back in the agent
keep alive free list.

Fixes: nodejs#29394
@nodejs-github-bot nodejs-github-bot added the http Issues or PRs related to the http subsystem. label Apr 30, 2020
@@ -712,10 +699,19 @@ function emitFreeNT(req) {
}

if (req.socket) {
freeSocket(req, req.socket);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the significant change.

@ronag ronag added the wip Issues and PRs that are still a work in progress. label Apr 30, 2020
@ronag
Copy link
Member Author

ronag commented Apr 30, 2020

@leidegre

@leidegre
Copy link
Contributor

leidegre commented Apr 30, 2020

@ronag unfortunately the problem is still there...

image

Looks like the problem is connected to onIncoming. It is holding on to the server and socket. The socket in turn is holding on to the _httpResponse which has an event handler that is holding on to the data.

Now, I have some concerns that the free list is running the constructor when the HTTPParser is allocated from the free list and not when it is put back into the free list for reuse. parsersCb in _http_common.js actually does clear onIncoming but this code is being run when the parser is recycled, not when it is finished, i.e. returned to the pool. This means that as long as the HTTPParser isn't reused it will retain data. It should release all its references when it is put back into the pool.

I'm also confused by why the code below isn't merged with cleanParser.

parser.onIncoming = null;
parser[kOnHeaders] = parserOnHeaders;
parser[kOnHeadersComplete] = parserOnHeadersComplete;
parser[kOnBody] = parserOnBody;
parser[kOnMessageComplete] = parserOnMessageComplete;
parser[kOnTimeout] = null;

After adding the parser.onIncoming = null; I get this:

image

Similar situation but different retainer. In this case it's the onParserTimeout function in _http_server.js. So I added parser[kOnTimeout] = null; as well to the cleanParser function and now finally, the memory is not being retained.

Echoing my original comment about whether it is truly meaningful to pool the HTTPParser. There's a lot of complexity here that goes away if we do not pool HTTPParser objects.

In summary, the memory leak goes away if we add these two lines to the cleanParser function.

+  parser[kOnTimeout] = null;
+  parser.onIncoming = null;

@ronag
Copy link
Member Author

ronag commented Apr 30, 2020

@leidegre Good digging there. I would add that comment to the original issue. I would not object to remove the pooling but I'm not sure about its performance implications.

@ronag
Copy link
Member Author

ronag commented Apr 30, 2020

@nodejs/http This PR does not solve the references issue. Is it still worth trying to land or should we close it?

@ronag
Copy link
Member Author

ronag commented Apr 30, 2020

In summary, the memory leak goes away if we add these two lines to the cleanParser function.

@leidegre Would you mind opening a PR with this change?

@leidegre
Copy link
Contributor

leidegre commented May 1, 2020

@ronag I'll open a PR. I think I did mention it in my first comment but I can reiterate my fidings here the as well put emphasis on whether the pooling is necessary.

@ronag ronag closed this May 1, 2020
leidegre added a commit to leidegre/node that referenced this pull request May 3, 2020
Flarna pushed a commit that referenced this pull request May 5, 2020
Fixes: #29394
Refs: #33167 (comment)

PR-URL: #33190
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Juan José Arboleda <soyjuanarbol@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
codebytere pushed a commit that referenced this pull request May 7, 2020
Fixes: #29394
Refs: #33167 (comment)

PR-URL: #33190
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Juan José Arboleda <soyjuanarbol@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
addaleax pushed a commit that referenced this pull request Sep 22, 2020
Fixes: #29394
Refs: #33167 (comment)

PR-URL: #33190
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Juan José Arboleda <soyjuanarbol@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Rich Trott <rtrott@gmail.com>
Reviewed-By: Gerhard Stöbich <deb2001-github@yahoo.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
http Issues or PRs related to the http subsystem. wip Issues and PRs that are still a work in progress.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Keepalive prevents garbage collection on unread responses
3 participants