Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timing out HTTP requests via the etcd proxy causes leaked HTTP connections #9335

Closed
rkday-pro opened this issue Feb 18, 2018 · 0 comments · Fixed by #9336
Closed

Timing out HTTP requests via the etcd proxy causes leaked HTTP connections #9335

rkday-pro opened this issue Feb 18, 2018 · 0 comments · Fixed by #9336

Comments

@rkday-pro
Copy link
Contributor

Since etcd v3.2.0, if you run the following:

./etcd --name mymaster
./etcd --name myproxy --proxy on --listen-client-urls=http://localhost:3379
wget -O - --timeout=0.5 --tries 1 'http://localhost:3379/v2/keys/xyz?waitIndex=703&recursive=false&wait=true'

then the wget request will time out, but netstat will show a connection staying up between the etcd proxy and the master. If you make enough wget requests, etcd will eventually run out of file handles.

This was regressed by 9def4cb. I've confirmed that by running the above command sequence with an etcd built from that commit - which did leak a connection - and from its parent, which didn't.

That commit:

  • doesn't call cancel() on the request when it detects that the client connection has been closed
  • instead, it has defer cancel() so the request is cancelled when the function exits. (This is necessary according to https://golang.org/pkg/context/#WithCancel - "Canceling this context releases resources associated with it, so code should call cancel as soon as the operations running in this Context complete.")
  • however, if the client connection is closed but cancel() isn't called, the io.Copy on the final line blocks forever, so the function never exits and cancel() isn't called.

Fortunately, cancel() can be called multiple times safely (https://golang.org/pkg/context/#CancelFunc - "After the first call, subsequent calls to a CancelFunc do nothing."), so the fix should just be to reinstate the cancel() when we detect that the connection has closed, alongside the defer cancel().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging a pull request may close this issue.

1 participant