-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ostree pull hangs in low bandwidth environments (libcurl/http2?) #1362
Comments
Ah, hmm. Yes I've seen this too even with a fast connection, but I didn't diagnose it this far. |
This seems to work around ostreedev#1362 Though I'm not entirely sure why yet. But at least with this it'll be easier for people to work around things locally.
I don't quite understand why this suddenly exploded 💣 😢...possibly it's related to the server or libcurl version? Anyways one workaround appears to be #1368 |
Starting to see reports of this in the wild - https://twitter.com/Erstejahre/status/947795020597227521 |
I'm beating Colin to the punch here and making use of the new `--disable-http2`. What pushed me over the edge was the fact that I literally could not upgrade my Atomic Workstation with a simple `rpm-ostree upgrade` without hitting: ostreedev/ostree#1362 I originally thought this only occurred in low-bandwidth environments, when in fact it still happens even after upgrading my Internet bundle and can actually be reproduced very easily in environments where bandwidth is not an issue. E.g. this currently hangs in the latest Fedora 27 Atomic Host provisioned in the cloud: ostree remote add --no-gpg-verify faw27 https://dl.fedoraproject.org/ostree/27/ ostree pull faw27 fedora/27/x86_64/workstation Until we investigate deeper, let's just play it safe and disable HTTP2. This should also fix the HTTP2 framing layer people have been hitting. Let's get this in to confirm it doesn't break the rdgo streams, and then we can backport the patch for f27 to make sure it gets into the next TWR?
i'm hit this issue twice today with http2=false , i need to run three times rpm-ostree upgrade |
Using the libcurl backend on a machine with low bandwidth (< 500 kB/s here), it seems pretty easy to get ostree to hang during a pull operation. I've reproduced this in two separate environments. The issue seems present in git master/latest release/previous release. Disabling pipelining works around the issue.
Once it apparently hangs,
top
shows CPU usage goes to 100%. Usingstrace
andG_MESSAGES_DEBUG=all
shows that it's not actually waiting on a specific resource. In fact, data does come in, but it seems like only 1 of the 8 open sockets are attended to, while thePOLLIN
for the other sockets are completely ignored yet continuously polled.Notice how
poll
indicates that 7 of the sockets have data ready to be read. In other traces I've seen, only the 1 socket that is not ready is the one serviced, which of course fails withEAGAIN
. Running with:does show that we're passing on those notifications to libcurl.
The text was updated successfully, but these errors were encountered: