"Could not check out a connection in 5.0 seconds" - YouTube servers stopped serving HTTP3/QUIC #2577

ThatIsAPseudo · 2021-11-12T10:37:28Z

⚠️ We pushed a fix to Invidious, update your instance against the latest code in order to have a working experience again.
Make sure to also turn to false use_quic in the config file if updating doesn't fix the bug. Removing the parameter use_quic from the config file works too.

If you use a public instance, and it doesn't work anymore, then please try to use another one from this list: https://instances.invidious.io

Basically, Google servers stopped serving HTTP3/QUIC and Invidious used to exclusively use HTTP3/QUIC for connecting to YouTube servers. That's why Invidious didn't really work anymore.

Original issue:

Describe the bug
Can't load anything on any invidious instance (edit: except the home page)

Steps to Reproduce

Go to any invidious instance
Click on a video/search for a term
See error

Logs

Title: Could not check out a connection in 5.0 seconds (DB::PoolTimeout)
Date: 2021-11-12T10:31:35Z
Route: /search?q=test+search
Version: 2021.11.08-21879da8 @ master

Backtrace

Could not check out a connection in 5.0 seconds (DB::PoolTimeout)
  from /usr/share/crystal/src/int.cr:540:9 in 'checkout'
  from src/invidious/yt_backend/connection_pool.cr:34:14 in '_post_json'
  from src/invidious/search.cr:36:11 in 'search'
  from /usr/share/crystal/src/primitives.cr:266:3 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/filter_handler.cr:21:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from src/invidious/helpers/static_file_handler.cr:189:11 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call'
  from /usr/share/crystal/src/http/server/handler.cr:28:7 in 'call_next'
  from lib/kemal/src/kemal/init_handler.cr:12:7 in 'process'
  from /usr/share/crystal/src/http/server.cr:500:5 in '->'
  from /usr/share/crystal/src/primitives.cr:266:3 in 'run'
  from ???

The text was updated successfully, but these errors were encountered:

ThatIsAPseudo · 2021-11-12T10:42:50Z

Probably linked to an undergoing Google outage in Europe : https://news.ycombinator.com/item?id=29197832

unixfox · 2021-11-12T11:37:50Z

Basically, Google servers stopped serving HTTP3/QUIC and Invidious exclusively use HTTP3/QUIC for connecting to YouTube servers. That's why Invidious didn't really work anymore.

See iv-org#2577

Rjevski · 2021-11-12T12:27:04Z

Edit: as a fix has been merged to master I will restrict my instance back for my own personal usage so you will see 403s on the URL below. This instance was never supposed to be public to begin with and this was just a temporary exception. Apologies for the inconvenience.

@syeopite your branch has been deployed to https://invidious.rjevski.io and appears to work. Feel free to use it in the meantime until other instances are updated. Just FYI, this instance does log minimal personal data such as IP addresses and user-agents in HTTP logs, though I have no incentive to use any of it and it's just there for debug purposes.

SamantazFox · 2021-11-12T16:39:09Z

Also poke @vituperative and @Cysea

trentwiles · 2021-11-12T17:18:50Z

All done, thanks, any idea why Google shut down quic support?

TheFrenchGhosty · 2021-11-12T17:19:38Z

@RiversideRocks We have no idea, it doesn't even make sense considering they use it themselves. It's just weird.

Rjevski · 2021-11-12T22:32:02Z

@RiversideRocks @TheFrenchGhosty there's been outages on Google services today so I wonder if disabling QUIC is a mitigation strategy for them (maybe QUIC infrastructure is dead but legacy HTTP isn't)?

unixfox · 2021-11-12T22:36:14Z

@RiversideRocks @TheFrenchGhosty there's been outages on Google services today so I wonder if disabling QUIC is a mitigation strategy for them (maybe QUIC infrastructure is dead but legacy HTTP isn't)?

Probably QUIC/HTTP3 doesn't even work:

[emilien@pcryzen]: ~>$ curl3 --http3 -I -v https://www.youtube.com
*   Trying 2a00:1450:4009:822::200e:443...
* Connect socket 5 over QUIC to 2a00:1450:4009:822::200e:443
* Sent QUIC client Initial, ALPN: h3-29,h3-28,h3-27
*   Trying 216.58.212.238:443...
* Connect socket 6 over QUIC to 216.58.212.238:443
* Sent QUIC client Initial, ALPN: h3-29,h3-28,h3-27
* After 149973ms connect time, move on!
* connect to 2a00:1450:4009:822::200e port 443 failed: Connection timed out
*   Trying 2a00:1450:4009:823::200e:443...
* Connect socket 7 over QUIC to 2a00:1450:4009:823::200e:443
* Sent QUIC client Initial, ALPN: h3-29,h3-28,h3-27
* After 74887ms connect time, move on!
* connect to 2a00:1450:4009:823::200e port 443 failed: Connection timed out
*   Trying 2a00:1450:4009:817::200e:443...
* Connect socket 5 over QUIC to 2a00:1450:4009:817::200e:443
* Sent QUIC client Initial, ALPN: h3-29,h3-28,h3-27
* After 37444ms connect time, move on!
* connect to 2a00:1450:4009:817::200e port 443 failed: Connection timed out
*   Trying 2a00:1450:4009:80a::200e:443...
* Connect socket 7 over QUIC to 2a00:1450:4009:80a::200e:443
* Sent QUIC client Initial, ALPN: h3-29,h3-28,h3-27
* After 18722ms connect time, move on!
* connect to 2a00:1450:4009:80a::200e port 443 failed: Connection timed out
* Connection timed out after 300000 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 300000 milliseconds

And youtube servers are trying to tell every browser to stop using QUIC/HTTP3 with this HTTP header:

[emilien@pcryzen]: ~>$ curl -I https://www.youtube.com
HTTP/2 200
alt-svc: clear

More about alt-svc here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Alt-Svc

trentwiles · 2021-11-12T22:38:50Z

Maybe they are trying to limit UDP packets to determine the problem? No idea

trentwiles · 2021-11-12T22:46:14Z

Also, would it be possible to make the code fall back to HTTP/2 or HTTP/1.1 if it detects QUIC is down?

unixfox · 2021-11-12T22:48:06Z

Also, would it be possible to make the code fall back to HTTP/2 or HTTP/1.1 if it detects QUIC is down?

Not easily, but I guess we will probably stop using QUIC because the native HTTP client in crystal is much more stable than lsquic.

@syeopite thoughts?

Joshndroid · 2021-11-13T00:47:05Z

I did a fresh docker update about 20 minutes ago and am also seeing this error

Title: Could not check out a connection in 5.0 seconds (DB::PoolTimeout)Date:2021-11-13T00:41:42Z``

Edit - Is the latest docker on quay.io up to date with the changes?

unixfox · 2021-11-13T00:49:49Z

I did a fresh docker update about 20 minutes ago and am also seeing this error

Title: Could not check out a connection in 5.0 seconds (DB::PoolTimeout)Date:2021-11-13T00:41:42Z``

Check if use_quic is set to true in the config file. If that's the case then turn it to false.

Joshndroid · 2021-11-13T00:58:34Z

I did a fresh docker update about 20 minutes ago and am also seeing this error
Title: Could not check out a connection in 5.0 seconds (DB::PoolTimeout)Date:2021-11-13T00:41:42Z``

Check if use_quic is set to true in the config file. If that's the case then turn it to false.

Absolutely perfect. This was exactly the next step I needed to get it going again. Thanks for the super quick reply

DmitrySandalov · 2021-11-13T10:32:17Z

Built from d214a0b, works fine for me (use_quic is absent in the config).

ianswett · 2021-11-16T17:23:45Z

Given UDP can also be blocked, I'd highly suggest falling back to a prior version of TCP when HTTP/3 fails.

From the HTTP/3 spec, section 3:
"Connectivity problems (e.g., blocking UDP) can result in QUIC
connection establishment failure; clients SHOULD attempt to use TCP-
based versions of HTTP in this case."

unixfox · 2021-11-16T18:29:42Z

Given UDP can also be blocked, I'd highly suggest falling back to a prior version of TCP when HTTP/3 fails.

From the HTTP/3 spec, section 3: "Connectivity problems (e.g., blocking UDP) can result in QUIC connection establishment failure; clients SHOULD attempt to use TCP- based versions of HTTP in this case."

While I completely agree with you, the current state of the invidious code is not made to support an automatic fallback to HTTP1/HTTP2 in TCP. That's why we allow the user, for example if UDP is blocked, to switch from QUIC to HTTP1 with the setting use_quic.

unixfox · 2021-11-17T09:41:41Z

Closing as YouTube just recently restored QUIC connections.

[emilien@pcryzen]: ~>$ curl -I https://www.youtube.com
HTTP/2 200
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

You may now turn use_quic to true if you are experiencing some issue when QUIC is not enabled, or you may not touch anything if "everything" works on your side.

ghost · 2021-11-28T10:39:48Z

Bonjour @unixfox
Where did you get curl with http3 support? This is an experimental option and as far as I know is not enabled by default.

Of course, I compiled curl --with-nghttp3 --with-ngtcp2 and openssl-quic library in Ubuntu 16.04 i386, it wasn't easy though. You probably have a package from AUR with quiche (rust)?

UPD1: Yea, I see from the binary name that you took from AUR. So I have a pretty rare alternate implementation. It works too, but requires c++17.

UPD2: I also compiled curl with quiche (boringssl). It's easier because it uses only one external quiche lib and enough c++11/14.

UPD3: People are so fond of not saying anything back, but I'm used to it.

This update disables QUIC by default which fixes Invidious not loading anything except for the home page due to YouTube no longer accepting HTTP/3 (Upstream Issue: iv-org/invidious#2577). It therefore uses Crystal’s internal HTTP client, which failed because the statically linked boringssl (required by lsquic) overrides OpenSSL’s CA certificate file location. This is fixed by applying the same patch to boringssl that is applied to openssl for using the correct CA certificate file.

See iv-org#2577

ThatIsAPseudo added the bug Something isn't working label Nov 12, 2021

This comment has been minimized.

Sign in to view

This was referenced Nov 12, 2021

[Bug] #2578

Closed

bug #2579

Closed

unixfox changed the title ~~[Bug] Could not check out a connection in 5.0 seconds (DB::PoolTimeout) - [every invidious instances]~~ YouTube servers stopped serving HTTP3/QUIC Nov 12, 2021

unixfox added the type:youtube-change Google changed something (again) label Nov 12, 2021

unixfox changed the title ~~YouTube servers stopped serving HTTP3/QUIC~~ YouTube servers stopped serving HTTP3/QUIC - Could not check out a connection in 5.0 seconds Nov 12, 2021

thansk mentioned this issue Nov 12, 2021

[Bug] #2582

Closed

syeopite pinned this issue Nov 12, 2021

This comment has been minimized.

Sign in to view

syeopite added a commit to syeopite/invidious that referenced this issue Nov 12, 2021

Disable quic by default

a120f14

See iv-org#2577

This was referenced Nov 12, 2021

Add compile-time flag to remove code for QUIC #2364

Merged

[Bug] #2583

Closed

This comment has been minimized.

Sign in to view

syeopite mentioned this issue Nov 12, 2021

[Bug] i don't really know what this is #2584

Closed

This comment has been minimized.

Sign in to view

unixfox mentioned this issue Nov 12, 2021

[Bug] Could not check out a connection in 5.0 seconds (DB::PoolTimeout) #2599

Closed

This comment has been minimized.

Sign in to view

syeopite mentioned this issue Nov 14, 2021

[Bug] "YouTube removed the dislike button by PewDiePie" will not load on Invidious #2609

Closed

sbruder mentioned this issue Nov 14, 2021

invidious: unstable-2021-11-08 -> unstable-2021-11-13 NixOS/nixpkgs#145882

Closed

12 tasks

SamantazFox mentioned this issue Nov 15, 2021

[Bug] Healthchecks error 500 keep restarting container #2619

Closed

unixfox closed this as completed Nov 17, 2021

unixfox unpinned this issue Nov 17, 2021

sriegler pushed a commit to sriegler/invidious that referenced this issue Nov 14, 2023

Disable quic by default

0537245

See iv-org#2577

"Could not check out a connection in 5.0 seconds" - YouTube servers stopped serving HTTP3/QUIC #2577

"Could not check out a connection in 5.0 seconds" - YouTube servers stopped serving HTTP3/QUIC #2577

Comments

ThatIsAPseudo commented Nov 12, 2021 • edited by unixfox Loading

This comment has been minimized.

ThatIsAPseudo commented Nov 12, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

unixfox commented Nov 12, 2021 • edited Loading

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Rjevski commented Nov 12, 2021 • edited Loading

This comment has been minimized.

SamantazFox commented Nov 12, 2021

This comment has been minimized.

This comment has been minimized.

trentwiles commented Nov 12, 2021

TheFrenchGhosty commented Nov 12, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Rjevski commented Nov 12, 2021

unixfox commented Nov 12, 2021 • edited Loading

trentwiles commented Nov 12, 2021

trentwiles commented Nov 12, 2021

unixfox commented Nov 12, 2021

Joshndroid commented Nov 13, 2021 • edited Loading

unixfox commented Nov 13, 2021 • edited Loading

Joshndroid commented Nov 13, 2021

DmitrySandalov commented Nov 13, 2021 • edited Loading

ianswett commented Nov 16, 2021

unixfox commented Nov 16, 2021

unixfox commented Nov 17, 2021

ghost commented Nov 28, 2021 • edited by ghost Loading

ThatIsAPseudo commented Nov 12, 2021 •

edited by unixfox

Loading

unixfox commented Nov 12, 2021 •

edited

Loading

Rjevski commented Nov 12, 2021 •

edited

Loading

unixfox commented Nov 12, 2021 •

edited

Loading

Joshndroid commented Nov 13, 2021 •

edited

Loading

unixfox commented Nov 13, 2021 •

edited

Loading

DmitrySandalov commented Nov 13, 2021 •

edited

Loading

ghost commented Nov 28, 2021 •

edited by ghost

Loading