DNS resolution operating but no HTTP(S) traffic succeeding towards HA.io inside container

[ghost]

The problem

Hi,

This is a fresh docker-compose setup, no modifications done to the suggested yaml.

When started, the following comes up:

homeassistant    | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
homeassistant    | [s6-init] ensuring user provided files have correct perms...exited 0.
homeassistant    | [fix-attrs.d] applying ownership & permissions fixes...
homeassistant    | [fix-attrs.d] done.
homeassistant    | [cont-init.d] executing container initialization scripts...
homeassistant    | [cont-init.d] done.
homeassistant    | [services.d] starting services
homeassistant    | [services.d] done.
homeassistant    | 2021-10-06 12:37:28 ERROR (MainThread) [homeassistant.components.updater] Error requesting Home Assistant update data: Cannot connect to host www.home-assistant.io:443 ssl:default [Try again]

Executing a shell in the container for diagnostics gets me this:

/config # dig @XXX www.home-assistant.io

  ; <<>> DiG 9.16.20 <<>> @XXX www.home-assistant.io
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46264
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.home-assistant.io.         IN      A

;; ANSWER SECTION:
www.home-assistant.io.  563     IN      A       104.26.5.238
www.home-assistant.io.  563     IN      A       172.67.68.90
www.home-assistant.io.  563     IN      A       104.26.4.238

;; Query time: 4 msec
;; SERVER: XXX#53(XXX)
;; WHEN: Wed Oct 06 12:50:43 UTC 2021
;; MSG SIZE  rcvd: 98

/config # dig www.home-assistant.io

; <<>> DiG 9.16.20 <<>> www.home-assistant.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13364
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.home-assistant.io.         IN      A

;; ANSWER SECTION:
www.home-assistant.io.  552     IN      A       104.26.5.238
www.home-assistant.io.  552     IN      A       172.67.68.90
www.home-assistant.io.  552     IN      A       104.26.4.238

;; Query time: 0 msec
;; SERVER: XXX#53(XXX)
;; WHEN: Wed Oct 06 12:50:54 UTC 2021
;; MSG SIZE  rcvd: 98

/config # 

Which confirms DNS resolution is working fine, inside and outside the container.

Trying to directly access the addresses (they are Cloudflare hosted so this won't work obviously):

/config # curl https://104.26.5.238 -vvv
*   Trying 104.26.5.238:443...
* Connected to 104.26.5.238 (104.26.5.238) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
* Closing connection 0
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
/config # curl https://172.67.68.90
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
/config # curl https://172.67.68.90 -v
*   Trying 172.67.68.90:443...
* Connected to 172.67.68.90 (172.67.68.90) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
* Closing connection 0
curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

From outside the container:

# curl https://www.home-assistant.io  -vvv | head
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 104.26.4.238:443...
* Connected to www.home-assistant.io (104.26.4.238) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2347 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Jul 16 00:00:00 2021 GMT
*  expire date: Jul 15 23:59:59 2022 GMT
*  subjectAltName: host "www.home-assistant.io" matched cert's "*.home-assistant.io"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5586f688d470)
} [5 bytes data]
> GET / HTTP/2
> Host: www.home-assistant.io
> user-agent: curl/7.74.0
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 200 
....%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 699f0aa4ae476635-MAD
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
< 
{ [407 bytes data]
<!doctype html>
...

So, DNS is operating perfectly fine in my network, HTTP(S) access is not filtered for the outbound connections to the right hosts and there is no apparent reason for HA inside the container to fail.

I have exhausted all options to resolve the issue as an operator mistake/local configuration problem, because it obviously "isn't", so I am filing this bug hoping to get some guidance in locating the actual culprit.

What is version of Home Assistant Core has the issue?

core-(latest?)

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

No response

Link to integration documentation on our website

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[ghost]

OK, so odds are high this will happen to other people and I found the solution:

The culprit is a change in Alpine, the way it does DNS resolution is that it queries for both v4 and v6 records (A, AAAA) despite no v6 stack being present or used. When the DNS server responds with an empty AAAA record, it goes limp... regardless of whether the A records are perfectly fine. This is quite dumb, but it is an upstream alpine issue.

See here:
#52667
alpinelinux/docker-alpine#149
alpinelinux/docker-alpine#165
alpinelinux/docker-alpine#185

Now, on to the show:

If you are using a corporate firewall, or pfsense, or something similar, this is what you need to do to overcome this problem:

• Enable DNS64 support in your resolver (Unbound or whatever you use). Put on a prefix like 64:ff9b::/96 (RFC 6052) so AAAA records will be expanded/synthesized from A records gracefully, if not present at the upstream server.
• If you use any secondary DNS resolvers or forwarders, make sure the IPv6 functionality is enabled, so AAAA records aren't dropped.

Once that is done, this issue will be fixed and suddenly internal DNS resolution will work in the Alpine container.

Please make a very visible documentation note in the site to let people know about this. Your container depends on successful AAAA responses from the DNS server. This is NOT standard per se, it is entirely possible and normal to NOT provide AAAA records, and it is a security loophole to leave IPv6 enabled in networks where it isn't actually used (I could go into this topic but won't, it affects Windows hosts more so than *nix, due to some quirks in the Windows network stack and the way it prioritizes v6 over v4).

[acseven]

Based on your info, and commenting from a layman's point of view, I had this issue of components in general not being able to connect to hosts, using HA in a Docker container (running as net host) on a Synology DS1819+ NAS and [I believe to have] solved it by enabling IPV6 to Automatic in the network connection settings, which was disabled.

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.