dns issue with docker alpine:3.13 (that does not happen with 3.12) when running on a vagrant image with --natdnshostresolver1

[gbenguria]

docker run -it --rm alpine:3.12 ping -c1 google.com -> this works
docker run -it --rm alpine:3.13 ping -c1 google.com -> this does not work
docker run -it --rm –dns 8.8.8.8 alpine:3.13 ping -c1 google.com -> this works

In the first two statements docker uses the dns provided by virtualbox "10.0.2.3", which provides some advatages when switching frequently of work place.

How to reproduce the issue:

• I use virtual box 6.1.18 and vagrant 2.2.14
• just run the following vagrantfile with vagrant up and vagrant provision to repeat.

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/bionic64"
  config.vm.provider "virtualbox" do |vb|
	vb.name = "alpine_3.13_dns_issue"
	vb.gui = true
	vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
  end
  
  config.vm.provision "shell", name: "dockerInstall", inline: <<-SHELL
    apt-get update
    DEBIAN_FRONTEND=noninteractive apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
	curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
	add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
    DEBIAN_FRONTEND=noninteractive apt-get install -y docker-ce docker-ce-cli containerd.io
  SHELL
  
  config.vm.provision "shell", name: "pull required images", inline: <<-SHELL
    docker pull alpine:3.13
    docker pull alpine:3.12
    docker pull debian:latest
  SHELL

  config.vm.provision "shell", name: "dns check", inline: <<-SHELL
    echo "VBOX DNS with alpine:3.12 -> SUCCESS"
    docker run -i --rm alpine:3.12 /bin/sh -c "nslookup google.com; ping -c1 google.com"
	echo
	echo
    echo "VBOX DNS with debian:latest -> SUCCESS"
    docker run -i --rm debian:latest /bin/sh -c "cat /etc/resolv.conf | grep nameserver; ping -c1 google.com"
	echo
	echo
    echo "VBOX DNS with alpine:3.13 -> ***FAIL***"
    docker run -i --rm alpine:3.13 /bin/sh -c "nslookup google.com; ping -c1 google.com"
	echo
	echo
    echo "GOOGLE DNS with alpine:3.13 -> SUCCESS"
    docker run -i --rm --dns 8.8.8.8 alpine:3.13 /bin/sh -c "nslookup google.com; ping -c1 google.com"
  SHELL
end

Expected output

    default: VBOX DNS with alpine:3.12 -> SUCCESS
    default: Server:            10.0.2.3
    default: Address:   10.0.2.3:53
    default:
    default: Non-authoritative answer:
    default: Name:      google.com
    default: Address: 216.58.215.142
    default:
    default: ** server can't find google.com: NXDOMAIN
    default: PING google.com (216.58.215.142): 56 data bytes
    default: 64 bytes from 216.58.215.142: seq=0 ttl=114 time=19.382 ms
    default:
    default: --- google.com ping statistics ---
    default: 1 packets transmitted, 1 packets received, 0% packet loss
    default: round-trip min/avg/max = 19.382/19.382/19.382 ms
    default:
    default:
    default: VBOX DNS with debian:latest -> SUCCESS
    default: nameserver 10.0.2.3
    default: PING google.com (216.58.215.142) 56(84) bytes of data.
    default: 64 bytes from mad41s04-in-f14.1e100.net (216.58.215.142): icmp_seq=1 ttl=114 time=19.4 ms
    default:
    default: --- google.com ping statistics ---
    default: 1 packets transmitted, 1 received, 0% packet loss, time 0ms
    default: rtt min/avg/max/mdev = 19.400/19.400/19.400/0.000 ms
    default:
    default:
    default: VBOX DNS with alpine:3.13 -> ***FAIL***
    default: Server:            10.0.2.3
    default: Address:   10.0.2.3:53
    default:
    default: Non-authoritative answer:
    default: Name:      google.com
    default: Address: 216.58.215.142
    default:
    default: ** server can't find google.com: NXDOMAIN
    default: ping: bad address 'google.com'
    default:
    default:
    default: GOOGLE DNS with alpine:3.13 -> SUCCESS
    default: Server:            8.8.8.8
    default: Address:   8.8.8.8:53
    default:
    default: Non-authoritative answer:
    default: Name:      google.com
    default: Address: 172.217.17.14
    default:
    default: Non-authoritative answer:
    default: Name:      google.com
    default: Address: 2a00:1450:4003:808::200e
    default: PING google.com (172.217.17.14): 56 data bytes
    default: 64 bytes from 172.217.17.14: seq=0 ttl=114 time=27.025 ms
    default:
    default: --- google.com ping statistics ---
    default: 1 packets transmitted, 1 packets received, 0% packet loss
    default: round-trip min/avg/max = 27.025/27.025/27.025 ms

[JeroenvdV]

This is not limited to VirtualBox. I run into the same issue as described in docker-library/php#1134 using Docker version 20.10.3, build 48d30b5 in WSLv1 with Docker Desktop 3.1.0 (51484)

[Logiqx]

Another setup - Docker Desktop 3.2.1 on Windows 10 + WSL2

I'm also seeing DNS issues with alpine:3.13. These issues aren't present in 3.11 or 3.12.

#155 (comment)

[danielefranceschi]

Verified also in kubernetes 1.19.6 on docker 19.3.14:

$ kubectl run -ti --rm --image=alpine:3.13 --image-pull-policy=Always testalpine313
/ # apk update
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/main: DNS lookup error
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/main: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/community: DNS lookup error
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/community: No such file or directory
2 errors; 14 distinct packages available
/ # wget http://www.google.com
wget: bad address 'www.google.com'

while 3.12 is fine:

$ kubectl run -ti --rm --image=alpine:3.12 --image-pull-policy=Always testalpine312
/ # apk update
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
v3.12.4-15-g2a00f4f299 [http://dl-cdn.alpinelinux.org/alpine/v3.12/main]
v3.12.4-16-gd3ffa75891 [http://dl-cdn.alpinelinux.org/alpine/v3.12/community]
OK: 12748 distinct packages available
/ # wget http://www.google.com
Connecting to www.google.com (172.217.10.36:80)
saving to 'index.html'

[connesc]

I’m facing the same problem in a VirtualBox guest running with --natdnshostresolver1.

In case it can help, the relevant change has been introduced in musl 1.2.2 with commit 19f8642494b7d27b2ceed5c14d4a0b27cb749afe (2020-08-24 22:02:14 -0400).

Alpine also backported this change to musl-1.2.1-r1 with commit 9f0c2c144a17e79f28b4f6544401d399dc77c0f8 (2020-09-03 21:12:39 +0000), which is included in Alpine 1.13.

The root cause seems to be how the internal NAT DNS resolver of VirtualBox handles AAAA requests.
More details here: alpine/aports#11879.

[avoidik]

I was also affected by this problem, I'm on Linux Mint 20.3, I thought it could be related to the system-resolved configuration, but in the system-resolved daemon logs I was able to clearly see that DNS requests were processed and forwarded back to requestor (in my case back to VirtualBox VM), per https://gitlab.alpinelinux.org/alpine/aports/-/issues/11879 thread there is a workaround to change NAT DNS configuration as follows:

VBoxManage modifyvm "minikube" --natdnshostresolver1 off
VBoxManage modifyvm "minikube" --natdnsproxy1 on

With this change I was able to build alpine-based Docker image. HTH

[jstangroome]

This is a bug in Virtualbox, finally fixed in v6.1.36.

NAT: Prevent issue when host resolver incorrectly returned NXDOMAIN for unsupported queries (bug #20977)

[JeroenvdV]

This is a bug in Virtualbox, finally fixed in v6.1.36.

NAT: Prevent issue when host resolver incorrectly returned NXDOMAIN for unsupported queries (bug #20977)

Is there also a solution for the same situation outside of Virtualbox?

#149 (comment)

This is not limited to VirtualBox. I run into the same issue as described in docker-library/php#1134 using Docker version 20.10.3, build 48d30b5 in WSLv1 with Docker Desktop 3.1.0 (51484)