DNS names that can't be resolved in Colima, possibly only with gvproxy network driver #466

rfay · 2022-11-07T17:00:19Z

Description

I'm starting this issue so we can start to track down the specific DNS addresses that fail in colima/lima, and the sources of information. I get this question all the time, and tell people to use --dns 1.1.1.1 and it almost always fixes. But I think we should start to track what they are so maybe we can solve this someday.

Issue	hostname
ddev/ddev#4372	mavtek-840225427682.d.codeartifact.us-east-1.amazonaws.com
ddev/ddev#4413	www.youtube.com (seems to be youtube-ui.l.google.com)
#466 (comment)	test12345.s3.ap-northeast-1.amazonaws.com

Version

Colima Version: Various
Lima Version:
Qemu Version:

Operating System

macOS Intel
macOS M1
Linux

Workarounds

Many people have reported in the comments that changing to the slirp network driver resolved the issue.

The text was updated successfully, but these errors were encountered:

paihu · 2022-11-26T04:23:16Z

I have the same issue.

Reproduction Steps

docker network create test
docker run --rm -it --network test alpine
apk add --no-cache curl && curl test12345.s3.ap-northeast-1.amazonaws.com
docker network rm test

Not Reproduction Steps

docker network create test
docker run --rm -it --network test alpine
apk add --no-cache curl && curl test1234.s3.ap-northeast-1.amazonaws.com
docker network rm test

Reproduce if fqdn is more than 41 characters and non default docker network

workaround

In my case...

open ~/.colima/default/colima.yml

edit network.driver

network:
  driver: slirp

rfay · 2022-11-29T22:23:49Z

Added youtube.com to the list in OP

renatho · 2022-11-29T22:35:27Z

Added youtube.com to the list in OP

Notice that youtube.com worked for me, but www.youtube.com didn't work. 😉

rfay · 2022-11-29T22:47:40Z

Edited, thanks @renatho

abiosoft · 2022-11-30T02:25:46Z

If indeed using slirp as the network driver fixes it, this should be resolved by the next release v0.5.0.

Schrank · 2022-12-15T09:56:22Z

I can add sbp-plugin-binaries.s3.eu-west-1.amazonaws.com

abiosoft · 2022-12-15T10:16:32Z

I would like to know if this is still the case for v0.5.0.

paihu · 2022-12-15T11:24:00Z

Thanks.

Fixed in my environment.

adrienthebo · 2023-02-27T17:52:51Z

I've observed sporadic failures with golang.org; I'm running on a 2021 Mac M1 Silicon using the vz virtualization driver. This manifests when using the devcontainer cli to build workspace images.

 $ yq '.network.driver' "$(colima template --print)"
gvproxy

$ colima version
colima version 0.5.2
git commit: 6b5b6fe0540e708f0c9d6e8919fab292c671fc72

runtime: docker
arch: aarch64
client: v23.0.1
server: v20.10.20

taylorchu · 2023-03-08T22:10:03Z

this is still not fixed in 0.5.4

abiosoft · 2023-03-29T16:53:43Z

I got bitten by this today as well and I can confirm it only happens with gvproxy network.

It appears some DNS queries fail for whatever reason.

I am still investigating.

gpsa · 2023-03-29T16:55:43Z

Same here:

When I:

nslookup test.s3-website-us-east-1.amazonaws.com

Server:		192.168.107.1
Address:	192.168.107.1:53

Non-authoritative answer:

**server can't find test.s3-website-us-east-1.amazonaws.com: NXDOMAIN**

But if I use Google's 8.8.8.8:

nslookup  test.s3-website-us-east-1.amazonaws.com 8.8.8.8
Server:		8.8.8.8
Address:	8.8.8.8:53

Non-authoritative answer:
test.s3-website-us-east-1.amazonaws.com	canonical name = s3-website.us-east-1.amazonaws.com

Non-authoritative answer:
test.s3-website-us-east-1.amazonaws.com	canonical name = s3-website.us-east-1.amazonaws.com
Name:	s3-website.us-east-1.amazonaws.com
Address: 52.217.87.195
Name:	s3-website.us-east-1.amazonaws.com
Address: 52.216.27.3
Name:	s3-website.us-east-1.amazonaws.com
Address: 52.216.98.42
Name:	s3-website.us-east-1.amazonaws.com
Address: 52.216.243.67
Name:	s3-website.us-east-1.amazonaws.com
Address: 52.217.140.13
Name:	s3-website.us-east-1.amazonaws.com
Address: 52.216.57.53
Name:	s3-website.us-east-1.amazonaws.com
Address: 52.217.10.139
Name:	s3-website.us-east-1.amazonaws.com
Address: 52.217.137.173

If I change nw driver for: slirp then now the problem is that host.docker.internal is being resolved via /etc/hosts but I need to be resolved via DNS Lookup:

nslookup host.docker.internal
Server:		127.0.0.11
Address:	127.0.0.11:53

** server can't find host.docker.internal: NXDOMAIN

** server can't find host.docker.internal: NXDOMAIN

abiosoft · 2023-03-29T16:59:52Z

If I change nw driver for: slirp then now the problem is that host.docker.internal is being resolved via /etc/hosts but I need to be resolved via DNS Lookup

@gpsa can you kindly open another issue for this? This is likely a bug.

gpsa · 2023-03-29T17:39:50Z

If I change nw driver for: slirp then now the problem is that host.docker.internal is being resolved via /etc/hosts but I need to be resolved via DNS Lookup

@gpsa can you kindly open another issue for this? This is likely a bug.

I could, but just to clarify, is the slirp driver expected to resolve host.docker.internal via DNS Lookup?

abiosoft · 2023-03-29T17:45:50Z

I could, but just to clarify, is the slirp driver expected to resolve host.docker.internal via DNS Lookup?

@gpsa I suspect your issue was changing the network driver of an existing VM.

This is what I get for slirp, it uses DNS lookup as well.

nslookup host.docker.internal
Server:		192.168.5.3
Address:	192.168.5.3:53

Non-authoritative answer:
Name:	host.docker.internal
Address: 192.168.5.2

Non-authoritative answer:

gpsa · 2023-03-29T17:49:16Z

I could, but just to clarify, is the slirp driver expected to resolve host.docker.internal via DNS Lookup?

@gpsa I suspect your issue was changing the network driver of an existing VM.

This is what I get for slirp, it uses DNS lookup as well.
nslookup host.docker.internal
Server:		192.168.5.3
Address:	192.168.5.3:53

Non-authoritative answer:
Name:	host.docker.internal
Address: 192.168.5.2

Non-authoritative answer:

@abiosoft Is there a way to recreate it without destroying everything? I could try to see if by recreating would work

abiosoft · 2023-03-29T17:53:16Z

@abiosoft Is there a way to recreate it without destroying everything? I could try to see if by recreating would work

@gpsa yeah. It's a regression actually, used to work before. You can edit the /etc/resolv.conf file in the VM and set the nameserver IP to 192.168.5.3.

In fact, it is the only entry in the file so you can simply replace it

colima ssh -- sudo sh -c 'echo "nameserver 192.168.5.3" > /etc/resolv.conf'

gpsa · 2023-03-30T13:46:12Z

@abiosoft Is there a way to recreate it without destroying everything? I could try to see if by recreating would work

@gpsa yeah. It's a regression actually, used to work before. You can edit the /etc/resolv.conf file in the VM and set the nameserver IP to 192.168.5.3.

In fact, it is the only entry in the file so you can simply replace it
colima ssh -- sudo sh -c 'echo "nameserver 192.168.5.3" > /etc/resolv.conf'

@abiosoft thank you so much, that worked like a breeze. Now both internal Docker DNS and external domains work just fine on SLIRP.

henrik242 · 2023-03-30T14:04:05Z

Could the DNS issues somehow be related to Alpine?

From https://martinheinz.dev/blog/92:

Usually, you would not notice this difference, because most of the time a single UDP packet (512 bytes) is enough to resolve hostnames... until it isn't enough and your application (running on Kubernetes) that previously worked completely fine for months suddenly starts throwing "Unknown Host" exceptions for one particular (very critical) hostname. The worst part is that this can manifest randomly, anytime when some external network change causes the resolution of some particular domain to require more than the 512 bytes available in single UDP packet.

abiosoft · 2023-03-30T14:42:19Z

Could the DNS issues somehow be related to Alpine?

@henrik242 I have actually read something similar before but I do not think this situation is related to Alpine, considering that slirp works fine.

As for why Alpine is the choice for Colima, you can check this comment #291 (comment).

gpsa · 2023-03-30T18:12:19Z

Could the DNS issues somehow be related to Alpine?

@henrik242 I have actually read something similar before but I do not think this situation is related to Alpine, considering that slirp works fine.

As for why Alpine is the choice for Colima, you can check this comment #291 (comment).

SLIRP mode is now "crashing" the same way Lima alone was behaving. So, basically the mounting points stop working and:
On the Host

docker ps
Cannot connect to the Docker daemon at unix:///Users/user/.colima/default/docker.sock. Is the docker daemon running?

rfay · 2023-03-30T19:00:19Z

@gpsa you're making a bit of a mess of this issue. Could you please open one that's on-topic for your issues?

gpsa · 2023-04-04T13:58:44Z

@gpsa you're making a bit of a mess of this issue. Could you please open one that's on-topic for your issues?

Sorry about that, I've then created a separated issue for the SLIRP one

AndreasA · 2023-04-26T08:17:07Z

when starting colima with VZ vmtype and virtiofs and providing --dns 192.168.5.3 then AWS hostname resolution seems to fail as well. without it seems to work but results in the pulling speed issues #648 - no matter if slirp or gvproxy is used though i think for VZ vm type the network driver setting is probably ignored..

taylorchu · 2023-04-26T20:44:03Z

@abiosoft
https://wiki.musl-libc.org/functional-differences-from-glibc.html

Multiple reports on weird musl dns incompatibility with glibc. I think it is safer to use base image like debian for this.

ryancurrah · 2023-06-08T15:51:26Z

I would like to use Debian as well to see if it resolves this issue for us. Is that possible?

gchait · 2023-08-09T10:16:48Z

After some messing around, this seems to be the fix:

colima delete
colima start --edit

Change gvproxy to slirp.
With such a limitation/bug, I wonder why it's not the default.

mandrasch · 2023-08-09T13:40:05Z

If anyone wants to switch, the following should also possible

colima start --edit
# change value with "i" insert mode, switch to slirp
# save via ":wq:"

Or edit ~/.colima/default/colima.yaml and re-start colima via colima stop and colima start.

No need for colima delete (as far as I know).

gchait · 2023-08-09T14:35:41Z

If anyone wants to switch, the following should also possible
colima start --edit
# change value with "i" insert mode, switch to slirp
# save via ":wq:"
Or edit ~/.colima/default/colima.yaml and re-start colima via colima stop and colima start.

No need for colima delete (as far as I know).

For me, after simply restarting nothing seemed to be working.
To be more specific, a docker build failed right at the beginning, because it could not even resolve registry-1.docker.io. It was an i/o timeout right there, suggesting all/most networking was broken in the VM.
I got the idea for the delete from here.

mandrasch · 2023-08-09T16:15:42Z

Hi! I started with colima version 0.5.5 two months ago and changing the config + restart worked fine for me today (without deleting).

@rfay just mentioned in DDEV discord the following:

If you have had your colima instance through many updates, it's a worthwhile thing to delete it and recreate it. (After saving away databases of course via ddev snapshot -a)

So depends on how many updates happened in the meantime I guess?

AndreasA · 2023-08-09T17:51:14Z

Change gvproxy to slirp. With such a limitation/bug, I wonder why it's not the default.

Just wondering, but colima start --network-driver slirp should work as well, shouldn't it? It would be easier to use in a command for setup (no need to search/replace in the config file).

Though the last time I tried it, it made no difference with virtual machine type vz, but I admit I did not delete the instance, so maybe that helps, though not sure if the network driver is even relevant for vz but it is worth a try.

mandrasch · 2023-08-10T07:47:05Z

Just wondering, but colima start --network-driver slirp should work as well, shouldn't it? It would be easier to use in a command for setup (no need to search/replace in the config file).

Does this replace and save things in the current configuration before starting? Would be cool! (I'll try later, thanks for hint).

skirsdeda · 2023-09-01T12:21:35Z

I hit this while running a container which does a lot of AWS service requests. DNS resolution would fail after some time when using vz vm, then subsequent run would fail almost immediately and only colima restart helped to get more time without DNS failures. And with qemu and slirp network driver it was actually even worse. So I resorted to Docker Desktop which runs without problems. Sad.

admxxi · 2023-10-25T17:10:00Z

Same here, having issue while using vmType: vz and network drivers gvproxy or slirp still getting loads of error while trying to solve DNS, but I would say 50% of the requests fail.

AndreasA · 2023-11-02T05:39:00Z

Hi, just wondering but which lima version are you using because #648 seems to be fixed - at least it looks like it so far - with the latest lima 0.18.x update and it was related to DNS as well, so it might also fix these issues?

jdmarshall · 2023-12-12T18:21:20Z

I'm still getting connection refused on 127.0.0.11

I don't have a local dns server on dev machines and I can't figure out what the solution is here. How do we avoid this?

The latest version of Colima doesn't even have a driver field in the yaml file and I'm still having this problem.

xuwhite · 2024-08-03T20:18:11Z

I'm still getting connection refused on 127.0.0.11

I don't have a local dns server on dev machines and I can't figure out what the solution is here. How do we avoid this?

The latest version of Colima doesn't even have a driver field in the yaml file and I'm still having this problem.

same here
sadly the only workaround that works for me is to add a dns address with colima start --dns 8.8.8.8 or in the config file ~/.colima/default/colima.yaml if the dns changes I have to restart colima colima restart to make the dns work again
see #711

rfay mentioned this issue Nov 7, 2022

Colima: One DNS address lookup failing in container (*. us-east-1.amazonaws.com) ddev/ddev#4372

Closed

1 task

rfay mentioned this issue Nov 29, 2022

Can't resolve hostname www.youtube.com inside web container on Colima ddev/ddev#4413

Closed

1 task

abiosoft mentioned this issue Dec 21, 2022

Fast dns lookups freeze the docker vm #529

Closed

3 tasks

rfay changed the title ~~DNS names that can't be resolved in Colima~~ DNS names that can't be resolved in Colima, possibly only with gvproxy network driver Mar 16, 2023

rfay mentioned this issue Mar 29, 2023

DNS not resolving AWS domains #674

Closed

5 tasks

AndreasA mentioned this issue Apr 26, 2023

Slow image pulling when using VZ VM type #648

Closed

rfay mentioned this issue May 12, 2023

Colima not inheriting system DNS - cannot docker login or pull #711

Open

5 tasks

rfay mentioned this issue Nov 1, 2023

dns stops working after a while #779

Closed

5 tasks

rfay mentioned this issue Nov 7, 2023

Calling URLs with curl works only sporadic ddev/ddev#5506

Closed

1 task

abiosoft added this to the v0.6.0 milestone Nov 10, 2023

abiosoft closed this as completed Nov 12, 2023

DNS names that can't be resolved in Colima, possibly only with gvproxy network driver #466

DNS names that can't be resolved in Colima, possibly only with gvproxy network driver #466

Comments

rfay commented Nov 7, 2022 • edited Loading

Description

Version

Operating System

Workarounds

paihu commented Nov 26, 2022 • edited by abiosoft Loading

Reproduction Steps

Not Reproduction Steps

workaround

rfay commented Nov 29, 2022

renatho commented Nov 29, 2022

rfay commented Nov 29, 2022

abiosoft commented Nov 30, 2022

Schrank commented Dec 15, 2022

abiosoft commented Dec 15, 2022

paihu commented Dec 15, 2022

adrienthebo commented Feb 27, 2023 • edited Loading

taylorchu commented Mar 8, 2023

abiosoft commented Mar 29, 2023

gpsa commented Mar 29, 2023

abiosoft commented Mar 29, 2023

gpsa commented Mar 29, 2023

abiosoft commented Mar 29, 2023

gpsa commented Mar 29, 2023

abiosoft commented Mar 29, 2023

gpsa commented Mar 30, 2023 • edited Loading

henrik242 commented Mar 30, 2023

abiosoft commented Mar 30, 2023

gpsa commented Mar 30, 2023

rfay commented Mar 30, 2023

gpsa commented Apr 4, 2023

AndreasA commented Apr 26, 2023 • edited Loading

taylorchu commented Apr 26, 2023

ryancurrah commented Jun 8, 2023

gchait commented Aug 9, 2023

mandrasch commented Aug 9, 2023

gchait commented Aug 9, 2023 • edited Loading

mandrasch commented Aug 9, 2023 • edited Loading

AndreasA commented Aug 9, 2023

mandrasch commented Aug 10, 2023

skirsdeda commented Sep 1, 2023

admxxi commented Oct 25, 2023

AndreasA commented Nov 2, 2023

jdmarshall commented Dec 12, 2023 • edited Loading

xuwhite commented Aug 3, 2024

rfay commented Nov 7, 2022 •

edited

Loading

paihu commented Nov 26, 2022 •

edited by abiosoft

Loading

adrienthebo commented Feb 27, 2023 •

edited

Loading

gpsa commented Mar 30, 2023 •

edited

Loading

AndreasA commented Apr 26, 2023 •

edited

Loading

gchait commented Aug 9, 2023 •

edited

Loading

mandrasch commented Aug 9, 2023 •

edited

Loading

jdmarshall commented Dec 12, 2023 •

edited

Loading