-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to set extra_hosts
when using consul-connect (bridged networking)
#7746
Comments
extra_hosts
when using consul-connect (bridged networking)
I can confirm that this problem in not limited to just the I'm running Nomad v0.10.5 and Docker v18.06.1 with API version 1.38 and minimum version 1.12. It looks like the problem was fixed in Docker API version v1.12.0, see:
Nomad is developed against Docker version 1.8.2 and 1.9 (Official docs), meaning API version 1.20 and above (See Docker version matrix). For the time being I am unable to run connect enabled jobs with custom DNS servers because of this problem. The error I get is |
I tried to run a docker container using the command line and I am able to use docker run --rm -it --net bridge --dns 1.1.1.1 ubuntu:20.04 cat /etc/resolv.conf
search us-east-1.compute.internal
nameserver 1.1.1.1
options timeout:2 attempts:5
docker run --rm -it --net bridge --dns 8.8.8.8 ubuntu:20.04 cat /etc/resolv.conf
search us-east-1.compute.internal
nameserver 8.8.8.8
options timeout:2 attempts:5
docker run --rm -it --net bridge --dns 8.8.8.8 --dns 1.1.1.1 ubuntu:20.04 cat /etc/resolv.conf
search us-east-1.compute.internal
nameserver 8.8.8.8
nameserver 1.1.1.1
options timeout:2 attempts:5 |
So I captured the traffic between Nomad and the Docker socket and it turns out that the network mode is The new information for me is that the network mode specified in the jobspec is used for some other purpose. I tried to run a container with this new configuration e.g. |
Did some more digging. Now I'm certain that this is because of the CNI based network setup. Here is the call trace of network setup before the allocation is started:
In my opinion the final call to dnsConfig := cni.DNS{
Servers: []string{"1.1.1.1"},
Searches: []string{},
Options: []string{},
}
b.cni.Setup(ctx,
alloc.ID,
spec.Path,
cni.WithCapabilityPortMap(getPortMapping(alloc)),
cni.WithCapabilityDNS(dnsConfig)) should do the job just fine. I can try this change locally in a while, but it'd be great if someone who knows the codebase can verify the correctness of this patch in the meantime. |
@nickethier could you offer some insight here please? |
Hey all I missed this one when linking issues but the dns part of this issue is merged and will be in the next major release. See: #7661 We're still evaluating the |
With regards to the extra_hosts option, would using a template block to write out an |
Thats a great idea. I think that may be a viable work around
|
May be my problem is some what similar so I hope I can post it here.
But I expected something like:
The bold line is the host of Docker container. My application was running Java, and a lot of library is rely on Hostname which cause error when try to resolve abcxyzaaa |
Looks like we've identified a workaround for the upstream issue. I'm going to mark this as a docs issue so that we can provide some official guidance in the networking and/or Connect docs for folks. |
The workaround requires another workaround in the case of wanting to use the special I'm wanting to use |
This is an issue for us, we need the app running inside the container to be able to resolve that randomly assigned container hostname. We're running containers in bridge mode and seems like nothing really working for that case. We tried to template the /etc/hosts with Any ideas or workarounds are welcome. |
Just to leave a comment as this breaks a lot of Java based application which rely on hostname -i resolution which can't be done. We are hitting this issue and mixing it up with template with hostname -I to resolve and fix it in /etc/hosts, but this is not a viable/generic solution, requires a lot of tooling in the image running to make it available also changing entrypoint to run this at start time, which is far from ideal. |
What is the workaround actually? Templating into /etc doesn't work in docker. |
The workaround is to create a new etc/hosts file at some arbitrary location like the nomad path '/local/etc/hosts' then doing a volume mount to overwrite '/etc/hosts' with '/local/etc/hosts'
|
containerd driver added that: Roblox/nomad-driver-containerd#69 |
#10766 will do that for the |
Hi, @tgross! I'm asking because I do have that and it's kinda not working. To be exact, I have: (To be clear, I'm just asking what are the requirements for it to work, not how exactly should I fix my thing ... although that would be also appreciated :) )
which results in
|
Hi @DejfCold!
The requirements from
Your jobspec there looks ok to me. The tests in |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Version
Nomad = 0.11.0
CNI Plugins = 0.8.4
Docker = 19.03.7, build 7141c199a2
Docker API = 1.40
OS = Ubuntu 18.04 (4.15.0-91-generic)
Problem
If you attempt to run a job that uses
extra_hosts
while using bridged networking, you will receive the following error.This is a major problem because it means any consul connect enabled job is not able to use custom hosts options.
I did find one related issue in docker where using
--net=host
and--add-hosts
was mutually exclusive before docker api version 1.12. I'm not sure which docker api version nomad is using, but 1.40 is the latestSteps to reproduce
Submit the following job
Workarounds
Possibly Related:
The text was updated successfully, but these errors were encountered: