Docker driver should expose ports when it creates a container #212
Comments
ghost
mentioned this issue
|
@rvm2015 dynamic_ports was broken for a while. Can you make sure you have latest code? |
|
@achanda I'll pull from master, rebuild, and retry. I'm also wondering if it might be a docker problem, it should be at least detecting my open port but the ports field of docker ps is empty. |
|
I don't believe so. I've been using docker 1.8.2 and it has been doing port bindings fine. You can try |
|
@achanda nada, same result. Pulled master a few minutes ago, replaced my binary, wiped the data dir, and fired everything back up. Using the config and code from my original post I'm still not getting a port binding. Docker inspect claims there's a binding but I don't see one. docker inspect docker ps and netstat output inside the actual container |
|
@rvm2015 that's weird. Can you see if you can set a |
|
@achanda No change, the redis example works with the network mode set (it worked before as well using a 6379 port) however if I set network_mode = "net" for httpenv it doesn't forward through. docker inspect on httpenv Irritatingly I can hit the container from the docker host it's just not forwarding through for httpenv, only redis using docker-proxy which seems odd to me. The documentation for docker seems to imply docker-proxy is a legacy option and that newer versions like 1.8.2 should be using hairpin nat in the docker application: Is nomad assuming any kind of specific docker configuration that you are aware of? I'm using the curl | bash setup from docker's website for CentOS 7. Service definitions are as follows: edit: I tried disabling the userland proxy in docker and restarted the jobs, no change. I'm crashing for the night but I'll check this tomorrow. |
|
Hey @rvm2015 it looks like the port forwarding is working looking at docker inspect output. Further you said you can access the port on the host? It sounds like there is just a port forwarding issue on the host if you are trying to access it externally�? Let me know if I am reading this wrong�. |
|
@dadgar I can access the service at the private ip address assigned to the container (ie 172.17.whatever, my request is being routed through the docker0 bridge). I cannot access it by connecting to the forwarded port on the docker host. netstat -pan on the docker host shows the port is not actually opened by docker using my example, with the redis example it seems to work. This only happens if I start the task through nomad, if I start it manually with docker run using the -p option it works perfectly. The test command for that would be something like what I have below. You can see the port forward for the container show up in docker ps and netstat -pan should show port 10000 in LISTEN mode: The httpenv.nomad file I'm using is available below. The docker ps ports field will be blank. Running docker log httpenv will tell you what port it's bound to but doing a netstat -pan on the docker host will show that not open. Inexplicably though docker inspect will think it's been forwarded however: |
ghost
mentioned this issue
|
This worked for me when I developed the feature. Here are the demo app and job file I used. I developed on Ubuntu 15.04. I'm curious if the demo app works on Cent 7. There may be some OS-specific networking things we need to look for to make sure this works. I am not looking directly at this right now but I will include a Cent 7 build in the test matrix. |
|
The other bit of info that may be interesting is that go-dockerclient exposes two types of configuration which we use to start the container. One is configuration for the image (this typically includes The redis image has an If I did test with high ports where we create a 1:1 mapping instead of a high:local mapping, but I think that is a different scenario than the one described above that is causing problems. |
|
@rvm2015 Can you try adding |
|
This image wasn't built with a dockerfile, it was built manually off CentOS:7. I can recreate it using a Dockerfile this evening however and test. |
|
Sorry for the delay. I updated my code to take a PORT option and just set it to 8080 by default. I then created a docker file and had this port set to EXPOSE. I then set dynamic_ports=["8080"] in httpenv.nomad. Examples of all of these can be found in the repo above under tag alpha2. That works so somewhere in the port discovery process nomad is specifically looking for the EXPOSE option to do the binding. It doesn't work if EXPOSE isn't there even if the port is actually open (though docker inspect seems to think it's doing it). I also can't seem to get this to work with dynamic_ports["http"]. My understanding is that option sets a variable NOMAD_PORT_http to an ephemeral port on the docker host forwarding to the same port on the container, then passes this variable through to the container expecting your code to bind base off that port. The issue is you don't have this information at the time you're creating the docker file so there's no way to know what to set EXPOSE to (if I'm understanding all this properly). I'm working off of master as of about 30 minutes ago. Single node on CentOS 7, docker 1.8.2. |
|
Thanks for testing and following up!
Expose and port forwarding are two different configs in docker. In your example you used |
cbednarski
changed the title
|
Sure thing, I'll watch this for a branch or PR and test it with the original configuration. |
|
+1 since docker containers built with packer do not expose any ports: How Docker sees Packer built containers vs Dockerfile built containers In the docker logs I do not see the allocate_port task run without an exposed port By wrapping a packer buillt container in a Dockerfile and setting and EXPORT the new container was successfully recognized by Nomdad+Docker |
Ensure installSnapshot consume stream. fixes issue hashicorp#212
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
This is related to #201 and #203. Running nomad master as of today on CentOS 7 and docker 1.8.2 the dynamic_ports option seems to be inconsistently working.
To better understand how dynamic ports work I've created a small web service that just outputs the running environment.
Test code and nomad HCL definition can be found here (written in golang with net/http): https://bitbucket.org/howihumanteam/httpenv/src
There is a working docker image named howihuman/httpenv in docker hub that I've been testing with.
I'm finding that I cannot get nomad to run this image and bridge a port through to it. Using the standard nomad init example (with the 'redis' port swapped to '6379' as suggested in #203) works normally using some kind of docker-proxy process I'm not familiar with (edit: it's the docker userland proxy, I disabled it with no change).
I've tested the image by execing into it, running netstat -pan to verify the port binding, and curling the endpoint to verify I get os.Environ() spewed back at me. It is binding to the eth0 interface properly with the port specified by our environment variable, nomad just isn't telling docker to forward traffic to it. edit: I can also run the container manually specifying the environment variables and port options myself and it works, just not in nomad.
It doesn't matter if you bind to all interfaces or specific interfaces, use named dynamic_ports or use numbered dynamic_ports, bind to 8080 and pass NOMAD_PORT_8080 with dynamic_port = ["8080"], nothing. It just will not forward traffic.
I could be missing something really stupid but I doubt it at this point.
I see the following in my nomad logs (nothing really useful)
This is running on a 1 node nomad setup NOT using -dev mode. The service is started by systemd with this service definition:
The text was updated successfully, but these errors were encountered: