-
Notifications
You must be signed in to change notification settings - Fork 667
set up app container networking before containers start #47
Comments
My current thinking is that docker should add a Then
would become
Bonus point: no more wrapping of 'docker run'! |
This PR moby/moby#7436 allow user to set ip and netmask before containers start, but still under review. |
That doesn't help. Firstly, it either sets an IP, or specifies a range (via CIDR notation) from which it then picks an IP. So this is not at all what weave does/needs, which is to set both the IP and also the netmask. Secondly, it determines IPs for the docker0 bridge, whereas weave doesn't touch docker0 (so it can play nicely with containers not connected to weave and thus doesn't become an all-or-nothing choice) and operates via its own bridge. |
@jpetazzo points out that we can pretty close to that with the existing --net=container:CONTAINERID option, namely by launching a placeholder container, configure the weave networking in that, and then starting the application container with --net=container:. The main problem with taking this approach is that we end up with an extra container per application container. That container will show up in 'docker ps', which may confuse users. More importantly, it won't get removed when the corresponding application container stops. Furthermore, any network related 'docker run' options, such as exposing ports, setting the hostname, configuring dns, must be supplied when starting the placeholder container; they won't work (and in some case even cause errors) when supplied to the application container. So 'weave run' would have to parse all the docker args, figure out which ones are networking related, and pass them to the placeholder container start. yuck. Plus it means we are still wrapping 'docker run'. |
@rade You are right. |
+1 to this because it opens the possibility of running short lived jobs that actively make network connections right off the bat (like database backups) |
Following up our conversation with @rade:
I think it's both good and bad. Right, it may confuse users. But at the same time, it IMHO it's not a huge issue, especially if the name/image/... of the container can
Agreed, but it will allow to re-use it if needed. So it's both good and bad (again).
Ah, wait, some stuff won't work (exposing ports) however hostname and DNS should work, In the long run, I also hope that we can get something like e.g. If there are things that seem weird/impossible/etc don't hesitate to ping me. Thanks for making weave, anyway! |
It's not ok to leave behind placeholder containers every time an application container terminates.
Theoretically that is correct. Alas it doesn't work. I had actually tried this before making my claims ;).
What is |
Understood. So, hmm:
I totally agree! But IIRC, you can:
Regarding hostname and DNS: if it doesn't work, it's a bug in Docker, and we should totally fix it. I remember discussing "conflicting options" a while ago, and it's totally a bug, since the hostname I have mixed feelings about having Docker deal with the namespaces created by
If Docker starts to manage those namespaces as you suggest (giving them IP addresses etc.), I suggest that we rehash that a bit more, then we can draw the attention of the maintainers |
Ah! Yes, you are absolutely right. Have just tried that. Which leaves two problems then...
Let's assume that the second issue can be fixed. What can be done about the first?
afaict the namespaces created with
I would have thought that telling docker to work with an existing network namespace when starting a container, involves a sub-set of the steps it currently performs to configure container networking, namely everything except the namespace creation. What am I missing? |
As a workaround in the meantime consider offering a base image for people to use. |
Docker images aren't really flexible enough to make this work. We would want to arrive at a situation where derived images on startup execute some code that waits for the network, and then whatever code the user wanted. There is no way to compose process execution in Docker this way. You'd have to drag in a process manager, which, combined with the choice of underlying OS etc would make such a base image extremely opinionated. |
Or simply includes a command runWhenReady which executes it's parameters i.e.
More talking convenience than technical solution. On Thu, Sep 25, 2014 at 1:58 PM, rade notifications@github.com wrote:
|
Ah yes. that could work. Or at the very least we could have such a script available so folks can construct their own images with it. |
This might be totally dumb so ignore if it is - its more of a write up of my explorations in this area. I've had some success in hijacking the The #!/bin/sh
while ! grep -q ^1$ /sys/class/net/ethwe/carrier 2>/dev/null
do sleep 1
done
$@ I have a box with Then an ubuntu based container that has an entrypoint of weave run 10.255.0.2/8 binocarlos/ping -c 1 10.255.0.1 And then a version that hijacks the entrypoint: weave run 10.255.0.2/8 \
-v /tmp/waitfornetwork.sh:/bin/waitfornetwork.sh \
--entrypoint="/bin/waitfornetwork.sh" \
binocarlos/ping ping -c 1 10.255.0.1 And it works! However - there are many problems with this approach in terms of trying to generically apply it.
docker inspect --format '{{ range $index, $element := .Config.Entrypoint }}{{ if eq $index 0 }}{{ $element }}{{ end }}{{ end }}' binocarlos/ping
Because of the hackery above it feels better for the user to implicitly state their intention regarding this and not try and have weave do it generically. I have a couple of images to run that I need to wait for the network (db backups) but there is no way I can access or control the image and so this allows me to get around it. All of the above obviously goes away if we had |
@binocarlos Interesting. Not something I'd want to attempt to do in a shell script though. Also, |
@rade ahh yes good point about using a binary - it makes it totally agnostic to the underlying image of course. I will do a golang equivalent of the shell script and report back. |
Ok, but overriding |
The question is whether there is a way of getting the original entry point somehow like through and environment variable or something? |
Of course you can get the original entrypoint. Docker knows what it is, and it's returned by |
My bad, missed that bit. Looks doable. |
One challenge with the "rewrite entrypoint to a volume-mounted exe" approach is making it work when we don't have direct access to the docker host and instead run weave itself in a container (as per #312)? Where would the exe live s.t. we can volume-mount it into containers? The answer is Note that weavetools doesn't need to remain running for this to work, though the container needs to continue to exist. We'd need to take care of removing it with |
Hey guys sorry for delay have been out of action for a coupla months... @rade Great idea r.e. mounting the exe from a volume - makes this whole thing much more portable and allows the exe to be distributed as a container. It's mad how fiendishly hard parsing the Had not thought of using dockers @errordeveloper pausing the container would be great if it works! The challenge I suppose would be how to check |
Not sure what you mean. The whole point of
That's just racy; there's no way to guarantee we'd be able to pause the container before it attempts to access the network. |
@squaremo suggested an alternative technique for modifying the entrypoint: run a docker proxy. i.e. a process (in a container, naturally) that the That same proxy could also take care of #251. |
@rade yes the proxy would be a great way to bypass the cli arguments issue - I've been messing with a docker proxy in node - mdock - and it works a treat because as you say, the arguments are now in nice, already processed JSON format. When doing this, I had lots of fun and games realizing that So perhaps I can make a simple container based on |
@binocarlos Do you have your own fork of I'm working on my own single-host Docker proxy in Twisted, but would be curious to see the node version. |
@lukemarsden I've published Slight distraction: I had originally called this library @rade - here is a rough plan for the proxy I have in mind - I have no idea if a) it will work or b) how it might integrate into weave so feedback is welcome! We need a container that can be named 'weave-proxy' and:
We need another container that contains a The HTTP proxy port (2475) will forward all requests to the docker server (2375) by default. Any requests to
Any requests to
We point the docker client at our proxy: $ export DOCKER_HOST=tcp://127.0.0.1:2475 Now we can run long-lived servers: $ docker run -d --env WEAVE_CIDR=10.0.1.10/24 mystack/mysql And interactive jobs: $ docker run --env WEAVE_CIDR=10.0.1.11/24 ubuntu bash -c "ping 10.0.1.10" The interactive job will be blocked by There are a couple of problems I can see:
I'll give this a try this week and discover all the things that I've missed out :) |
@binocarlos Thanks for this. By the way, do you live in Bristol? (I do...) |
@lukemarsden No problem - yes I live in Bristol - its a great city! Perhaps I should come and say hello sometime - I met Richard Wall a few months ago @ a meetup in the engine shed and he was telling about what you guys were doing upstairs. |
I'm making good progress with a container that:
In the meantime - I've uploaded wait-for-weave which is the golang program that will wait for This binary is what will be mounted in a volume in a container called @rade if you could check this (golang being very new to me) I would be most grateful :) |
@binocarlos you may want to use |
@rade thanks! - that is far more civilized :) I've added an exit with a non-zero code if it decides there is no |
Agreed. |
Hi, @rade |
@tangzhankun The answer is Docker Network Plugins. Which obviously does a whole lot more than just wait for an entrypoint. |
hi, @rade |
Submit a PR :) This should be a straightforward extension. |
hi, @rade |
Is there somekind summary, what is correct solution to fix this. There is situations like DNS server in docker & weave networks, which needs to be up and running before other containers in reboot. |
@apassi, what are you looking for exactly? Weave Net's Docker proxy already waits for the NIC to be ready before your container gets started, and Weave Net's Docker plugin relies on Docker to do that, but leads to the same outcomes, so dependencies like NIC, DNS servers, etc. should be ready by the time your container's |
Basically i like to understand how weave and containers handle host's reboot situation. I havent able to find any "internals" document about weave. After i run weave launch, there seems to run new weave related containers, weavedb etc. but i havent been able to find documents for those.. |
@apassi, I understand what you meant now, thanks clarifying it. If you think a "Weave Net's internals" documentation could be useful, feel free to open a new GitHub issue (as this is unrelated to this one) and to list exactly what you would like to see there, since as developers, we are biaised and may think some details are obvious. Thanks! |
Currently 'weave run' sets up the app container's interface into the weave network after the container has been launched with 'docker run -d'. That means the network may not be available to the container process straight away. Depending on what the container is doing, that can be benign, annoying, or disastrous.
Containers can themselves ensure that the interface is available, by running s.t. like https://github.com/jpetazzo/pipework/blob/master/pipework#L30, i.e.
before starting the actual container process, but this of course requires containers to have been constructed with weave in mind, which is limiting.
There is no way around this issue w/o some changes to docker.
The text was updated successfully, but these errors were encountered: