Replace docker commands with docker sdk

[iwilltry42]

Hey,

I started to replace calls to the docker executable with calls to the docker API using the Docker client SDK.

The only point where it wasn't really straight forward was the docker cp command, since the docker cli has some extensive checks there...

What do you think about this?

Benefits:

• more flexibility
• no dependency on docker (or at some point docker-compose) executables
• no parsing of command outputs to gather information

Drawbacks:

• more code
• a bit more code complexity

Why?

This will enable us to spawn sets of worker nodes (as requested in an issue), without having to rely on the docker-compose executable.

[iwilltry42]

Since I forgot to add all functionality (e.g. adding env vars), this PR will receive some updates.
But as the change is quite large you might want to enjoy it in smaller pieces :)

[zeerorg]

Got the below error when testing the PR

2019/04/17 10:20:40 Creating cluster [k3s_default]
2019/04/17 10:20:47 ERROR: failed to create cluster
ERROR: couldn't create container k3d-k3s_default-server
Error: No such image: docker.io/rancher/k3s:v0.4.0

[zeerorg]

Can you remove k3s:v0.4.0 from your docker images and try to do this again ?

[zeerorg]

When I stop k3s cluster, only the server stops but the worker nodes keep on running.

[zeerorg]

kubectl get nodes only shows the server node and not the worker nodes that were created.

[iwilltry42]

Thanks for the feedback @zeerorg, I will check this tomorrow. I didn't have the problem with workers not shutting down yet.
I will investigate those issues.
This PR is definitely not ready yet, I only opened it this early to give you more time to check out the switch from executing docker cli commands to using the docker API via client sdk

[zeerorg]

Not a problem, I just wanted to look a bit at this since it is a big change that's all. You are doing way too good work

[iwilltry42]

@zeerorg

Got the below error when testing the PR

2019/04/17 10:20:40 Creating cluster [k3s_default]
2019/04/17 10:20:47 ERROR: failed to create cluster
ERROR: couldn't create container k3d-k3s_default-server
Error: No such image: docker.io/rancher/k3s:v0.4.0

Fixed in commit 7d5964c
Problem was that the returned ReadCloser from docker.ImagePull was not read in non-verbose mode, which apparently makes docker to not pull the image at all. Also verbose mode couldn't be enabled there, because it was using c.Bool instead of c.GlobalBool

[iwilltry42]

When I stop k3s cluster, only the server stops but the worker nodes keep on running.

@zeerorg this happened when there was more than one cluster active at a time and using delete -a, right?
Because then the getClusters function got confused with label matching.
I fixed this in commit 92039ed 👍

[iwilltry42]

kubectl get nodes only shows the server node and not the worker nodes that were created.

@zeerorg the containers couldn't communicate with each other.
We now create a docker network for each cluster so that this will work.
Changes are in commit 6c072fb

[iwilltry42]

Lots of significant changes here, please test again :)

FYI @zeerorg & @ibuildthecloud

[iwilltry42]

get-kubeconfig now also uses the docker SDK, so I was able to remove run.go

[iwilltry42]

@zeerorg and @ibuildthecloud I think this should be on par with the old version now (+ some new features)

[zeerorg]

Almost all the things have been fixed. Really nice 👍
Somethings you can look at:

1. export KUBECONFIG... should come after workers are started
   

2. The --wait flag should also include waiting for workers to come up and respect the timeout value.

[iwilltry42]

Almost all the things have been fixed. Really nice +1
Somethings you can look at:

1. `export KUBECONFIG...` should come after workers are started
   ![image](https://user-images.githubusercontent.com/13547997/56580963-88505300-65f1-11e9-9ec7-35b6f9d02e83.png)

2. The `--wait` flag should also include waiting for workers to come up and respect the timeout value.

@zeerorg , thanks for the review.

1. fixed in 068172a ✔️
2. Hmm... I'm not sure if we should only consider it up if all the workers are up and running, since the API-Server will be up and kubectl will work already if they are not ready yet. Anyways, I'd put this out of scope for this PR, or what do you think? I'd rather phrase it as a new issue and starting point for a new PR 👍

[zeerorg]

Yeah, we can keep this out. We can include it in the next update. Another thing we could do is parallelize worker creation so that if someone creates a lot of workers they can be up and running more quickly.

[zeerorg]

LGTM

[iwilltry42]

Yeah, we can keep this out. We can include it in the next update. Another thing we could do is parallelize worker creation so that if someone creates a lot of workers they can be up and running more quickly.

Sure thing. Parallelizing the work is a super good idea 👍 I also already thought about using goroutines if we decided on including the workers in the timeout. Some more ideas, but I'm happy to first merge this PR :) Maybe I can get another approval today, but since we're already actively using it without any problems, I guess we're good to go.