using kind with gitlab kuberenetes executor fails #379
Comments
|
This is the complete content of the dumped logs: |
|
please pass |
|
The git / dockerd is not an issue. Kubelet not seeing a config file initially is normal, with kubeadm you let systemd crashloop kubelet until the config file is written. Which executor are you using? Can you run kind create cluster with --loglevel=debug? |
|
This is the debug output: There seems to be a problem in finding an IP |
We use the Kubernetes Executor (https://docs.gitlab.com/runner/executors/kubernetes.html#exposing-varrundockersock) with the docker.sock exposed from the node to the Kubernetes runner and then in turn to the Kubernetes Executor pod. |
|
ahh, I remember some previous issues with this. We need docs for it. see also #303 though that assumes docker-in-docker as opposed to mounting the host socket. |
|
Thanks for that information! As of now I see 3 possibilities for running kind in a gitlab-ci build stage:
Which option is the preferable one? Is the third option what was used in #303? |
|
I'm not quite familiar enough with gitlab to know which is preferable unfortunately, perhaps @munnerz is. I think 2.) mostly works but either requires scheduling your workload that talks to the cluster as a container in that dind, or fixing up the network + kubeconfig to properly point at the cluster since the network will be inside that dind. 1.) may have similar issues. 3.) sounds more like what we currently do on prow (Kubernetes CI). |
|
I successfully followed option 3. Thank you for your help! For reference: I created a Docker image based on dind, added kubectl, helm, kind, and a saved kind node image that is preloaded with some images (tiller, postgres, openfaas). After I added the two mounts mentioned in #303 the setup worked. A tricky part was that the docker daemon had to be started manually in the job script because gitlab-ci does not use the image's entrypoint. I.e. the script started with: (overlay2 is the storage driver that the underlying gitlab kubernetes runner is using.) == Addendum == In the meantime we use option 2, i.e. our Gitlab-Ci-Job has a dind service that is used to create a kind cluster. - Works nicely! Many thanks to the kind team! |
|
I run into a similar problem, I'm using dind as a service, the cluster is created but the master doesn't get ready, it's a problem with the CNI i think, tried on kind 0.2.1 and 0.2.0 gave as logs (only the tail) : At the end of docker-info.txt, I get When I tried with kind 0.0.1, the cluster is not created due to:
|
please enable more logs with |
|
Using kind 0.0.1, I get |
move the latest release please. |
|
I tried all the versions, please read the first part of my first comment (#379 (comment)), I said using 0.2.0 and 0.2.1 the cluster is created but it gets stuck. |
|
@machine424 I guess that the comment was related to paste the logs of the failure when running the last version 😄 |
|
yes, please post logs using |
|
Here we go: (I repeat, I don't know if the volumes specified in #303 are mounted to the runner/k8s executor) |
|
This time the cluster creation succeeded. Maybe the reason is that you now use kindest/node:v1.13.4 instead of v1.13.3. |
|
Actually, as I said, the cluster is created with kind 0.2.0 and 0.2.1 but the master doesn't get ready (CNI problem), with kind 0.0.1 the cluster is not created at all, please read my first comment: #379 (comment) |
|
@machine424 we'll need more output to look into that, perhaps |
|
I'm also having issues on Gitlab with dind (latest kind): test:
stage: test
image: docker:stable
variables:
DOCKER_HOST: tcp://docker:2375/
DOCKER_DRIVER: overlay2
services:
- docker:dind
before_script:
- cat /etc/resolv.conf
- cat /etc/hosts
- cat /etc/nsswitch.conf
- docker info
script:
- apk add bash curl openssl
- wget https://github.com/kubernetes-sigs/kind/releases/download/0.2.1/kind-linux-amd64
- chmod +x kind-linux-amd64
- mv kind-linux-amd64 /usr/local/bin/kind
- kind create cluster --name $CI_PIPELINE_ID --wait 180s --loglevel=debugAny ideas? |
|
If you are using the Kubernetes runner then you must address docker at localhost, i.e. Additionally, I think that you do not have to set the |
|
No, I use the Docker runner, but I wonder: maybe I don't even need kind with the Kubernetes runner, given I can access the Kubernetes cluster from the runner. |
|
That error could come from any number of things unfortunately, though we also haven't tested with remote docker yet or gitlab dind (we can't set that up on this repo due to how kubernetes manages github integrations). I'm not sure who has the dind runner working if any. kind does work in dind, but see: #303 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I try to create a kind cluster for integration test in a gitlab-ci job. Cluster creation fails:
In the dumped logs there are two entries that may give a clue on the reason.
In
docker.log:In
kubelet.log:The text was updated successfully, but these errors were encountered: