Potential resource leak

[howardjohn]

What happened:
On July 10th, we started using kind in our prow cluster for Istio (see istio/test-infra#1455). We have about 6 tests or so that use kind, and maybe run 40 tests a day, so I'd estimate we are creating roughly 250 kind clusters a day across 60 nodes.

Ever since July 10th, we have run into problems on our cluster. Almost immediately, we ran out of inotify watches, which we fixed by just increasing the limit -- see #717.

Since then, the CPU of our nodes has slowly increased, like a memory leak but CPU. This can be attributed to the Kubelet process, which a profile indicates most of the time spent by cAdvisor.

Here is a graph of the minimum CPU of our nodes for a 4 hour interval. During a 4 hour time, it is almost certain we will have no test pods scheduled on the node, so this essentially shows the base overhead CPU of the nodes.

The big drops are times when we got rid of nodes. You can see the problems seem to start right around when we started using kind (note - it is possible it is a coincidence. My only evidence it is related to kind is the timing).

Within two weeks we see some nodes using 90% of CPU just idling.

What you expected to happen:
kind does not cause cluster-wide degradation.

How to reproduce it (as minimally and precisely as possible):
I am not sure, but we consistently run into this problem so I can run some diagnostics on the node.

Environment:

• kind version: (use kind version): v0.3, v0.4
• Kubernetes version: (use kubectl version): GKE versions 1.11, 1.12, 1.13 (tried upgrading twice to attempt to resolve this)
• OS (e.g. from /etc/os-release): COS

[aojea]

this issue remembers me this other one #421

[howardjohn]

I did run into that while investigating, but it seems the conclusion was

I can confirm that repeated cluster creation / deletion does NOT leak cgroups. The output of lscgroup | grep -c memory is the same after each creation / deletion cycle.

By the way, if it is relevant, we never do kind cluster delete, we just do kind create cluster then run some test and exit.

[aojea]

By the way, if it is relevant, we never do kind cluster delete, we just do kind create cluster then run some test and exit.

but ... then the old cluster containers keep running forever ... is it possible to add the kind cluster delete to your workflow once the tests finish and check if this solves the problem?

[howardjohn]

But we run it in a pod, once the pod is removed shouldn't everything be cleaned up? Or maybe because we have

        volumeMounts:
        - mountPath: /lib/modules
          name: modules
          readOnly: true
        - mountPath: /sys/fs/cgroup
          name: cgroup
      volumes:
      - hostPath:
          path: /lib/modules
          type: Directory
        name: modules
      - hostPath:
          path: /sys/fs/cgroup
          type: Directory
        name: cgroup

in our pod spec, so it never gets properly cleaned up.

I'll try adding the delete cluster to the end.

Does Kubernetes prow do this in their tests using kind? I am worried if the test crashes part way through we won't properly clean up

[aojea]

Does Kubernetes prow do this in their tests using kind? I am worried if the test crashes part way through we won't properly clean up

@BenTheElder is the authority on this, but the tests that are running in the CI execute the hack/ci/e2e.sh, and it does have a cleanup function that delete the cluster on EXIT

kind/hack/ci/e2e.sh

Lines 26 to 41 in 991e45e

	# our exit handler (trap)
	cleanup() {
	# always attempt to dump logs
	kind "export" logs "${ARTIFACTS}/logs" || true
	# KIND_IS_UP is true once we: kind create
	if [[ "${KIND_IS_UP:-}" = true ]]; then
	kind delete cluster || true
	fi
	# clean up e2e.test symlink
	rm -f _output/bin/e2e.test
	# remove our tempdir
	# NOTE: this needs to be last, or it will prevent kind delete
	if [[ -n "${TMP_DIR:-}" ]]; then
	rm -rf "${TMP_DIR}"
	fi
	}

[howardjohn]

Thank you! we will try that out

[howardjohn]

Well, it seems resolved - hard to be 100% sure since its only been 8 hrs, but seems good. I feel pretty dumb trying to figure this out for a month or so when it was such a simple fix -- thanks for the help!!

[BenTheElder]

The docker in docker runner / wrapper script we use in test-infra / prow.k8s.io also terminates all containers in an exit handler, amongst other things, redundantly to the cluster deletion we do in the kind specific scripts.