Possible mutation in statefulset sync

[mfojtik]

Addressed in #49285 using big hammer. We still need to investigate where the mutation happens exactly to avoid deep copying the object for no reason.

Initially this was discovered in this issue: openshift/origin#15324

Some logs:

I0719 02:12:14.503031   10463 stateful_set.go:452] Syncing StatefulSet default/mysql with 1 pods
CACHE *v1beta1.StatefulSet[4] ALTERED!
{"
A: kind":"StatefulSet","apiVersion":"apps/v1beta1","metadata":{"name":"mysql","namespace":"default","selfLink":"/apis/apps/v1beta1/namespaces/default/statefulsets/mysql","uid":"8281bdf3-6c27-11e7-a824-0242ac110002","resourceVersion":"1733","generation":1,"creationTimestamp":"2017-07-19T02:11:00Z","labels":{"app":"mysql"}},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"mysql"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"mysql"},"annotations":{"pod.alpha.kubernetes.io/init-containers":"[{\"name\":\"install\",\"image\":\"gcr.io/google_containers/galera-install:0.1\",\"args\":[\"--work-dir=/work-dir\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"workdir\",\"mountPath\":\"/work-dir\"},{\"name\":\"config\",\"mountPath\":\"/etc/mysql\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"Always\"},{\"name\":\"bootstrap\",\"image\":\"debian:jessie\",\"command\":[\"/work-dir/peer-finder\"],\"args\":[\"-on-start=\\\"/work-dir/on-start.sh\\\"\",\"-service=galera\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"workdir\",\"mountPath\":\"/work-dir\"},{\"name\":\"config\",\"mountPath\":\"/etc/mysql\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\"}]","pod.alpha.kubernetes.io/initialized":"true","pod.beta.kubernetes.io/init-containers":"[{\"name\":\"install\",\"image\":\"gcr.io/google_containers/galera-install:0.1\",\"args\":[\"--work-dir=/work-dir\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"workdir\",\"mountPath\":\"/work-dir\"},{\"name\":\"config\",\"mountPath\":\"/etc/mysql\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"Always\"},{\"name\":\"bootstrap\",\"image\":\"debian:jessie\",\"command\":[\"/work-dir/peer-finder\"],\"args\":[\"-on-start=\\\"/work-dir/on-start.sh\\\"\",\"-service=galera\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"workdir\",\"mountPath\":\"/work-dir\"},{\"name\":\"config\",\"mountPath\":\"/etc/mysql\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\"}]"}},"spec":{"volumes":[{"name":"config","emptyDir":{"sizeLimit":"0"}},{"name":"workdir","emptyDir":{"sizeLimit":"0"}}],"initContainers":[{"name":"install","image":"gcr.io/google_containers/galera-install:0.1","args":["--work-dir=/work-dir"],"resources":{},"volumeMounts":[{"name":"workdir","mountPath":"/work-dir"},{"name":"config","mountPath":"/etc/mysql"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"},{"name":"bootstrap","image":"debian:jessie","command":["/work-dir/peer-finder"],"args":["-on-start=\"/work-dir/on-start.sh\"","-service=galera"],"env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"workdir","mountPath":"/work-dir"},{"name":"config","mountPath":"/etc/mysql"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"containers":[{"name":"mysql","image":"erkules/galera:basic","args":["--defaults-file=/etc/mysql/my-galera.cnf","--user=root"],"ports":[{"name":"mysql","containerPort":3306,"protocol":"TCP"},{"name":"sst","containerPort":4444,"protocol":"TCP"},{"name":"replication","containerPort":4567,"protocol":"TCP"},{"name":"ist","containerPort":4568,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"datadir","mountPath":"/var/lib/"},{"name":"config","mountPath":"/etc/mysql"}],"readinessProbe":{"exec":{"command":["sh","-c","mysql -u root -e 'show databases;'"]},"initialDelaySeconds":15,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","securityContext":{},"schedulerName":"default-scheduler"}},"volumeClaimTemplates":[{"metadata":{"name":"datadir","creationTimestamp":null,"annotations":{"volume.alpha.kubernetes.io/storage-class":"anything"}},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}}},"status":{"phase":"Pending"}}],"serviceName":"galera","podManagementPolicy":"OrderedReady","updateStrategy":{"type":"OnDelete"},"revisionHistoryLimit":10},"status":{"observedGeneration":1,"replicas":1,"currentReplicas":1,"currentRevision":"mysql-174148028","updateRevision":"mysql-174148028"}}
B: metadata":{"name":"mysql","namespace":"default","selfLink":"/apis/apps/v1beta1/namespaces/default/statefulsets/mysql","uid":"8281bdf3-6c27-11e7-a824-0242ac110002","resourceVersion":"1733","generation":1,"creationTimestamp":"2017-07-19T02:11:00Z","labels":{"app":"mysql"}},"spec":{"replicas":3,"selector":{"matchLabels":{"app":"mysql"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"app":"mysql"},"annotations":{"pod.alpha.kubernetes.io/init-containers":"[{\"name\":\"install\",\"image\":\"gcr.io/google_containers/galera-install:0.1\",\"args\":[\"--work-dir=/work-dir\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"workdir\",\"mountPath\":\"/work-dir\"},{\"name\":\"config\",\"mountPath\":\"/etc/mysql\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"Always\"},{\"name\":\"bootstrap\",\"image\":\"debian:jessie\",\"command\":[\"/work-dir/peer-finder\"],\"args\":[\"-on-start=\\\"/work-dir/on-start.sh\\\"\",\"-service=galera\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"workdir\",\"mountPath\":\"/work-dir\"},{\"name\":\"config\",\"mountPath\":\"/etc/mysql\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\"}]","pod.alpha.kubernetes.io/initialized":"true","pod.beta.kubernetes.io/init-containers":"[{\"name\":\"install\",\"image\":\"gcr.io/google_containers/galera-install:0.1\",\"args\":[\"--work-dir=/work-dir\"],\"resources\":{},\"volumeMounts\":[{\"name\":\"workdir\",\"mountPath\":\"/work-dir\"},{\"name\":\"config\",\"mountPath\":\"/etc/mysql\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"Always\"},{\"name\":\"bootstrap\",\"image\":\"debian:jessie\",\"command\":[\"/work-dir/peer-finder\"],\"args\":[\"-on-start=\\\"/work-dir/on-start.sh\\\"\",\"-service=galera\"],\"env\":[{\"name\":\"POD_NAMESPACE\",\"valueFrom\":{\"fieldRef\":{\"apiVersion\":\"v1\",\"fieldPath\":\"metadata.namespace\"}}}],\"resources\":{},\"volumeMounts\":[{\"name\":\"workdir\",\"mountPath\":\"/work-dir\"},{\"name\":\"config\",\"mountPath\":\"/etc/mysql\"}],\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\"}]"}},"spec":{"volumes":[{"name":"config","emptyDir":{"sizeLimit":"0"}},{"name":"workdir","emptyDir":{"sizeLimit":"0"}}],"initContainers":[{"name":"install","image":"gcr.io/google_containers/galera-install:0.1","args":["--work-dir=/work-dir"],"resources":{},"volumeMounts":[{"name":"workdir","mountPath":"/work-dir"},{"name":"config","mountPath":"/etc/mysql"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"},{"name":"bootstrap","image":"debian:jessie","command":["/work-dir/peer-finder"],"args":["-on-start=\"/work-dir/on-start.sh\"","-service=galera"],"env":[{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"apiVersion":"v1","fieldPath":"metadata.namespace"}}}],"resources":{},"volumeMounts":[{"name":"workdir","mountPath":"/work-dir"},{"name":"config","mountPath":"/etc/mysql"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"containers":[{"name":"mysql","image":"erkules/galera:basic","args":["--defaults-file=/etc/mysql/my-galera.cnf","--user=root"],"ports":[{"name":"mysql","containerPort":3306,"protocol":"TCP"},{"name":"sst","containerPort":4444,"protocol":"TCP"},{"name":"replication","containerPort":4567,"protocol":"TCP"},{"name":"ist","containerPort":4568,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"datadir","mountPath":"/var/lib/"},{"name":"config","mountPath":"/etc/mysql"}],"readinessProbe":{"exec":{"command":["sh","-c","mysql -u root -e 'show databases;'"]},"initialDelaySeconds":15,"timeoutSeconds":5,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","securityContext":{},"schedulerName":"default-scheduler"}},"volumeClaimTemplates":[{"metadata":{"name":"datadir","creationTimestamp":null,"annotations":{"volume.alpha.kubernetes.io/storage-class":"anything"}},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}}},"status":{"phase":"Pending"}}],"serviceName":"galera","podManagementPolicy":"OrderedReady","updateStrategy":{"type":"OnDelete"},"revisionHistoryLimit":10},"status":{"observedGeneration":1,"replicas":1,"currentReplicas":1,"currentRevision":"mysql-174148028","updateRevision":"mysql-174148028"}}
panic: cache *v1beta1.StatefulSet modified
goroutine 30631 [running]:
github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache.(*defaultCacheMutationDetector).CompareObjects(0xc42cd75100)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:133 +0x38d
github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache.(*defaultCacheMutationDetector).Run.func1(0xc42cd75100, 0xc420010360)
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:84 +0x40
created by github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache.(*defaultCacheMutationDetector).Run
	/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/client-go/tools/cache/mutation_detector.go:92 +0x49

[mfojtik]

@Kargakis FYI

[mfojtik]

/sig apps

[0xmichalis]

@kubernetes/sig-apps-bugs

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close