Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

3.5: k8s API chunking tests flakes #13056

Closed
ptabor opened this issue May 28, 2021 · 3 comments · Fixed by #13060
Closed

3.5: k8s API chunking tests flakes #13056

ptabor opened this issue May 28, 2021 · 3 comments · Fixed by #13060
Milestone
etcd-v3.5

Comments

@ptabor
Copy link
Contributor

ptabor commented May 28, 2021

@serathius is reporting test flakes correlated with k8s switch to 3.5:

[sig-api-machinery] Servers with support for API chunking
...
[It] should return chunks of results for list calls
[STEP]: retrieving those results in paged fashion several times
May 27 00:41:11.678: INFO: Retrieved 16/16 results with rv 28160 and continue eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6MjgxNjAsInN0YXJ0IjoidGVtcGxhdGUtMDAxNVx1MDAwMCJ9
May 27 00:41:11.678: FAIL: Expected
    <int>: 17
to be ==
    <int>: 400

https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store.go#L640

[BeforeEach] [sig-api-machinery] Servers with support for API chunking
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:185
�[1mSTEP�[0m: Creating a kubernetes client
May 27 00:40:53.696: INFO: >>> kubeConfig: /workspace/.kube/config
�[1mSTEP�[0m: Building a namespace api object, basename chunking
�[1mSTEP�[0m: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in chunking-3010
�[1mSTEP�[0m: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [sig-api-machinery] Servers with support for API chunking
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/chunking.go:49
�[1mSTEP�[0m: creating a large number of resources
[It] should return chunks of results for list calls
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/chunking.go:77
�[1mSTEP�[0m: retrieving those results in paged fashion several times
May 27 00:41:11.678: INFO: Retrieved 16/16 results with rv 28160 and continue eyJ2IjoibWV0YS5rOHMuaW8vdjEiLCJydiI6MjgxNjAsInN0YXJ0IjoidGVtcGxhdGUtMDAxNVx1MDAwMCJ9
May 27 00:41:11.678: FAIL: Expected
    <int>: 17
to be ==
    <int>: 400
 
Full Stack Trace
k8s.io/kubernetes/test/e2e/apimachinery.glob..func3.2()
	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/apimachinery/chunking.go:102 +0xc6d
k8s.io/kubernetes/test/e2e.RunE2ETests(0xc000187080)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e.go:131 +0x36c
k8s.io/kubernetes/test/e2e.TestE2E(0xc000187080)
	_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/e2e_test.go:144 +0x2b
testing.tRunner(0xc000187080, 0x70fc490)
	/usr/local/go/src/testing/testing.go:1193 +0xef
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1238 +0x2b3
[AfterEach] [sig-api-machinery] Servers with support for API chunking
  /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:186
�[1mSTEP�[0m: Collecting events from namespace "chunking-3010".
�[1mSTEP�[0m: Found 0 events.
May 27 00:41:11.783: INFO: POD  NODE  PHASE  GRACE  CONDITIONS
May 27 00:41:11.783: INFO: 
May 27 00:41:11.837: INFO: 
Logging node info for node e2e-59163a61ce-a7d53-master
May 27 00:41:11.880: INFO: Node Info: &Node{ObjectMeta:{e2e-59163a61ce-a7d53-master    f46596c9-2dd2-48eb-bd67-fd35698a5726 25016 0 2021-05-27 00:29:26 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-1 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:e2e-59163a61ce-a7d53-master kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-1 topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kubelet Update v1 2021-05-27 00:29:26 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{},"f:unschedulable":{}}} } {kube-controller-manager Update v1 2021-05-27 00:29:27 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.0.0/24\"":{}},"f:taints":{}}} } {kube-controller-manager Update v1 2021-05-27 00:29:35 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kubelet Update v1 2021-05-27 00:29:57 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:config":{},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.0.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-016/us-west1-b/e2e-59163a61ce-a7d53-master,Unschedulable:true,Taints:[]Taint{Taint{Key:node-role.kubernetes.io/master,Value:,Effect:NoSchedule,TimeAdded:<nil>,},Taint{Key:node.kubernetes.io/unschedulable,Value:,Effect:NoSchedule,TimeAdded:<nil>,},},ConfigSource:nil,PodCIDRs:[10.64.0.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{20629221376 0} {<nil>} 20145724Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3862536192 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{18566299208 0} {<nil>} 18566299208 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{3600392192 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2021-05-27 00:29:35 +0000 UTC,LastTransitionTime:2021-05-27 00:29:35 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-05-27 00:39:59 +0000 UTC,LastTransitionTime:2021-05-27 00:29:26 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-05-27 00:39:59 +0000 UTC,LastTransitionTime:2021-05-27 00:29:26 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-05-27 00:39:59 +0000 UTC,LastTransitionTime:2021-05-27 00:29:26 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-05-27 00:39:59 +0000 UTC,LastTransitionTime:2021-05-27 00:29:33 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.40.0.2,},NodeAddress{Type:ExternalIP,Address:34.127.100.19,},NodeAddress{Type:InternalDNS,Address:e2e-59163a61ce-a7d53-master.c.k8s-infra-e2e-boskos-016.internal,},NodeAddress{Type:Hostname,Address:e2e-59163a61ce-a7d53-master.c.k8s-infra-e2e-boskos-016.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:3bce7452d8e2b1c23d9841705ca6ddef,SystemUUID:3bce7452-d8e2-b1c2-3d98-41705ca6ddef,BootID:9d47ca2c-7016-4421-8d90-9a36abc3b1eb,KernelVersion:5.4.0-1009-gcp,OSImage:Ubuntu 20.04 LTS,ContainerRuntimeVersion:containerd://1.5.2,KubeletVersion:v1.22.0-alpha.2.216+071c00975f417c,KubeProxyVersion:v1.22.0-alpha.2.216+071c00975f417c,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/kube-apiserver-amd64:v1.22.0-alpha.2.216_071c00975f417c],SizeBytes:127670739,},ContainerImage{Names:[k8s.gcr.io/kube-controller-manager-amd64:v1.22.0-alpha.2.216_071c00975f417c],SizeBytes:121023297,},ContainerImage{Names:[k8s.gcr.io/etcd@sha256:d7093a3257a25a03d5b118f0b16d1df73883566970e38b4d99dc640551be922b k8s.gcr.io/etcd:3.5.0-beta.3-1],SizeBytes:100128260,},ContainerImage{Names:[k8s.gcr.io/kube-scheduler-amd64:v1.22.0-alpha.2.216_071c00975f417c],SizeBytes:52255042,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64@sha256:be60ef505fc80879eeb5d8bf3ad8bb1146b395afc2394584645e99431806c26c gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.12.0],SizeBytes:32705362,},ContainerImage{Names:[k8s.gcr.io/addon-manager/kube-addon-manager@sha256:1f3ad8eb14dfaa2fc6dd60811e7d3dd1631c01f47cf1febc24c34d538a5409b1 k8s.gcr.io/addon-manager/kube-addon-manager:v9.1.5],SizeBytes:30464270,},ContainerImage{Names:[k8s.gcr.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a k8s.gcr.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:&NodeConfigStatus{Assigned:nil,Active:nil,LastKnownGood:nil,Error:,},},}
May 27 00:41:11.880: INFO: 
Logging kubelet events for node e2e-59163a61ce-a7d53-master
May 27 00:41:11.937: INFO: 
Logging pods the kubelet thinks is on node e2e-59163a61ce-a7d53-master
May 27 00:41:11.999: INFO: kube-addon-manager-e2e-59163a61ce-a7d53-master started at 2021-05-27 00:28:52 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:11.999: INFO: 	Container kube-addon-manager ready: true, restart count 0
May 27 00:41:11.999: INFO: l7-lb-controller-e2e-59163a61ce-a7d53-master started at 2021-05-27 00:28:52 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:11.999: INFO: 	Container l7-lb-controller ready: true, restart count 3
May 27 00:41:11.999: INFO: metadata-proxy-v0.1-94kcx started at 2021-05-27 00:29:34 +0000 UTC (0+2 container statuses recorded)
May 27 00:41:11.999: INFO: 	Container metadata-proxy ready: true, restart count 0
May 27 00:41:11.999: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
May 27 00:41:11.999: INFO: etcd-server-e2e-59163a61ce-a7d53-master started at 2021-05-27 00:28:17 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:11.999: INFO: 	Container etcd-container ready: true, restart count 0
May 27 00:41:11.999: INFO: kube-apiserver-e2e-59163a61ce-a7d53-master started at 2021-05-27 00:28:17 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:11.999: INFO: 	Container kube-apiserver ready: true, restart count 0
May 27 00:41:11.999: INFO: kube-controller-manager-e2e-59163a61ce-a7d53-master started at 2021-05-27 00:28:17 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:11.999: INFO: 	Container kube-controller-manager ready: true, restart count 2
May 27 00:41:11.999: INFO: kube-scheduler-e2e-59163a61ce-a7d53-master started at 2021-05-27 00:28:17 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:11.999: INFO: 	Container kube-scheduler ready: true, restart count 0
May 27 00:41:11.999: INFO: etcd-server-events-e2e-59163a61ce-a7d53-master started at 2021-05-27 00:28:17 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:11.999: INFO: 	Container etcd-container ready: true, restart count 0
W0527 00:41:12.088495   83975 metrics_grabber.go:118] Did not receive an external client interface. Grabbing metrics from ClusterAutoscaler is disabled.
May 27 00:41:12.593: INFO: 
Latency metrics for node e2e-59163a61ce-a7d53-master
May 27 00:41:12.593: INFO: 
Logging node info for node e2e-59163a61ce-a7d53-minion-group-lwff
May 27 00:41:12.655: INFO: Node Info: &Node{ObjectMeta:{e2e-59163a61ce-a7d53-minion-group-lwff    4f5e4e2e-012a-4d82-b311-1c67efd298eb 27726 0 2021-05-27 00:29:32 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:e2e-59163a61ce-a7d53-minion-group-lwff kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.hostpath.csi/node:e2e-59163a61ce-a7d53-minion-group-lwff topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-5009":"csi-mock-csi-mock-volumes-5009"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kube-controller-manager Update v1 2021-05-27 00:29:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.3.0/24\"":{}}}} } {kubelet Update v1 2021-05-27 00:29:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {node-problem-detector Update v1 2021-05-27 00:29:58 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2021-05-27 00:39:57 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}}}} status} {kubelet Update v1 2021-05-27 00:40:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:config":{},"f:images":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.3.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-016/us-west1-b/e2e-59163a61ce-a7d53-minion-group-lwff,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.3.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{103880232960 0} {<nil>} 101445540Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7819649024 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{93492209510 0} {<nil>} 93492209510 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7557505024 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2021-05-27 00:37:48 +0000 UTC,LastTransitionTime:2021-05-27 00:29:57 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:48 +0000 UTC,LastTransitionTime:2021-05-27 00:29:57 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:48 +0000 UTC,LastTransitionTime:2021-05-27 00:29:57 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:48 +0000 UTC,LastTransitionTime:2021-05-27 00:29:57 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:True,LastHeartbeatTime:2021-05-27 00:37:48 +0000 UTC,LastTransitionTime:2021-05-27 00:37:49 +0000 UTC,Reason:AUFSUmountHung,Message:kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds.,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2021-05-27 00:37:48 +0000 UTC,LastTransitionTime:2021-05-27 00:29:57 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2021-05-27 00:37:48 +0000 UTC,LastTransitionTime:2021-05-27 00:29:57 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2021-05-27 00:29:46 +0000 UTC,LastTransitionTime:2021-05-27 00:29:46 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-05-27 00:40:57 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-05-27 00:40:57 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-05-27 00:40:57 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-05-27 00:40:57 +0000 UTC,LastTransitionTime:2021-05-27 00:29:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.40.0.4,},NodeAddress{Type:ExternalIP,Address:34.145.36.167,},NodeAddress{Type:InternalDNS,Address:e2e-59163a61ce-a7d53-minion-group-lwff.c.k8s-infra-e2e-boskos-016.internal,},NodeAddress{Type:Hostname,Address:e2e-59163a61ce-a7d53-minion-group-lwff.c.k8s-infra-e2e-boskos-016.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:2efc666de99f49f26cbbb807b22578ad,SystemUUID:2efc666d-e99f-49f2-6cbb-b807b22578ad,BootID:33e5af57-5df9-41ce-9f53-18a9378531b9,KernelVersion:5.4.0-1009-gcp,OSImage:Ubuntu 20.04 LTS,ContainerRuntimeVersion:containerd://1.5.2,KubeletVersion:v1.22.0-alpha.2.216+071c00975f417c,KubeProxyVersion:v1.22.0-alpha.2.216+071c00975f417c,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/sig-storage/nfs-provisioner@sha256:c1bedac8758029948afe060bf8f6ee63ea489b5e08d29745f44fab68ee0d46ca k8s.gcr.io/sig-storage/nfs-provisioner:v2.2.2],SizeBytes:138177747,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.0-alpha.2.216_071c00975f417c],SizeBytes:133015759,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/volume/gluster@sha256:660af738347dd94cdd8069647136c84f11d03fc6dde3af0e746b302d3dfd10ec k8s.gcr.io/e2e-test-images/volume/gluster:1.2],SizeBytes:115217001,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/volume/nfs@sha256:124a375b4f930627c65b2f84c0d0f09229a96bc527eec18ad0eeac150b96d1c2 k8s.gcr.io/e2e-test-images/volume/nfs:1.2],SizeBytes:95843946,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 k8s.gcr.io/e2e-test-images/agnhost:2.32],SizeBytes:50002177,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[k8s.gcr.io/node-problem-detector/node-problem-detector@sha256:0ce71ef6d759425d22b10e65b439749fe5d13377a188e2fc060b731cdb4e6901 k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7],SizeBytes:47786458,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[k8s.gcr.io/build-image/debian-iptables@sha256:ace0d3b734717529f8cc0ff49ef54a13d53ae459d26ffff28a643ae26a32725f k8s.gcr.io/build-image/debian-iptables:buster-v1.6.1],SizeBytes:40448495,},ContainerImage{Names:[k8s.gcr.io/metrics-server/metrics-server@sha256:f8643f007c8a604388eadbdac43d76b95b56ccd13f7447dd0934b594b9f7b363 k8s.gcr.io/metrics-server/metrics-server:v0.4.4],SizeBytes:25196005,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-external-health-monitor-controller@sha256:46346c621c4d7e0976ddfb1e8722f1eb0af4e32a1d9648228b04364d9d4830ec k8s.gcr.io/sig-storage/csi-external-health-monitor-controller:v0.3.0],SizeBytes:21111424,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:7a5ba58a44e0d749e0767e4e37315bcf6a61f33ce3185c1991848af4db0fb70a k8s.gcr.io/sig-storage/csi-resizer:v1.1.0],SizeBytes:20096832,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-external-health-monitor-agent@sha256:c20d4a4772599e68944452edfcecc944a1df28c19e94b942d526ca25a522ea02 k8s.gcr.io/sig-storage/csi-external-health-monitor-agent:v0.2.0],SizeBytes:19983462,},ContainerImage{Names:[k8s.gcr.io/sig-storage/snapshot-controller@sha256:00fcc441ea9f72899c25eed61d602272a2a58c5f0014332bdcb5ac24acef08e4 k8s.gcr.io/sig-storage/snapshot-controller:v4.0.0],SizeBytes:18952261,},ContainerImage{Names:[k8s.gcr.io/cluster-proportional-autoscaler-amd64@sha256:be8875e5584750b7a490244ee56a121a714aa3d124164a5090cd8b3570c5650f k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.8.1],SizeBytes:15208262,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[k8s.gcr.io/coredns/coredns@sha256:cc8fb77bc2a0541949d1d9320a641b82fd392b0d3d8145469ca4709ae769980e k8s.gcr.io/coredns/coredns:v1.8.0],SizeBytes:12945155,},ContainerImage{Names:[k8s.gcr.io/autoscaling/addon-resizer@sha256:85c57faa812598eac877f97ef5a3eef7f62897c5aed62d774b36937ca31eb285 k8s.gcr.io/autoscaling/addon-resizer:1.8.13],SizeBytes:9353392,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/networking/ingress-gce-404-server-with-metrics-amd64@sha256:7eb7b3cee4d33c10c49893ad3c386232b86d4067de5251294d4c620d6e072b93 k8s.gcr.io/networking/ingress-gce-404-server-with-metrics-amd64:v1.10.11],SizeBytes:6463068,},ContainerImage{Names:[k8s.gcr.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a k8s.gcr.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[],VolumesAttached:[]AttachedVolume{},Config:&NodeConfigStatus{Assigned:nil,Active:nil,LastKnownGood:nil,Error:,},},}
May 27 00:41:12.655: INFO: 
Logging kubelet events for node e2e-59163a61ce-a7d53-minion-group-lwff
May 27 00:41:12.707: INFO: 
Logging pods the kubelet thinks is on node e2e-59163a61ce-a7d53-minion-group-lwff
May 27 00:41:12.764: INFO: liveness-fc623b5a-03b0-4dea-bd29-75bb887557f5 started at 2021-05-27 00:38:57 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container agnhost-container ready: true, restart count 0
May 27 00:41:12.764: INFO: npd-v0.8.7-l7fdp started at 2021-05-27 00:29:46 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container node-problem-detector ready: true, restart count 0
May 27 00:41:12.764: INFO: pod-subpath-test-preprovisionedpv-9lrk started at 2021-05-27 00:41:07 +0000 UTC (2+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Init container init-volume-preprovisionedpv-9lrk ready: true, restart count 0
May 27 00:41:12.764: INFO: 	Init container test-init-subpath-preprovisionedpv-9lrk ready: false, restart count 0
May 27 00:41:12.764: INFO: 	Container test-container-subpath-preprovisionedpv-9lrk ready: false, restart count 0
May 27 00:41:12.764: INFO: l7-default-backend-79858d8f86-zx5ht started at 2021-05-27 00:29:46 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container default-http-backend ready: true, restart count 0
May 27 00:41:12.764: INFO: netserver-0 started at 2021-05-27 00:40:54 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container webserver ready: false, restart count 0
May 27 00:41:12.764: INFO: netserver-0 started at 2021-05-27 00:40:08 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:12.764: INFO: liveness-cc06bb76-da6a-4280-a27c-592beaa9eb71 started at 2021-05-27 00:37:43 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container agnhost-container ready: true, restart count 0
May 27 00:41:12.764: INFO: test-rolling-update-with-lb-5ff6986c95-gbs4h started at 2021-05-27 00:41:06 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container agnhost ready: true, restart count 0
May 27 00:41:12.764: INFO: csi-mockplugin-attacher-0 started at 2021-05-27 00:40:30 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container csi-attacher ready: true, restart count 0
May 27 00:41:12.764: INFO: kube-proxy-e2e-59163a61ce-a7d53-minion-group-lwff started at 2021-05-27 00:29:32 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container kube-proxy ready: true, restart count 0
May 27 00:41:12.764: INFO: volume-snapshot-controller-0 started at 2021-05-27 00:29:46 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container volume-snapshot-controller ready: true, restart count 0
May 27 00:41:12.764: INFO: hostexec-e2e-59163a61ce-a7d53-minion-group-lwff-q85vp started at 2021-05-27 00:40:45 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container agnhost-container ready: true, restart count 0
May 27 00:41:12.764: INFO: metadata-proxy-v0.1-jd6pv started at 2021-05-27 00:29:33 +0000 UTC (0+2 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container metadata-proxy ready: true, restart count 0
May 27 00:41:12.764: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
May 27 00:41:12.764: INFO: csi-mockplugin-0 started at 2021-05-27 00:40:30 +0000 UTC (0+3 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container csi-provisioner ready: true, restart count 0
May 27 00:41:12.764: INFO: 	Container driver-registrar ready: true, restart count 0
May 27 00:41:12.764: INFO: 	Container mock ready: true, restart count 0
May 27 00:41:12.764: INFO: csi-mockplugin-attacher-0 started at 2021-05-27 00:40:17 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container csi-attacher ready: false, restart count 0
May 27 00:41:12.764: INFO: pod-0 started at <nil> (0+0 container statuses recorded)
May 27 00:41:12.764: INFO: coredns-755cd654d4-vx9tm started at 2021-05-27 00:29:46 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container coredns ready: true, restart count 0
May 27 00:41:12.764: INFO: exec-volume-test-preprovisionedpv-rvh4 started at 2021-05-27 00:41:07 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container exec-container-preprovisionedpv-rvh4 ready: false, restart count 0
May 27 00:41:12.764: INFO: test-pod-af380385-f7d3-4ec6-b2e5-24fa01c0e39c started at 2021-05-27 00:41:08 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container agnhost-container ready: false, restart count 0
May 27 00:41:12.764: INFO: ss2-2 started at 2021-05-27 00:40:13 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:12.764: INFO: 	Container webserver ready: true, restart count 0
W0527 00:41:12.819961   83975 metrics_grabber.go:118] Did not receive an external client interface. Grabbing metrics from ClusterAutoscaler is disabled.
May 27 00:41:14.360: INFO: 
Latency metrics for node e2e-59163a61ce-a7d53-minion-group-lwff
May 27 00:41:14.360: INFO: 
Logging node info for node e2e-59163a61ce-a7d53-minion-group-t4dr
May 27 00:41:14.407: INFO: Node Info: &Node{ObjectMeta:{e2e-59163a61ce-a7d53-minion-group-t4dr    781c402d-6635-466c-aea4-0f01f6f27e8d 27970 0 2021-05-27 00:29:32 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:e2e-59163a61ce-a7d53-minion-group-t4dr kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.hostpath.csi/node:e2e-59163a61ce-a7d53-minion-group-t4dr topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kube-controller-manager Update v1 2021-05-27 00:29:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.2.0/24\"":{}}}} } {kubelet Update v1 2021-05-27 00:29:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {node-problem-detector Update v1 2021-05-27 00:29:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2021-05-27 00:41:04 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2021-05-27 00:41:07 +0000 UTC FieldsV1 {"f:metadata":{"f:labels":{"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:config":{},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.2.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-016/us-west1-b/e2e-59163a61ce-a7d53-minion-group-t4dr,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.2.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{103880232960 0} {<nil>} 101445540Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7819649024 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{93492209510 0} {<nil>} 93492209510 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7557505024 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:52 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:52 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:52 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:True,LastHeartbeatTime:2021-05-27 00:37:52 +0000 UTC,LastTransitionTime:2021-05-27 00:37:52 +0000 UTC,Reason:AUFSUmountHung,Message:kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds.,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2021-05-27 00:37:52 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2021-05-27 00:37:52 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2021-05-27 00:37:52 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2021-05-27 00:29:46 +0000 UTC,LastTransitionTime:2021-05-27 00:29:46 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-05-27 00:41:07 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-05-27 00:41:07 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-05-27 00:41:07 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-05-27 00:41:07 +0000 UTC,LastTransitionTime:2021-05-27 00:29:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.40.0.5,},NodeAddress{Type:ExternalIP,Address:34.127.44.199,},NodeAddress{Type:InternalDNS,Address:e2e-59163a61ce-a7d53-minion-group-t4dr.c.k8s-infra-e2e-boskos-016.internal,},NodeAddress{Type:Hostname,Address:e2e-59163a61ce-a7d53-minion-group-t4dr.c.k8s-infra-e2e-boskos-016.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:d99706cb244140f7f2bd6d05b119497e,SystemUUID:d99706cb-2441-40f7-f2bd-6d05b119497e,BootID:d37fb64a-ee73-4c07-b79d-24bb05a1c3ce,KernelVersion:5.4.0-1009-gcp,OSImage:Ubuntu 20.04 LTS,ContainerRuntimeVersion:containerd://1.5.2,KubeletVersion:v1.22.0-alpha.2.216+071c00975f417c,KubeProxyVersion:v1.22.0-alpha.2.216+071c00975f417c,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/sig-storage/nfs-provisioner@sha256:c1bedac8758029948afe060bf8f6ee63ea489b5e08d29745f44fab68ee0d46ca k8s.gcr.io/sig-storage/nfs-provisioner:v2.2.2],SizeBytes:138177747,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.0-alpha.2.216_071c00975f417c],SizeBytes:133015759,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/volume/gluster@sha256:660af738347dd94cdd8069647136c84f11d03fc6dde3af0e746b302d3dfd10ec k8s.gcr.io/e2e-test-images/volume/gluster:1.2],SizeBytes:115217001,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/volume/nfs@sha256:124a375b4f930627c65b2f84c0d0f09229a96bc527eec18ad0eeac150b96d1c2 k8s.gcr.io/e2e-test-images/volume/nfs:1.2],SizeBytes:95843946,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 k8s.gcr.io/e2e-test-images/agnhost:2.32],SizeBytes:50002177,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nautilus@sha256:1f36a24cfb5e0c3f725d7565a867c2384282fcbeccc77b07b423c9da95763a9a k8s.gcr.io/e2e-test-images/nautilus:1.4],SizeBytes:49230179,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[k8s.gcr.io/node-problem-detector/node-problem-detector@sha256:0ce71ef6d759425d22b10e65b439749fe5d13377a188e2fc060b731cdb4e6901 k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7],SizeBytes:47786458,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-external-health-monitor-controller@sha256:46346c621c4d7e0976ddfb1e8722f1eb0af4e32a1d9648228b04364d9d4830ec k8s.gcr.io/sig-storage/csi-external-health-monitor-controller:v0.3.0],SizeBytes:21111424,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-external-health-monitor-agent@sha256:c20d4a4772599e68944452edfcecc944a1df28c19e94b942d526ca25a522ea02 k8s.gcr.io/sig-storage/csi-external-health-monitor-agent:v0.2.0],SizeBytes:19983462,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[k8s.gcr.io/coredns/coredns@sha256:cc8fb77bc2a0541949d1d9320a641b82fd392b0d3d8145469ca4709ae769980e k8s.gcr.io/coredns/coredns:v1.8.0],SizeBytes:12945155,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a k8s.gcr.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nonewprivs@sha256:8ac1264691820febacf3aea5d152cbde6d10685731ec14966a9401c6f47a68ac k8s.gcr.io/e2e-test-images/nonewprivs:1.3],SizeBytes:3263463,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/gce-pd/e2e-59163a61ce-a7d53-d-pvc-5f2e324f-a84a-4ce3-b02c-78185654edf5],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/gce-pd/e2e-59163a61ce-a7d53-d-pvc-5f2e324f-a84a-4ce3-b02c-78185654edf5,DevicePath:/dev/disk/by-id/google-e2e-59163a61ce-a7d53-d-pvc-5f2e324f-a84a-4ce3-b02c-78185654edf5,},},Config:&NodeConfigStatus{Assigned:nil,Active:nil,LastKnownGood:nil,Error:,},},}
May 27 00:41:14.408: INFO: 
Logging kubelet events for node e2e-59163a61ce-a7d53-minion-group-t4dr
May 27 00:41:14.485: INFO: 
Logging pods the kubelet thinks is on node e2e-59163a61ce-a7d53-minion-group-t4dr
May 27 00:41:14.573: INFO: netserver-1 started at 2021-05-27 00:40:08 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:14.573: INFO: startup-ff19e8ba-a307-4652-b31b-c1076f4492dc started at 2021-05-27 00:40:13 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container busybox ready: false, restart count 0
May 27 00:41:14.573: INFO: hostexec-e2e-59163a61ce-a7d53-minion-group-t4dr-qgr6f started at 2021-05-27 00:40:43 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container agnhost-container ready: true, restart count 0
May 27 00:41:14.573: INFO: pod-configmaps-4ced0492-c2f6-4571-8465-daa52b4c8911 started at 2021-05-27 00:40:18 +0000 UTC (0+3 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container createcm-volume-test ready: true, restart count 0
May 27 00:41:14.573: INFO: 	Container delcm-volume-test ready: true, restart count 0
May 27 00:41:14.573: INFO: 	Container updcm-volume-test ready: true, restart count 0
May 27 00:41:14.573: INFO: concurrent-27034601--1-7xw9s started at 2021-05-27 00:41:00 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container c ready: false, restart count 0
May 27 00:41:14.573: INFO: npd-v0.8.7-cqtzr started at 2021-05-27 00:29:46 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container node-problem-detector ready: true, restart count 0
May 27 00:41:14.573: INFO: pod-subpath-test-preprovisionedpv-j96g started at 2021-05-27 00:41:06 +0000 UTC (0+2 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container test-container-subpath-preprovisionedpv-j96g ready: false, restart count 0
May 27 00:41:14.573: INFO: 	Container test-container-volume-preprovisionedpv-j96g ready: false, restart count 0
May 27 00:41:14.573: INFO: netserver-1 started at 2021-05-27 00:39:20 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:14.573: INFO: test-webserver-de910bd4-842f-4b9c-b71a-83aae355291c started at 2021-05-27 00:40:51 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container test-webserver ready: true, restart count 0
May 27 00:41:14.573: INFO: ss2-0 started at 2021-05-27 00:40:57 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:14.573: INFO: pod2 started at 2021-05-27 00:40:57 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container agnhost-container ready: true, restart count 0
May 27 00:41:14.573: INFO: execpodqkz4x started at 2021-05-27 00:40:48 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container agnhost-container ready: true, restart count 0
May 27 00:41:14.573: INFO: gluster-server started at 2021-05-27 00:40:58 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container gluster-server ready: true, restart count 0
May 27 00:41:14.573: INFO: test-rolling-update-with-lb-5ff6986c95-lvwq4 started at 2021-05-27 00:41:12 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container agnhost ready: true, restart count 0
May 27 00:41:14.573: INFO: metadata-proxy-v0.1-bx4hg started at 2021-05-27 00:29:33 +0000 UTC (0+2 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container metadata-proxy ready: true, restart count 0
May 27 00:41:14.573: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
May 27 00:41:14.573: INFO: netserver-1 started at 2021-05-27 00:40:54 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:14.573: INFO: busybox-272698a2-535f-4eac-a9ce-26bd0e6c86af started at 2021-05-27 00:38:48 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container busybox ready: true, restart count 0
May 27 00:41:14.573: INFO: all-pods-removed--1-r45pw started at 2021-05-27 00:40:51 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container c ready: true, restart count 0
May 27 00:41:14.573: INFO: gcepd-injector started at 2021-05-27 00:41:00 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container gcepd-injector ready: true, restart count 0
May 27 00:41:14.573: INFO: kube-proxy-e2e-59163a61ce-a7d53-minion-group-t4dr started at 2021-05-27 00:29:32 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container kube-proxy ready: true, restart count 0
May 27 00:41:14.573: INFO: coredns-755cd654d4-rq4sw started at 2021-05-27 00:29:54 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container coredns ready: true, restart count 0
May 27 00:41:14.573: INFO: all-pods-removed--1-mmq2m started at 2021-05-27 00:40:51 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:14.573: INFO: 	Container c ready: true, restart count 0
W0527 00:41:14.621672   83975 metrics_grabber.go:118] Did not receive an external client interface. Grabbing metrics from ClusterAutoscaler is disabled.
May 27 00:41:14.994: INFO: 
Latency metrics for node e2e-59163a61ce-a7d53-minion-group-t4dr
May 27 00:41:14.994: INFO: 
Logging node info for node e2e-59163a61ce-a7d53-minion-group-z0q4
May 27 00:41:15.037: INFO: Node Info: &Node{ObjectMeta:{e2e-59163a61ce-a7d53-minion-group-z0q4    6faa48f2-11ea-42c0-97e2-7ed426576055 28228 0 2021-05-27 00:29:32 +0000 UTC <nil> <nil> map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/instance-type:n1-standard-2 beta.kubernetes.io/os:linux cloud.google.com/metadata-proxy-ready:true failure-domain.beta.kubernetes.io/region:us-west1 failure-domain.beta.kubernetes.io/zone:us-west1-b kubernetes.io/arch:amd64 kubernetes.io/hostname:e2e-59163a61ce-a7d53-minion-group-z0q4 kubernetes.io/os:linux node.kubernetes.io/instance-type:n1-standard-2 topology.hostpath.csi/node:e2e-59163a61ce-a7d53-minion-group-z0q4 topology.kubernetes.io/region:us-west1 topology.kubernetes.io/zone:us-west1-b] map[csi.volume.kubernetes.io/nodeid:{"csi-mock-csi-mock-volumes-4597":"csi-mock-csi-mock-volumes-4597"} node.alpha.kubernetes.io/ttl:0 volumes.kubernetes.io/controller-managed-attach-detach:true] [] []  [{kube-controller-manager Update v1 2021-05-27 00:29:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:node.alpha.kubernetes.io/ttl":{}}},"f:spec":{"f:podCIDR":{},"f:podCIDRs":{".":{},"v:\"10.64.1.0/24\"":{}}}} } {kubelet Update v1 2021-05-27 00:29:32 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:volumes.kubernetes.io/controller-managed-attach-detach":{}},"f:labels":{".":{},"f:beta.kubernetes.io/arch":{},"f:beta.kubernetes.io/instance-type":{},"f:beta.kubernetes.io/os":{},"f:cloud.google.com/metadata-proxy-ready":{},"f:failure-domain.beta.kubernetes.io/region":{},"f:failure-domain.beta.kubernetes.io/zone":{},"f:kubernetes.io/arch":{},"f:kubernetes.io/hostname":{},"f:kubernetes.io/os":{},"f:node.kubernetes.io/instance-type":{},"f:topology.kubernetes.io/region":{},"f:topology.kubernetes.io/zone":{}}},"f:spec":{"f:providerID":{}}} } {node-problem-detector Update v1 2021-05-27 00:29:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"CorruptDockerOverlay2\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentContainerdRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentDockerRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentKubeletRestart\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"FrequentUnregisterNetDevice\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"KernelDeadlock\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}},"k:{\"type\":\"ReadonlyFilesystem\"}":{".":{},"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{},"f:type":{}}}}} status} {kube-controller-manager Update v1 2021-05-27 00:40:52 +0000 UTC FieldsV1 {"f:status":{"f:conditions":{"k:{\"type\":\"NetworkUnavailable\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:volumesAttached":{}}} status} {kubelet Update v1 2021-05-27 00:40:57 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"f:csi.volume.kubernetes.io/nodeid":{}},"f:labels":{"f:topology.hostpath.csi/node":{}}},"f:status":{"f:allocatable":{"f:ephemeral-storage":{}},"f:capacity":{"f:ephemeral-storage":{}},"f:conditions":{"k:{\"type\":\"DiskPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"MemoryPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"PIDPressure\"}":{"f:lastHeartbeatTime":{}},"k:{\"type\":\"Ready\"}":{"f:lastHeartbeatTime":{},"f:lastTransitionTime":{},"f:message":{},"f:reason":{},"f:status":{}}},"f:config":{},"f:images":{},"f:volumesInUse":{}}} status}]},Spec:NodeSpec{PodCIDR:10.64.1.0/24,DoNotUseExternalID:,ProviderID:gce://k8s-infra-e2e-boskos-016/us-west1-b/e2e-59163a61ce-a7d53-minion-group-z0q4,Unschedulable:false,Taints:[]Taint{},ConfigSource:nil,PodCIDRs:[10.64.1.0/24],},Status:NodeStatus{Capacity:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{103880232960 0} {<nil>} 101445540Ki BinarySI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7819649024 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Allocatable:ResourceList{attachable-volumes-gce-pd: {{127 0} {<nil>} 127 DecimalSI},cpu: {{2 0} {<nil>} 2 DecimalSI},ephemeral-storage: {{93492209510 0} {<nil>} 93492209510 DecimalSI},hugepages-1Gi: {{0 0} {<nil>} 0 DecimalSI},hugepages-2Mi: {{0 0} {<nil>} 0 DecimalSI},memory: {{7557505024 0} {<nil>}  BinarySI},pods: {{110 0} {<nil>} 110 DecimalSI},},Phase:,Conditions:[]NodeCondition{NodeCondition{Type:FrequentUnregisterNetDevice,Status:False,LastHeartbeatTime:2021-05-27 00:37:56 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoFrequentUnregisterNetDevice,Message:node is functioning properly,},NodeCondition{Type:FrequentKubeletRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:56 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoFrequentKubeletRestart,Message:kubelet is functioning properly,},NodeCondition{Type:FrequentDockerRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:56 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoFrequentDockerRestart,Message:docker is functioning properly,},NodeCondition{Type:FrequentContainerdRestart,Status:False,LastHeartbeatTime:2021-05-27 00:37:56 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoFrequentContainerdRestart,Message:containerd is functioning properly,},NodeCondition{Type:KernelDeadlock,Status:True,LastHeartbeatTime:2021-05-27 00:37:56 +0000 UTC,LastTransitionTime:2021-05-27 00:37:56 +0000 UTC,Reason:AUFSUmountHung,Message:kernel: INFO: task umount.aufs:21568 blocked for more than 120 seconds.,},NodeCondition{Type:ReadonlyFilesystem,Status:False,LastHeartbeatTime:2021-05-27 00:37:56 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:FilesystemIsNotReadOnly,Message:Filesystem is not read-only,},NodeCondition{Type:CorruptDockerOverlay2,Status:False,LastHeartbeatTime:2021-05-27 00:37:56 +0000 UTC,LastTransitionTime:2021-05-27 00:29:51 +0000 UTC,Reason:NoCorruptDockerOverlay2,Message:docker overlay2 is functioning properly,},NodeCondition{Type:NetworkUnavailable,Status:False,LastHeartbeatTime:2021-05-27 00:29:46 +0000 UTC,LastTransitionTime:2021-05-27 00:29:46 +0000 UTC,Reason:RouteCreated,Message:RouteController created a route,},NodeCondition{Type:MemoryPressure,Status:False,LastHeartbeatTime:2021-05-27 00:40:57 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasSufficientMemory,Message:kubelet has sufficient memory available,},NodeCondition{Type:DiskPressure,Status:False,LastHeartbeatTime:2021-05-27 00:40:57 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasNoDiskPressure,Message:kubelet has no disk pressure,},NodeCondition{Type:PIDPressure,Status:False,LastHeartbeatTime:2021-05-27 00:40:57 +0000 UTC,LastTransitionTime:2021-05-27 00:29:32 +0000 UTC,Reason:KubeletHasSufficientPID,Message:kubelet has sufficient PID available,},NodeCondition{Type:Ready,Status:True,LastHeartbeatTime:2021-05-27 00:40:57 +0000 UTC,LastTransitionTime:2021-05-27 00:29:42 +0000 UTC,Reason:KubeletReady,Message:kubelet is posting ready status. AppArmor enabled,},},Addresses:[]NodeAddress{NodeAddress{Type:InternalIP,Address:10.40.0.3,},NodeAddress{Type:ExternalIP,Address:34.83.155.135,},NodeAddress{Type:InternalDNS,Address:e2e-59163a61ce-a7d53-minion-group-z0q4.c.k8s-infra-e2e-boskos-016.internal,},NodeAddress{Type:Hostname,Address:e2e-59163a61ce-a7d53-minion-group-z0q4.c.k8s-infra-e2e-boskos-016.internal,},},DaemonEndpoints:NodeDaemonEndpoints{KubeletEndpoint:DaemonEndpoint{Port:10250,},},NodeInfo:NodeSystemInfo{MachineID:6e62d40c45db20bf82ef4ee5bc923d28,SystemUUID:6e62d40c-45db-20bf-82ef-4ee5bc923d28,BootID:85f8c3e3-d156-40d2-a1a0-c00a439f3188,KernelVersion:5.4.0-1009-gcp,OSImage:Ubuntu 20.04 LTS,ContainerRuntimeVersion:containerd://1.5.2,KubeletVersion:v1.22.0-alpha.2.216+071c00975f417c,KubeProxyVersion:v1.22.0-alpha.2.216+071c00975f417c,OperatingSystem:linux,Architecture:amd64,},Images:[]ContainerImage{ContainerImage{Names:[k8s.gcr.io/sig-storage/nfs-provisioner@sha256:c1bedac8758029948afe060bf8f6ee63ea489b5e08d29745f44fab68ee0d46ca k8s.gcr.io/sig-storage/nfs-provisioner:v2.2.2],SizeBytes:138177747,},ContainerImage{Names:[k8s.gcr.io/kube-proxy-amd64:v1.22.0-alpha.2.216_071c00975f417c],SizeBytes:133015759,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/volume/gluster@sha256:660af738347dd94cdd8069647136c84f11d03fc6dde3af0e746b302d3dfd10ec k8s.gcr.io/e2e-test-images/volume/gluster:1.2],SizeBytes:115217001,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/jessie-dnsutils@sha256:702a992280fb7c3303e84a5801acbb4c9c7fcf48cffe0e9c8be3f0c60f74cf89 k8s.gcr.io/e2e-test-images/jessie-dnsutils:1.4],SizeBytes:112029652,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/volume/nfs@sha256:124a375b4f930627c65b2f84c0d0f09229a96bc527eec18ad0eeac150b96d1c2 k8s.gcr.io/e2e-test-images/volume/nfs:1.2],SizeBytes:95843946,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/agnhost@sha256:758db666ac7028534dba72e7e9bb1e57bb81b8196f976f7a5cc351ef8b3529e1 k8s.gcr.io/e2e-test-images/agnhost:2.32],SizeBytes:50002177,},ContainerImage{Names:[gke.gcr.io/prometheus-to-sd@sha256:e739643c3939ba0b161425f45a1989eedfc4a3b166db9a7100863296b4c70510 gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1],SizeBytes:48742566,},ContainerImage{Names:[k8s.gcr.io/node-problem-detector/node-problem-detector@sha256:0ce71ef6d759425d22b10e65b439749fe5d13377a188e2fc060b731cdb4e6901 k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7],SizeBytes:47786458,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:716d2f68314c5c4ddd5ecdb45183fcb4ed8019015982c1321571f863989b70b0 k8s.gcr.io/e2e-test-images/httpd:2.4.39-1],SizeBytes:41902332,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/httpd@sha256:b913fa234cc3473cfe16e937d106b455a7609f927f59031c81aca791e2689b50 k8s.gcr.io/e2e-test-images/httpd:2.4.38-1],SizeBytes:40765006,},ContainerImage{Names:[k8s.gcr.io/metrics-server/metrics-server@sha256:f8643f007c8a604388eadbdac43d76b95b56ccd13f7447dd0934b594b9f7b363 k8s.gcr.io/metrics-server/metrics-server:v0.4.4],SizeBytes:25196005,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:4e74c0492bceddc598de1c90cc5bc14dcda94cb49fa9c5bad9d117c4834b5e08 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.1],SizeBytes:22631062,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-provisioner@sha256:c8e03f60afa90a28e4bb6ec9a8d0fc36d89de4b7475cf2d613afa793ec969fe0 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.0],SizeBytes:22629806,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-resizer@sha256:36c31f7e1f433c9634d24f876353e8646246d81a03c4e351202c2644daff1620 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0],SizeBytes:21584611,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:c5be65d6679efabb969d9b019300d187437ae876f992c40911fd2892bbef3b36 k8s.gcr.io/sig-storage/csi-attacher:v3.2.0],SizeBytes:21367429,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-attacher@sha256:60ab9b3e6a030d3038c87c0d6bca2930f58d1d72823e6a4af09767dc83b696a2 k8s.gcr.io/sig-storage/csi-attacher:v3.2.1],SizeBytes:21366448,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-snapshotter@sha256:ed98431376c9e944e19a465fe8ea944806714dd95416a0821096c78d66b579bd k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1],SizeBytes:21331336,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-external-health-monitor-controller@sha256:46346c621c4d7e0976ddfb1e8722f1eb0af4e32a1d9648228b04364d9d4830ec k8s.gcr.io/sig-storage/csi-external-health-monitor-controller:v0.3.0],SizeBytes:21111424,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-external-health-monitor-agent@sha256:c20d4a4772599e68944452edfcecc944a1df28c19e94b942d526ca25a522ea02 k8s.gcr.io/sig-storage/csi-external-health-monitor-agent:v0.2.0],SizeBytes:19983462,},ContainerImage{Names:[k8s.gcr.io/cluster-proportional-autoscaler-amd64@sha256:be8875e5584750b7a490244ee56a121a714aa3d124164a5090cd8b3570c5650f k8s.gcr.io/cluster-proportional-autoscaler-amd64:1.8.1],SizeBytes:15208262,},ContainerImage{Names:[k8s.gcr.io/sig-storage/hostpathplugin@sha256:232fe80174d60d520d36043103853a1d7ab4b7f3782cf43e45034f04ccda58ce k8s.gcr.io/sig-storage/hostpathplugin:v1.7.1],SizeBytes:14930811,},ContainerImage{Names:[k8s.gcr.io/coredns/coredns@sha256:cc8fb77bc2a0541949d1d9320a641b82fd392b0d3d8145469ca4709ae769980e k8s.gcr.io/coredns/coredns:v1.8.0],SizeBytes:12945155,},ContainerImage{Names:[k8s.gcr.io/autoscaling/addon-resizer@sha256:85c57faa812598eac877f97ef5a3eef7f62897c5aed62d774b36937ca31eb285 k8s.gcr.io/autoscaling/addon-resizer:1.8.13],SizeBytes:9353392,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:a61d309da54641db41fb8f35718f744e9f730d4d0384f8c4b186ddc9f06cbd5f k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.1.0],SizeBytes:9068367,},ContainerImage{Names:[k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:2dee3fe5fe861bb66c3a4ac51114f3447a4cd35870e0f2e2b558c7a400d89589 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0],SizeBytes:8561694,},ContainerImage{Names:[k8s.gcr.io/sig-storage/mock-driver@sha256:a7b517f9e0f42ffade802eef9cefa271372386b85f55b702b493241e58459793 k8s.gcr.io/sig-storage/mock-driver:v4.1.0],SizeBytes:8223849,},ContainerImage{Names:[k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d k8s.gcr.io/sig-storage/livenessprobe:v2.3.0],SizeBytes:7933739,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/nginx@sha256:503b7abb89e57383eba61cc8a9cb0b495ea575c516108f7d972a6ff6e1ab3c9b k8s.gcr.io/e2e-test-images/nginx:1.14-1],SizeBytes:6979365,},ContainerImage{Names:[k8s.gcr.io/metadata-proxy@sha256:e914645f22e946bce5165737e1b244e0a296ad1f0f81a9531adc57af2780978a k8s.gcr.io/metadata-proxy:v0.1.12],SizeBytes:5301657,},ContainerImage{Names:[gcr.io/authenticated-image-pulling/alpine@sha256:7ff177862cb50c602bfe81f805969412e619c054a2bbead977d0c276988aa4a0 gcr.io/authenticated-image-pulling/alpine:3.7],SizeBytes:2110879,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/metadata-concealment@sha256:5cde3c696f251edad324efe91e858fdf47628e5a3c98f0af18edc765a33e1f12 k8s.gcr.io/e2e-test-images/metadata-concealment:1.6],SizeBytes:2097785,},ContainerImage{Names:[k8s.gcr.io/e2e-test-images/busybox@sha256:39e1e963e5310e9c313bad51523be012ede7b35bb9316517d19089a010356592 k8s.gcr.io/e2e-test-images/busybox:1.29-1],SizeBytes:732746,},ContainerImage{Names:[k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07 k8s.gcr.io/pause:3.5],SizeBytes:301416,},},VolumesInUse:[kubernetes.io/csi/csi-mock-csi-mock-volumes-4597^4],VolumesAttached:[]AttachedVolume{AttachedVolume{Name:kubernetes.io/csi/csi-mock-csi-mock-volumes-4597^4,DevicePath:,},AttachedVolume{Name:kubernetes.io/gce-pd/e2e-59163a61ce-a7d53-e53caaa1-fd38-4be9-9459-67041f639f7b,DevicePath:/dev/disk/by-id/google-e2e-59163a61ce-a7d53-e53caaa1-fd38-4be9-9459-67041f639f7b,},},Config:&NodeConfigStatus{Assigned:nil,Active:nil,LastKnownGood:nil,Error:,},},}
May 27 00:41:15.037: INFO: 
Logging kubelet events for node e2e-59163a61ce-a7d53-minion-group-z0q4
May 27 00:41:15.084: INFO: 
Logging pods the kubelet thinks is on node e2e-59163a61ce-a7d53-minion-group-z0q4
May 27 00:41:15.142: INFO: test-container-pod started at 2021-05-27 00:40:36 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:15.142: INFO: forbid-27034601--1-cgz7l started at 2021-05-27 00:41:00 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container c ready: false, restart count 0
May 27 00:41:15.142: INFO: netserver-2 started at 2021-05-27 00:40:54 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:15.142: INFO: gluster-server started at 2021-05-27 00:40:55 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container gluster-server ready: true, restart count 0
May 27 00:41:15.142: INFO: npd-v0.8.7-xw2mv started at 2021-05-27 00:29:46 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container node-problem-detector ready: true, restart count 0
May 27 00:41:15.142: INFO: ss2-1 started at 2021-05-27 00:40:36 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container webserver ready: false, restart count 0
May 27 00:41:15.142: INFO: netserver-2 started at 2021-05-27 00:40:08 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:15.142: INFO: test-container-pod started at 2021-05-27 00:39:45 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:15.142: INFO: startup-6bc23777-1f90-4946-8083-f4e757d3872c started at 2021-05-27 00:39:14 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container busybox ready: false, restart count 0
May 27 00:41:15.142: INFO: kube-proxy-e2e-59163a61ce-a7d53-minion-group-z0q4 started at 2021-05-27 00:29:32 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container kube-proxy ready: true, restart count 0
May 27 00:41:15.142: INFO: metrics-proxy started at 2021-05-27 00:30:33 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container nginx ready: true, restart count 0
May 27 00:41:15.142: INFO: csi-mockplugin-attacher-0 started at 2021-05-27 00:40:40 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container csi-attacher ready: true, restart count 0
May 27 00:41:15.142: INFO: hostexec-e2e-59163a61ce-a7d53-minion-group-z0q4-wqm8c started at 2021-05-27 00:41:04 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container agnhost-container ready: true, restart count 0
May 27 00:41:15.142: INFO: metadata-proxy-v0.1-jbr4l started at 2021-05-27 00:29:33 +0000 UTC (0+2 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container metadata-proxy ready: true, restart count 0
May 27 00:41:15.142: INFO: 	Container prometheus-to-sd-exporter ready: true, restart count 0
May 27 00:41:15.142: INFO: metrics-server-v0.4.4-6fccfbd69f-nvbcj started at 2021-05-27 00:30:00 +0000 UTC (0+2 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container metrics-server ready: true, restart count 0
May 27 00:41:15.142: INFO: 	Container metrics-server-nanny ready: true, restart count 0
May 27 00:41:15.142: INFO: pvc-tester-jp6bf started at 2021-05-27 00:41:09 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container write-pod ready: false, restart count 0
May 27 00:41:15.142: INFO: test-rolling-update-with-lb-5ff6986c95-htggx started at 2021-05-27 00:41:09 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container agnhost ready: true, restart count 0
May 27 00:41:15.142: INFO: projected-volume-0b81bfc2-fe14-48bc-8848-267bd0c92b2d started at 2021-05-27 00:41:13 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container projected-all-volume-test ready: true, restart count 0
May 27 00:41:15.142: INFO: netserver-2 started at 2021-05-27 00:39:20 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container webserver ready: true, restart count 0
May 27 00:41:15.142: INFO: csi-mockplugin-0 started at 2021-05-27 00:40:40 +0000 UTC (0+3 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container csi-provisioner ready: true, restart count 0
May 27 00:41:15.142: INFO: 	Container driver-registrar ready: true, restart count 0
May 27 00:41:15.142: INFO: 	Container mock ready: true, restart count 0
May 27 00:41:15.142: INFO: kube-dns-autoscaler-76478fcf46-dh5xf started at 2021-05-27 00:32:56 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container autoscaler ready: true, restart count 0
May 27 00:41:15.142: INFO: pvc-volume-tester-jtfm9 started at 2021-05-27 00:40:52 +0000 UTC (0+1 container statuses recorded)
May 27 00:41:15.142: INFO: 	Container volume-tester ready: false, restart count 0
W0527 00:41:15.190910   83975 metrics_grabber.go:118] Did not receive an external client interface. Grabbing metrics from ClusterAutoscaler is disabled.
May 27 00:41:15.752: INFO: 
Latency metrics for node e2e-59163a61ce-a7d53-minion-group-z0q4
May 27 00:41:15.752: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
�[1mSTEP�[0m: Destroying namespace "chunking-3010" for this suite.
@ptabor ptabor added this to the etcd-v3.5 milestone May 28, 2021
@ptabor ptabor mentioned this issue May 28, 2021
Closed
4 tasks
@serathius serathius mentioned this issue May 29, 2021
Closed
7 tasks
@serathius
Copy link
Member

Will take a look.

@serathius
Copy link
Member

serathius commented May 31, 2021 •
edited
Loading

Reproduced it on ETCD:

make
./bin/etcd --log-level=debug &
./bin/etcdctl put /registry/podtemplates/chunking-8554/template-c-0000 1
./bin/etcdctl put /registry/podtemplates/chunking-8554/template-c-0001 1
./bin/etcdctl put /registry/podtemplates/chunking-8554/template-c-0002 1
./bin/etcdctl get /registry/podtemplates/chunking-8554 /registry/podtemplates/chunking-85540 --limit 1

On etcd 3.5 setting limit causes response count to return 2

{"level":"debug","ts":"2021-05-31T16:24:54.602+0200","caller":"v3rpc/interceptor.go:182","msg":"request stats","start time":"2021-05-31T16:24:54.602+0200","time spent":"196.466µs","remote":"127.0.0.1:38310","response type":"/etcdserverpb.KV/Range","request count":0,"request size":79,"response count":2,"response size":97,"request content":"key:\"/registry/podtemplates/chunking-8554\" range_end:\"/registry/podtemplates/chunking-85540\" limit:1 "}

Same on Etcd 3.4 limit doesn't affect the response count and the result is 3 as expected.

{"level":"debug","ts":"2021-05-31T16:26:14.087+0200","caller":"v3rpc/interceptor.go:181","msg":"request stats","start time":"2021-05-31T16:26:14.086+0200","time spent":"214.469µs","remote":"127.0.0.1:38972","response type":"/etcdserverpb.KV/Range","request count":0,"request size":79,"response count":3,"response size":97,"request content":"key:\"/registry/podtemplates/chunking-8554\" range_end:\"/registry/podtemplates/chunking-85540\" limit:1 "}

@serathius
Copy link
Member

Kubernetes depends on this to provide number of remaining objects when using pagination. Will look into what changed between 3.4 and 3.5

This was referenced May 31, 2021
Merged
Merged
@wilsonwang371 wilsonwang371 mentioned this issue Jun 2, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
etcd-v3.5
Development

Successfully merging a pull request may close this issue.

etcdserver: Fix invalid count returned on Range with Limit serathius/etcd
2 participants
@serathius @ptabor