Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm 1.7.2 init blocks on waiting for control plane to be ready #360

Closed
billmilligan opened this issue Jul 25, 2017 · 10 comments
Closed
Labels
kind/support Categorizes issue or PR as a support question.

Comments

@billmilligan
Copy link

/kind bug

What happened:

Never continues from "Created API client, waiting for the control plane to become ready"

What you expected to happen:

Init completes

How to reproduce it (as minimally and precisely as possible):

  1. Freshly created CoreOS 1409.7.0 using Vagrant
  2. Install kubeadm 1.7.2
  3. kubeadm init

Anything else we need to know?:

journalctl -u kubelet edited output:

client.go:72] Connecting to docker on unix:///var/run/docker.sock
client.go:92] Start docker client with request timeout=2m0s
cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
manager.go:143] cAdvisor running in container: "/system.slice/kubelet.service"
manager.go:151] unable to connect to Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
fs.go:117] Filesystem partitions: map[/dev/sda9:{mountpoint:/var/lib/docker major:8 minor:9 fsType:ext4 blockSize:0} /dev/mapper/usr:{mountpoint:/usr/share/ca-certificates major:254 minor:0 fsType:ext4 blockSize:0}]
manager.go:198] Machine: {NumCores:1 CpuFrequency:2533541 MemoryCapacity:1045028864 MachineID:833e0926ee21aed71ec075d726cbcfe0 SystemUUID:9E7AA0DB-00AB-4D2E-BE1B-FE7ADBA70BCF BootID:ab4436e5-d5fe-47c9-ac48-4fbf8c54316c Filesystems:[{Device:/dev/sda9 DeviceMajor:8 DeviceMinor:9 Capacity:16718393344 Type:vfs Inodes:4234880 HasInodes:true} {Device:/dev/mapper/usr DeviceMajor:254 DeviceMinor:0 Capacity:1031946240 Type:vfs Inodes:260096 HasInodes:true} {Device:overlay DeviceMajor:0 DeviceMinor:44 Capacity:16718393344 Type:vfs Inodes:4234880 HasInodes:true}] DiskMap:map[254:0:{Name:dm-0 Major:254 Minor:0 Size:1065345024 Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size:19818086400 Scheduler:cfq}] NetworkDevices:[{Name:eth0 MacAddress:08:00:27:f4:44:6b Speed:-1 Mtu:1500} {Name:eth1 MacAddress:08:00:27:1c:e7:6e Speed:-1 Mtu:1500}] Topology:[{Id:0 Memory:1045028864 Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:262144 Type:Unified Level:2}]}] Caches:[{Size:8388608 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
manager.go:204] Version: {KernelVersion:4.11.11-coreos ContainerOsVersion:Container Linux by CoreOS 1409.7.0 (Ladybug) DockerVersion:1.12.6 DockerAPIVersion:1.24 CadvisorVersion: CadvisorRevision:}
server.go:536] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
container_manager_linux.go:263] container manager verified user specified cgroup-root exists: /
container_manager_linux.go:268] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} ExperimentalQOSReserved:map[]}
kubelet.go:263] Adding manifest file: /etc/kubernetes/manifests
kubelet.go:273] Watching apiserver
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Drogue-kube-master-01&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Drogue-kube-master-01&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
kubelet_network.go:70] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
kubelet.go:508] Hairpin mode set to "hairpin-veth"
cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
docker_service.go:208] Docker cri networking managed by cni
docker_service.go:225] Setting cgroupDriver to cgroupfs
remote_runtime.go:42] Connecting to runtime service unix:///var/run/dockershim.sock
kuberuntime_manager.go:166] Container runtime docker initialized, version: 1.12.6, apiVersion: 1.24.0
server.go:943] Started kubelet v1.7.2+coreos.0
kubelet.go:1229] Image garbage collection failed once. Stats initialization may not have completed yet: unable to find data for container /
kubelet_node_status.go:247] Setting node annotation to enable volume controller attach/detach
server.go:128] Starting to listen on 0.0.0.0:10250
server.go:298] Adding debug handlers to kubelet server.
event.go:209] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt: connection refused' (may retry after sleeping)
kubelet.go:1729] Failed to check if disk space is available for the runtime: failed to get fs info for "runtime": unable to find data for container /
kubelet.go:1737] Failed to check if disk space is available on the root partition: failed to get fs info for "root": unable to find data for container /
fs_resource_analyzer.go:66] Starting FS ResourceAnalyzer
status_manager.go:140] Starting to sync pod status with apiserver
kubelet.go:1809] Starting kubelet main sync loop.
kubelet.go:1820] skipping pod synchronization - [container runtime is down PLEG is not healthy: pleg was last seen active 2562047h47m16.854775807s ago; threshold is 3m0s]
volume_manager.go:245] Starting Kubelet Volume Manager
cni.go:189] Unable to update cni config: No networks found in /etc/cni/net.d
kubelet.go:2136] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
factory.go:351] Registering Docker factory
manager.go:247] Registration of the rkt container factory failed: unable to communicate with Rkt api service: rkt: cannot tcp Dial rkt api service: dial tcp [::1]:15441: getsockopt: connection refused
factory.go:54] Registering systemd factory
factory.go:86] Registering Raw factory
manager.go:1121] Started watching for new ooms in manager
oomparser.go:185] oomparser using systemd
manager.go:288] Starting recovery of all containers
kubelet_node_status.go:247] Setting node annotation to enable volume controller attach/detach
manager.go:293] Recovery completed
eviction_manager.go:238] eviction manager: unexpected err: failed GetNode: node 'rogue-kube-master-01' not found
kubelet_node_status.go:82] Attempting to register node rogue-kube-master-01
kubelet_node_status.go:106] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
kubelet_node_status.go:247] Setting node annotation to enable volume controller attach/detach
kubelet_node_status.go:82] Attempting to register node rogue-kube-master-01
kubelet_node_status.go:106] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Drogue-kube-master-01&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Drogue-kube-master-01&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
kubelet_node_status.go:247] Setting node annotation to enable volume controller attach/detach
kubelet_node_status.go:82] Attempting to register node rogue-kube-master-01
kubelet_node_status.go:106] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelector=spec.nodeName%3Drogue-kube-master-01&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
kubelet_node_status.go:247] Setting node annotation to enable volume controller attach/detach
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:400: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:408: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=metadata.name%3Drogue-kube-master-01&resourceVersion=0: dial tcp 10.0.2.15:6443: getsockopt: connection refused
kubelet_node_status.go:82] Attempting to register node rogue-kube-master-01
kubelet_node_status.go:106] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.0.2.15:6443: getsockopt: connection refused
... repeats from here

Environment:

  • Kubernetes version (use kubectl version):
    Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"922a86cfcd65915a9b2f69f3f193b8907d741d9c", GitTreeState:"clean", BuildDate:"2017-07-21T08:23:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

  • Cloud provider or hardware configuration**:
    Metal: HP Proliant DL360 Gp with 48GB memory and 8 cores
    VM: Vagrant provisioned 1 CPU, 1 GB memory on VirtualBox, NAT address

  • OS (e.g. from /etc/os-release):
    NAME="Container Linux by CoreOS"
    ID=coreos
    VERSION=1409.7.0
    VERSION_ID=1409.7.0
    BUILD_ID=2017-07-19-0005
    PRETTY_NAME="Container Linux by CoreOS 1409.7.0 (Ladybug)"
    ANSI_COLOR="38;5;75"
    HOME_URL="https://coreos.com/"
    BUG_REPORT_URL="https://issues.coreos.com"
    COREOS_BOARD="amd64-usr"

  • Kernel (e.g. uname -a):
    4.11.11-coreos kubeadm join on slave node fails preflight checks #1 SMP Tue Jul 18 23:06:59 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5540 @ 2.53GHz GenuineIntel GNU/Linux

  • Install tools:
    Vagrant
    https://github.com/xakraz/kubeadm-installer (recent fork from @luxas https://github.com/luxas/kubeadm-installer)

  • Others:

/sig area/kubeadm

@xakraz
Copy link

xakraz commented Jul 26, 2017

I think this duplicates #353

@billmilligan
Copy link
Author

Not sure if it duplicates 353 but I'm pretty sure it does not duplicate 308, as you suggested offline. I'm using Docker 1.12.6, which I believe doesn't use cgroupfs by default.

@billmilligan
Copy link
Author

Six days and counting and I still have no answers, just "waiting for control plane to become ready". I have backed down to kubeadm 1.6.1, tried in both Ubuntu and CoreOS and same infuriating result. Anybody, please... I am way outside my area of expertise and just need to get some pods running so I can get back to development.

journalctl -r -u kubelet

Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.050418    1446 kubelet_node_status.go:101] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: di
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:07.048091    1446 kubelet_node_status.go:77] Attempting to register node rogue-kube-master-01
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.423922    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.401717    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.391884    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.205132    1446 eviction_manager.go:214] eviction manager: unexpected err: failed GetNode: node 'rogue-kube-master-01' not found
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:07.185036    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:07.184006    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.141038    1446 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:07.112209    1446 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.835662    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:06.773403    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:06.773312    1446 pod_container_deletor.go:77] Container "a3650c8412931a0abff190a4fce9e1440cc7c757f5879544947b05d598d94347" not found in pod's containers
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.421750    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.396589    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.385556    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:06.369196    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/a3650c8412931a0abff190a4
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.359002    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.358962    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.358939    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.358877    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:05.503205    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:05.418940    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:05.392252    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:05.382552    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:05.200085    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:05.132917    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:05.132403    1446 pod_container_deletor.go:77] Container "a2647c063874c926dae0275121c30c1e237dbe89b939e65146a796e6faf35275" not found in pod's containers
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:04.938119    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/a2647c063874c926dae02751
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.933594    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.933554    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.933533    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.933480    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:04.484742    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.415837    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.391754    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.379933    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.181954    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.179580    1446 kubelet.go:1535] Failed creating a mirror pod for "kube-controller-manager-rogue-kube-master-01_kube-system(e7e87d5242c09b93d44863d2a2cf
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:04.068369    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:04.068054    1446 pod_container_deletor.go:77] Container "7693aa2527a9bc264e9e4d94e81c4e7fb3f93454855d842f063f73c177477c52" not found in pod's containers
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:04.067456    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:03.671309    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/7693aa2527a9bc264e9e4d94
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.667259    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.667212    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.667186    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.667120    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.415118    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.390491    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.376664    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:03.340237    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.039397    1446 kubelet.go:1535] Failed creating a mirror pod for "kube-controller-manager-rogue-kube-master-01_kube-system(e7e87d5242c09b93d44863d2a2cf
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:03.038933    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:03.038328    1446 status_manager.go:465] Failed to update status for pod "_()": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/kube-control
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:03.023170    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:03.022814    1446 pod_container_deletor.go:77] Container "61d547a5bc591860ca613b0d2cdb99526078a96156d2321b5bfb1736b3bd380c" not found in pod's containers
Jul 27 03:17:03 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:03.006716    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:02.766361    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/61d547a5bc591860ca613b0d
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.760033    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.759983    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.759958    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.759890    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.414586    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.386633    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.360481    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:02.316339    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.167862    1446 event.go:208] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt:
Jul 27 03:17:02 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:02.014728    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:17:01 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:01.994488    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:01 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:01.993558    1446 pod_container_deletor.go:77] Container "cd749ab759898511d9c7e7c66eee507ce5def2fa3c061360c9a7dd162fc5e026" not found in pod's containers
Jul 27 03:17:01 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:01.964678    1446 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is
Jul 27 03:17:01 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:01.964490    1446 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 27 03:17:01 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:01.406295    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:17:01 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:01.385023    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:17:01 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:01.352713    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:00.726090    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/cd749ab759898511d9c7e7c6
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:00.718514    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:00.718469    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:00.718446    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:00.718391    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:00.405175    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:00.383401    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:00.352036    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:00.133027    1446 kubelet_node_status.go:101] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: di
Jul 27 03:17:00 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:00.132613    1446 kubelet_node_status.go:77] Attempting to register node rogue-kube-master-01
Jul 27 03:16:59 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:59.992314    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:59 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:59.403529    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:59 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:59.382852    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:59 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:59.350757    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:58.737987    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:58.434726    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:58.401957    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:58.392071    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:58.390308    1446 pod_container_deletor.go:77] Container "43043fa39140ad34825b91334369fa3291d3265400600567e023d9d9902e6f74" not found in pod's containers
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:58.370544    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:58.344390    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:58.104846    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/43043fa39140ad34825b9133
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:58.097543    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:58.097498    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:58.097477    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:58 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:58.097417    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:57.581557    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:57.400472    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:57.354691    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:57.321426    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:57.271625    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:57.237270    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:57.236478    1446 pod_container_deletor.go:77] Container "695b2ed5fd3d5d4537ef71e0ec490340222354c6a9ccd038d8c7f753290bdb3f" not found in pod's containers
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:57.028834    1446 eviction_manager.go:214] eviction manager: unexpected err: failed GetNode: node 'rogue-kube-master-01' not found
Jul 27 03:16:57 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:57.028292    1446 summary.go:70] Partial failure issuing GetContainerInfoV2: partial failures: ["/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:56.919076    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/695b2ed5fd3d5d4537ef71e0
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.912496    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.912446    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.912408    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.912333    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.694634    1446 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:56.694463    1446 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:56.545261    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.360708    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.354242    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.319530    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.241922    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.240423    1446 kubelet.go:1535] Failed creating a mirror pod for "kube-scheduler-rogue-kube-master-01_kube-system(92f24e628fe9b547ed9825f1df37f486)": P
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:56.207468    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:56.207151    1446 pod_container_deletor.go:77] Container "af08b2035c84912237b6cdd62d828c23d484dd04239b4b0b0c4507f9087f5b74" not found in pod's containers
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:56.205431    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:56.059692    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/af08b2035c84912237b6cdd6
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.054130    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.054047    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.054022    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:56 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:56.053887    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:55.506475    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:55.360258    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:55.353723    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:55.318352    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:55.204151    1446 kubelet.go:1535] Failed creating a mirror pod for "kube-scheduler-rogue-kube-master-01_kube-system(92f24e628fe9b547ed9825f1df37f486)": P
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:55.203546    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:55.202907    1446 status_manager.go:465] Failed to update status for pod "_()": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/kube-schedul
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:55.184273    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:55.178347    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:55 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:55.177842    1446 pod_container_deletor.go:77] Container "6a9a6f79c4f3647d0ecc1786a3b3705a083c98a7630f071e904fd2d936dff713" not found in pod's containers
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:54.742862    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/6a9a6f79c4f3647d0ecc1786
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.734138    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.734092    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.734063    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.734015    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:54.479074    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.358598    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.348668    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.312470    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.176687    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:54.145312    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:54.144874    1446 pod_container_deletor.go:77] Container "b15a246cb5fc900f59bf7c4dd0921720124c35778a4f1708d9683b9c07b2ed35" not found in pod's containers
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:54.113489    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/b15a246cb5fc900f59bf7c4d
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.101997    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.101956    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.101934    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.101877    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:53 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:53.344652    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:53 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:53.344460    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:53 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:53.300678    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.991969    1446 kubelet_node_status.go:101] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: di
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:52.991547    1446 kubelet_node_status.go:77] Attempting to register node rogue-kube-master-01
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:52.836061    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:52.491454    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.338562    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.338315    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.298487    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.189658    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.166759    1446 event.go:208] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt:
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:52.099647    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:52.099112    1446 pod_container_deletor.go:77] Container "6a734324beb7b96441e6488cc091234fe2bbb390cf475c433ce9bf1e81e05f36" not found in pod's containers
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:51.868313    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/6a734324beb7b96441e6488c
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.863100    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.863056    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.863034    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.862981    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.693151    1446 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:51.692929    1446 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:51.369769    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.334271    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.334002    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.292496    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.063959    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:51.012474    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:51.011996    1446 pod_container_deletor.go:77] Container "8957ecf910706e41bd78cd47af03a11d65b9bb76a187b23a222306fd55e83c78" not found in pod's containers
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:50.913197    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/8957ecf910706e41bd78cd47
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.906263    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.906212    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.906189    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.906125    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.332592    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.329826    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:50.321058    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.290296    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.018541    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:49.999115    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:49.998555    1446 pod_container_deletor.go:77] Container "6f5fe9544be73e01cefebfacf220210fc151ff6e7e9dec7d686fe375446b5030" not found in pod's containers
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:49.548298    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/6f5fe9544be73e01cefebfac
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:49.542054    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:49.541970    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:49.541946    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:49.541884    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:49.327218    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:49.327142    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:49.309148    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:49.288605    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:49.008060    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:49 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:49.007274    1446 status_manager.go:465] Failed to update status for pod "_()": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/etcd-rogue-k
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:48.983716    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:48.983246    1446 pod_container_deletor.go:77] Container "08fa8f7bfe9f9e45a48132b11809ff14a805b2d7fbc83136aad7987c74f47878" not found in pod's containers
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:48.952889    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:48.951809    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:48.950463    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:48.950172    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:48.947911    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/08fa8f7bfe9f9e45a48132b1
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:48.322872    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:48.322392    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:48 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:48.287554    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:47 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:47.312022    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:47 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:47.310388    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:47 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:47.286969    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: start it.
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.815195    1446 kuberuntime_manager.go:458] Container {Name:kube-apiserver Image:gcr.io/google_containers/kube-apiserver-amd64:v1.6.0 Command:[kube-apis
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.804214    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:46.802859    1446 eviction_manager.go:214] eviction manager: unexpected err: failed GetNode: node 'rogue-kube-master-01' not found
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.767360    1446 kuberuntime_manager.go:458] Container {Name:kube-scheduler Image:gcr.io/google_containers/kube-scheduler-amd64:v1.6.0 Command:[kube-sche
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.730750    1446 kuberuntime_manager.go:458] Container {Name:kube-controller-manager Image:gcr.io/google_containers/kube-controller-manager-amd64:v1.6.0
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:46.689173    1446 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:46.688623    1446 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.631065    1446 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/8d8257d361a666453c62a8be554771d8-k8s" (spec.
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.630489    1446 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/8d8257d361a666453c62a8be554771d8-certs" (spe
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:46.547239    1446 status_manager.go:465] Failed to update status for pod "_()": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/kube-apiserv
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.532212    1446 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/7075157cfd4524dbe0951e00a8e3129e-k8s" (spec.
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.531732    1446 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/7075157cfd4524dbe0951e00a8e3129e-etcd" (spec
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.530210    1446 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/host-path/7075157cfd4524dbe0951e00a8e3129e-certs" (spe
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.529751    1446 reconciler.go:231] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/8d8257d361a666453c62a8be554771d8
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:46.529303    1446 reconciler.go:231] VerifyControllerAttachedVolume operation started for volume "kubernetes.io/host-path/8d8257d361a666453c62a8be554771d8
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:46.511520    1446 kubelet.go:1535] Failed creating a mirror pod for "kube-apiserver-rogue-kube-master-01_kube-system(8d8257d361a666453c62a8be554771d8)": P
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:46.486079    1446 status_manager.go:465] Failed to update status for pod "_()": Get https://10.0.2.15:6443/api/v1/namespaces/kube-system/pods/etcd-rogue-k
Jul 27 03:16:46 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:46.485568    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:54.145312    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:54.144874    1446 pod_container_deletor.go:77] Container "b15a246cb5fc900f59bf7c4dd0921720124c35778a4f1708d9683b9c07b2ed35" not found in pod's containers
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:54.113489    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/b15a246cb5fc900f59bf7c4d
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.101997    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.101956    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.101934    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:54 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:54.101877    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:53 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:53.344652    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:53 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:53.344460    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:53 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:53.300678    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.991969    1446 kubelet_node_status.go:101] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: di
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:52.991547    1446 kubelet_node_status.go:77] Attempting to register node rogue-kube-master-01
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:52.836061    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:52.491454    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.338562    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.338315    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.298487    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.189658    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:52.166759    1446 event.go:208] Unable to write event: 'Post https://10.0.2.15:6443/api/v1/namespaces/default/events: dial tcp 10.0.2.15:6443: getsockopt:
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:52.099647    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:52 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:52.099112    1446 pod_container_deletor.go:77] Container "6a734324beb7b96441e6488cc091234fe2bbb390cf475c433ce9bf1e81e05f36" not found in pod's containers
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:51.868313    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/6a734324beb7b96441e6488c
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.863100    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.863056    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: r
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.863034    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rp
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.862981    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "e
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.693151    1446 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:51.692929    1446 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:51.369769    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=htt
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.334271    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?r
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.334002    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?f
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.292496    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSe
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:51.063959    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https:
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:16:51.012474    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:16:51 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:51.011996    1446 pod_container_deletor.go:77] Container "8957ecf910706e41bd78cd47af03a11d65b9bb76a187b23a222306fd55e83c78" not found in pod's containers
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:16:50.913197    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/8957ecf910706e41bd78cd47
Jul 27 03:16:50 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:16:50.906263    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a
lines 153-189
-- Logs begin at Thu 2017-07-27 03:09:09 UTC, end at Thu 2017-07-27 03:17:07 UTC. --
-- Logs begin at Thu 2017-07-27 03:09:09 UTC, end at Thu 2017-07-27 03:17:07 UTC. --
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.050418    1446 kubelet_node_status.go:101] Unable to register node "rogue-kube-master-01" with API server: Post https://10.0.2.15:6443/api/v1/nodes: dial tcp 10.
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:07.048091    1446 kubelet_node_status.go:77] Attempting to register node rogue-kube-master-01
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.423922    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelect
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.401717    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?resourceVer
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.391884    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=met
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.205132    1446 eviction_manager.go:214] eviction manager: unexpected err: failed GetNode: node 'rogue-kube-master-01' not found
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:07.185036    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=http://127.0.
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:07.184006    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:07.141038    1446 kubelet.go:2067] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready:
Jul 27 03:17:07 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:07.112209    1446 cni.go:157] Unable to update cni config: No networks found in /etc/cni/net.d
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.835662    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https://10.0.2.1
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:06.773403    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:06.773312    1446 pod_container_deletor.go:77] Container "a3650c8412931a0abff190a4fce9e1440cc7c757f5879544947b05d598d94347" not found in pod's containers
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.421750    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelect
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.396589    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?resourceVer
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.385556    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=met
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:06.369196    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/a3650c8412931a0abff190a4fce9e1440c
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.359002    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)")
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.358962    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rpc error:
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.358939    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rpc error: c
Jul 27 03:17:06 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:06.358877    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "etcd-rogue-
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:05.503205    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=http://127.0.
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:05.418940    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelect
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:05.392252    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?resourceVer
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:05.382552    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.0.2.15:6443/api/v1/nodes?fieldSelector=met
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:05.200085    1446 kubelet.go:1535] Failed creating a mirror pod for "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)": Post https://10.0.2.1
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:05.132917    1446 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
Jul 27 03:17:05 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:05.132403    1446 pod_container_deletor.go:77] Container "a2647c063874c926dae0275121c30c1e237dbe89b939e65146a796e6faf35275" not found in pod's containers
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: W0727 03:17:04.938119    1446 container.go:352] Failed to create summary reader for "/kubepods/besteffort/pod7075157cfd4524dbe0951e00a8e3129e/a2647c063874c926dae0275121c30c1e23
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.933594    1446 pod_workers.go:182] Error syncing pod 7075157cfd4524dbe0951e00a8e3129e ("etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)")
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.933554    1446 kuberuntime_manager.go:619] createPodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rpc error:
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.933533    1446 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "etcd-rogue-kube-master-01_kube-system(7075157cfd4524dbe0951e00a8e3129e)" failed: rpc error: c
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.933480    1446 remote_runtime.go:86] RunPodSandbox from runtime service failed: rpc error: code = 2 desc = failed to start sandbox container for pod "etcd-rogue-
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: I0727 03:17:04.484742    1446 kuberuntime_manager.go:458] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.0.17 Command:[etcd --listen-client-urls=http://127.0.
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.415837    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.0.2.15:6443/api/v1/pods?fieldSelect
Jul 27 03:17:04 rogue-kube-master-01 kubelet-wrapper[1446]: E0727 03:17:04.391754    1446 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:382: Failed to list *v1.Service: Get https://10.0.2.15:6443/api/v1/services?resourceVer
core@rogue-kube-master-01 ~ $

/help-wanted

@jjfiguep
Copy link

jjfiguep commented Aug 4, 2017

I have the same problem. Any help would be appreciated.

@pipejakob
Copy link
Contributor

We released 1.7.3 yesterday. Can you confirm that you're still seeing this issue with kubeadm 1.7.3?

@christopher-avila
Copy link

@pipejakob I can confirm that the issue persists on 1.7.3, but when using 1.6.0, this issue does not exists. Except that in 1.6.0 the issue is that kubeadm init get stucked on "First node has registered, but is not ready yet". So...

@pipejakob
Copy link
Contributor

@christopher-avila Thanks for confirming. So, based on @billmilligan's kubelet logs, it looks like it's failing to connect to the apiserver, which also explains why kubeadm is stuck waiting for the control plane to come up.

In order to diagnose what's going wrong, can you look at the Docker logs for kube-apiserver to see what's failing there? It's possible that it's just crashing over and over again with an error message, so you'll probably want to do something like:

$ docker ps -a | grep kube-apiserver

Then, you can use docker logs <container_id> to view the output from the container. If the problem is straight-forward, hopefully the apiserver is just instantly crashing with an error. If docker ps -a doesn't even show a kube-apiserver, then you'll probably need to restart kubelet to cancel the exponential backoff for pod restart so it will try to start the kube-apiserver again.

@warmchang
Copy link
Contributor

warmchang commented Aug 18, 2017

I have the same problem, and wait "54286.796454 seconds" for control plane componetes are healthy:

[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 54286.796454 seconds
[apiclient] Waiting for at least one node to register
[apiclient] First node has registered after 7.506955 seconds
[token] Using token: 3be765.3be8a227cbb80000
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns

Your Kubernetes master has initialized successfully!

@jamiehannaford
Copy link
Contributor

@billmilligan @christopher-avila Would you mind checking this occurs on newer kubeadm versions? If so, can you follow Jakob's comment here and post the output?

@jamiehannaford jamiehannaford added the kind/support Categorizes issue or PR as a support question. label Oct 17, 2017
@luxas
Copy link
Member

luxas commented Oct 27, 2017

Closing this for now, probably fixed with v1.8.
If this wasn't fixed with v1.8, please open a new, fresh issue.
Thank you!

@luxas luxas closed this as completed Oct 27, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/support Categorizes issue or PR as a support question.
Projects
None yet
Development

No branches or pull requests

8 participants