Yet another "... Container * is not running" as version #1879
Comments
|
this means the node crashed during the entrypoint, in the past due to e.g. running on an unsupported filesystem or other environmental oddities. normally we'd look at the container logs using |
|
FWIW I recommend docker desktop vs docker-machine. https://docs.docker.com/machine/overview/#why-should-i-use-it
#1288 is not actually a specific issue. If for any reason the node fails to finish the entrypoint / is terminated early then we'll see similar symptoms. We've fixed all known causes of the entrypoint failing or the container being terminated early by the host, this must be something new. I don't have windows enterprise or use docker machine so I won't be able to reproduce this. If you can capture the node container logs we can see if something failed there. If nothing was logged there, then it's likely the host / dockerd killing the container (perhaps you're out of resources on the host), which we can't do anything about. |
BenTheElder
added
the
triage/needs-information
|
We need to run these containers detached ( If we can't get logs from there, we won't really have much actionable. |
|
Thanks @BenTheElder for the quick reply! What I actually meant is: Related to #1288 (comment) ;)
Can I run the
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 52
Server Version: 19.03.5
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.154-boot2docker
Operating System: Boot2Docker 19.03.5 (TCL 10.1)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67GiB
Name: default
ID: 4B4L:7WVK:UP3R:ZPOS:ZATA:SUB5:J7JX:S7LJ:ITGK:Q3XG:JXNW:XLFC
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
[
{
"Id": "cec3b6664d735d495ceabf1052810508e955a226c27223f1a220ff1719c114eb",
"Created": "2020-10-01T18:57:03.506215881Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "created",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 128,
"Error": "cgroups: cannot find cgroup mount destination: unknown",
"StartedAt": "0001-01-01T00:00:00Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:37ddbc9063d2ae1a9587f79fd184bef370fd49bb853cf0521aef6ddd17399c99",
"ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/cec3b6664d735d495ceabf1052810508e955a226c27223f1a220ff1719c114eb/resolv.conf",
"HostnamePath": "/mnt/sda1/var/lib/docker/containers/cec3b6664d735d495ceabf1052810508e955a226c27223f1a220ff1719c114eb/hostname",
"HostsPath": "/mnt/sda1/var/lib/docker/containers/cec3b6664d735d495ceabf1052810508e955a226c27223f1a220ff1719c114eb/hosts",
"LogPath": "/mnt/sda1/var/lib/docker/containers/cec3b6664d735d495ceabf1052810508e955a226c27223f1a220ff1719c114eb/cec3b6664d735d495ceabf1052810508e955a226c27223f1a220ff1719c114eb-json.log",
"Name": "/kind-control-plane",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "kind",
"PortBindings": {
"6443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "57765"
}
]
},
"RestartPolicy": {
"Name": "on-failure",
"MaximumRetryCount": 1
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"Capabilities": null,
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "shareable",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 0,
"NanoCpus": 0,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 0,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/mnt/sda1/var/lib/docker/overlay2/3992cde94ea8cf3b2b7c8a578e346aa75d6eab06f9ecc4831c1dd2a2c81902bc-init/diff:/mnt/sda1/var/lib/docker/overlay2/7d69fe6ec2e1c537a4a35a95aae3f1fd3828bdaf760e9701a9f2a1004df32e51/diff:/mnt/sda1/var/lib/docker/overlay2/30e3098feffba1be8a684e076e0537febaa93c3c252cc6c39432bf138e6b125a/diff:/mnt/sda1/var/lib/docker/overlay2/87fa1831b77d25891034274d80b39bc4c0c5dc23adbe8d09f470b905884fdcd7/diff:/mnt/sda1/var/lib/docker/overlay2/4fc1e5d28eb9d9432d98c5e9503885ea894e5efb4cbf96c49def2357dcfa1f7a/diff:/mnt/sda1/var/lib/docker/overlay2/3c61016116368449f4d988c8cf2dab542a885ceb6eb2e063f9679851b7d602ae/diff:/mnt/sda1/var/lib/docker/overlay2/217edbc4da74c24a08729a80dedb3e59abc8718346188aef03e00ba9312d138b/diff:/mnt/sda1/var/lib/docker/overlay2/487af58ea9f453c48c8d08144b4271cd4a37de37e8a2c698efd04b7fae826253/diff",
"MergedDir": "/mnt/sda1/var/lib/docker/overlay2/3992cde94ea8cf3b2b7c8a578e346aa75d6eab06f9ecc4831c1dd2a2c81902bc/merged",
"UpperDir": "/mnt/sda1/var/lib/docker/overlay2/3992cde94ea8cf3b2b7c8a578e346aa75d6eab06f9ecc4831c1dd2a2c81902bc/diff",
"WorkDir": "/mnt/sda1/var/lib/docker/overlay2/3992cde94ea8cf3b2b7c8a578e346aa75d6eab06f9ecc4831c1dd2a2c81902bc/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "e85862a93d0ccd42dfe6cae95025561d890e09e4e661f398d219d5e22d07f418",
"Source": "/mnt/sda1/var/lib/docker/volumes/e85862a93d0ccd42dfe6cae95025561d890e09e4e661f398d219d5e22d07f418/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "kind-control-plane",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"6443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=docker"
],
"Cmd": null,
"Image": "kindest/node:v1.19.1@sha256:98cf5288864662e37115e362b23e4369c8c4a408f99cbc06e58ac30ddc721600",
"Volumes": {
"/var": {}
},
"WorkingDir": "",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"io.x-k8s.kind.cluster": "kind",
"io.x-k8s.kind.role": "control-plane"
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "2194745dd8c4218681497e8dc54f0bcb2c0fa01857d7b48a3b387378922fd79a",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"6443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "57765"
}
]
},
"SandboxKey": "/var/run/docker/netns/2194745dd8c4",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"kind": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"cec3b6664d73"
],
"NetworkID": "54a89324fb73656d1a0cf8f00dc83fcf9c93c53d42d11e264578e7eb97cf8afe",
"EndpointID": "ab0f51ebed00b3889f0cb0bb625ceea2b5b666b6dc8e2187545ddf61fbe51b1e",
"Gateway": "172.19.0.1",
"IPAddress": "172.19.0.2",
"IPPrefixLen": 16,
"IPv6Gateway": "fc00:f853:ccd:e793::1",
"GlobalIPv6Address": "fc00:f853:ccd:e793::2",
"GlobalIPv6PrefixLen": 64,
"MacAddress": "02:42:ac:13:00:02",
"DriverOpts": null
}
}
}
}
] |
|
The error field of inspecting the container has the issue. |
|
Mounting And I got a lot farther... Until...
and afterwards repeatedly:
I'm currently checking issues #979 #303 for possible solutions... |
these are both noise from kubeadm's bootstrapping process :| |
|
Perhaps there's an additional port going on here? cc @aojea we should probably make this section more robust. |
|
/cc |
|
Oh :D Sorry @BenTheElder and @aojea for the hassle about The Ok... Putting the wrapper aside... o.O And now it's getting really weird! When I let And when switching back to my @@ -190,7 +190,7 @@ func (p *provider) GetAPIServerEndpoint(cluster string) (string, error) {
cmd = exec.Command(
"docker", "inspect",
"--format", fmt.Sprintf(
- "{{ with (index (index .NetworkSettings.Ports \"%d/tcp\") 0) }}{{ printf \"%%s\t%%s\" .HostIp .HostPort }}{{ end }}", common.APIServerInternalPort,
+ "{{ with (index (index .NetworkSettings.Ports \"%d/tcp\") 0) }}{{ printf \"%%%%s\t%%%%s\" .HostIp .HostPort }}{{ end }}", common.APIServerInternalPort,
),
n.String(),
)... then I get a new error afterwards: With a little further debugging revealing that Seems like I should really try a different Docker installation... :) |
I went down that road before #377, if is not one thing is going to be other 🤷 ;) |
after running with |
|
I strongly suggest docker desktop as a known-working alternative in the meantime. As far as I know everything works fine with that. |
|
|
It sounds like for some reason we're unable to pipe input ..? Is this still with your shim script? |
|
will reopen if we find more actionable details, as it stands I don't think there's anything more to do on our end. |
Related to #1288
Microsoft Windows 10 Enterprise 10.0.18363 Build 18363docker-machine.exe version 0.15.0, build b48dc28dDocker version 17.06.0-ce, build 02c1d87kind version 0.9.0VirtualBox:
Linux default 4.14.154-boot2docker #1 SMP Thu Nov 14 19:19:08 UTC 2019 x86_64 GNU/LinuxDocker version 19.03.5, build 633a0ea838CMD:
Unfortunately, the related
/mnt/sda1/var/lib/docker/containers/cec3b6664d735d495ceabf1052810508e955a226c27223f1a220ff1719c114eb/ cec3b6664d735d495ceabf1052810508e955a226c27223f1a220ff1719c114eb-json.logis emptyHow to find any detailed information about what went wrong?
The text was updated successfully, but these errors were encountered: