Bottlerocket node never fully starts - CNI plugin not initialized

[cyrus-mc]

Experimenting with Bottlerocket (currently use FedoraCoreOS). Spun up a singular node in a dev cluster.

Node successfully registers as it shows up in kubectl get nodes but in a NotReady status. SSH into admin container and I see the following error messages

• kubelet:

Jul 28 17:52:42 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T17:52:42.811446615Z" level=error msg="ExecSync for \"35617392921806dc1dfa16a17fb41b5c78b81a9f7af102a78f8d59882197723e\" failed" error="rpc error: code = DeadlineExceeded desc = failed to exec in container: timeout 1s exceeded: context deadline exceeded"

So this means AWS CNI container isn't starting. I did not that the image was not listed when doing ctr images list therefore I pulled it manually to stage the image on the system. This at least allow the aws-node POD to show up for that node - just in a CrashLoopBackoff status.

Looking at the containerd logs I see:

d06ac55720975c74bf360d70635164def972dd22a29301989d1579f\" for container &ContainerMetadata{Name:aws-node,Attempt:0,}"
Jul 27 22:30:30 ip-10-36-8-160.us-west-2.compute.internal containerd[1681]: time="2021-07-27T22:30:30.769316145Z" level=info msg="CreateContainer within sandbox \"a2291a407
d06ac55720975c74bf360d70635164def972dd22a29301989d1579f\" for &ContainerMetadata{Name:aws-node,Attempt:0,} returns container id \"ff4117a49ce375fb2f0e2ff721268c2d91dd0fb341
3cc0b18e86efb4e8f00ae4\""
Jul 27 22:30:30 ip-10-36-8-160.us-west-2.compute.internal containerd[1681]: time="2021-07-27T22:30:30.769826038Z" level=info msg="StartContainer for \"ff4117a49ce375fb2f0e2
ff721268c2d91dd0fb3413cc0b18e86efb4e8f00ae4\""
Jul 27 22:30:30 ip-10-36-8-160.us-west-2.compute.internal containerd[1681]: time="2021-07-27T22:30:30.862879383Z" level=info msg="StartContainer for \"ff4117a49ce375fb2f0e2
ff721268c2d91dd0fb3413cc0b18e86efb4e8f00ae4\" returns successfully"
Jul 27 22:30:41 ip-10-36-8-160.us-west-2.compute.internal containerd[1681]: time="2021-07-27T22:30:41.446398019Z" level=info msg="Timeout received while waiting for exec pr
ocess kill \"181edc7b01fffefa0e86f84344a88410b64dd7904311f4bd9c59f6f26b1b1427\" code 137 and error <nil>"
Jul 27 22:30:41 ip-10-36-8-160.us-west-2.compute.internal containerd[1681]: time="2021-07-27T22:30:41.448594152Z" level=error msg="ExecSync for \"ff4117a49ce375fb2f0e2ff721
268c2d91dd0fb3413cc0b18e86efb4e8f00ae4\" failed" error="rpc error: code = DeadlineExceeded desc = failed to exec in container: timeout 1s exceeded: context deadline exceede
d"
Jul 28 18:14:02 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:02.809540067Z" level=error msg="ExecSync for \"d68355f0c8aaf80e6068256ce353c7ebe46920f168f774013c191a4652200881\" failed" error="rpc error: code = DeadlineExceeded desc = failed to exec in container: timeout 1s exceeded: context deadline exceeded"
Jul 28 18:14:06 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:06.755712150Z" level=info msg="Kill container \"d68355f0c8aaf80e6068256ce353c7ebe46920f168f774013c191a4652200881\""
Jul 28 18:14:06 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:06.765001183Z" level=info msg="Finish piping stdout of container \"d68355f0c8aaf80e6068256ce353c7ebe46920f168f774013c191a4652200881\""
Jul 28 18:14:06 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:06.765064904Z" level=info msg="Finish piping stderr of container \"d68355f0c8aaf80e6068256ce353c7ebe46920f168f774013c191a4652200881\""
Jul 28 18:14:06 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:06.771626380Z" level=info msg="TaskExit event &TaskExit{ContainerID:d68355f0c8aaf80e6068256ce353c7ebe46920f168f774013c191a4652200881,ID:d68355f0c8aaf80e6068256ce353c7ebe46920f168f774013c191a4652200881,Pid:352603,ExitStatus:137,ExitedAt:2021-07-28 18:14:06.771458087 +0000 UTC,XXX_unrecognized:[],}"
Jul 28 18:14:06 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:06.826693914Z" level=info msg="shim disconnected" id=d68355f0c8aaf80e6068256ce353c7ebe46920f168f774013c191a4652200881
Jul 28 18:14:06 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:06.826765921Z" level=error msg="copy shim log" error="read /proc/self/fd/45: file already closed"
Jul 28 18:14:06 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:06.828858731Z" level=info msg="StopContainer for \"d68355f0c8aaf80e6068256ce353c7ebe46920f168f774013c191a4652200881\" returns successfully"
Jul 28 18:14:07 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:07.421430382Z" level=info msg="RemoveContainer for \"ae4ca6d68ad857f4c76d45c3792719a293c01860d43264e97d6fd41bac3f38a2\""
Jul 28 18:14:07 ip-10-36-8-160.us-west-2.compute.internal containerd[344078]: time="2021-07-28T18:14:07.425221886Z" level=info msg="RemoveContainer for \"ae4ca6d68ad857f4c76d45c3792719a293c01860d43264e97d6fd41bac3f38a2\" returns successfully"

In terms of pulling the CNI image the correct policy is attached to the node (ECRReadOnly). Kind of stuck as to where to look next.

[webern]

Thanks for checking out Bottlerocket. Taking a look at the instance roles that I use when launching Bottlerocket, I have the AmazonEC2ContainerServiceforEC2Role AWS managed policy attached to my role. Is this what you are using?

Edit: I was looking at the wrong role, it looks like I use AmazonEC2ContainerRegistryReadOnly. Is this the one you are using?

[cyrus-mc]

I can't connect to any resources within my VPC. Yet I can hit external resources (such as Google.com).

[webern]

Could you open an AWS Support case and send me the case number brigmatt at amazon dot com?

Here is some additional information that we may need:

• EKS Control Plane Version
• AWS-VPC-CNI Version
• Bottlerocket Version
• Bottlerocket Variant
• Instance Architecture (x86_64/Arm)

Thank you.

[webern]

Still thinking...

It sounds like maybe the security group of the control plane does not allow ingress from the node.

• Did you set up the cluster with eksctl?
  • And if so, does the Bottlerocket node have the eksctl-clustername-cluster-ClusterSharedNodeSecurityGroup-... and eksctl-clustername-nodegroup-ng-... security groups attached? (eksctl sets up a control plane security group and a node security group and allows ingress from the node sg to the control plane sg.)
• How different was your cluster creation and launch process from what is described in the quickstart?

[cyrus-mc]

So long story short what I was seeing wasn't making sense to me. It wasn't EKS related. The instance didn't seem to be able to communicate with anything within the VPC. So I thought - "Am I on a bad host". I spun up a new instance and it works. Must have gotten onto some wonky host that is experiencing issues.

[webern]

I spun up a new instance and it works.

OK, that's great news that you've gotten past the blocker. Definitely let us know if you see this behavior again.