Reduce kubelet logs 'Failed to create existing container' when kubelet is using crio

[dsxing]

fix #3456

when We found that these logs are related to crio sandbox containers. The root cause is that when a pod is created, there will be a sandbox container folder in the cgroup file. When cadvisor scans the cgroup directory to identify containers, it recognizes the sandbox container. It then uses CanHandleAndAccept to determine that it needs to retrieve container information from crio using the id parameter. However, when attempting to get container information from crio via grpc, crio does not query the sandbox container, resulting in a 404 error(https://github.com/cri-o/cri-o/blob/main/server/inspect.go#L68). so cadvisor createContainer failed and print the logs periodically(https://github.com/google/cadvisor/blob/master/manager/manager.go#L1116)

We think that this situation should be handled in the crio factory, filtering out sandbox containers. similar to this pr:#2957 for https://bugzilla.redhat.com/show_bug.cgi?id=1978528#c4

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[k8s-ci-robot]

Hi @dsxing. Thanks for your PR.

I'm waiting for a google member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[iwankgb]

/ok-to-test

[dsxing]

@iwankgb hi, can you help to review the code? thanks!

[dims]

@dsxing please see cri-o/cri-o#8034 - this commit caused a regression when systemd is absent, We are trying to see if we should revert this PR

[dims]

xref: #3565

[haircommander]

in the case of systemd cgroup manager in cri-o and when we drop the infra container (as in, we "create" it but it's not a running container because it's not technically required), we create a dummy cgroup so cadvisor recognizes the pod and can report networking metrics.

I think this is actually a bug in crio if we're not correctly finding the infra container. we should be adding it to the container server even though we've dropped it. Do you have a succinct reproducer @dsxing ?

[dsxing]

@dsxing please see cri-o/cri-o#8034 - this commit caused a regression when systemd is absent, We are trying to see if we should revert this PR

I'm sorry for submitted an incorrect PR. I believe we should revert this PR.

[dsxing]

in the case of systemd cgroup manager in cri-o and when we drop the infra container (as in, we "create" it but it's not a running container because it's not technically required), we create a dummy cgroup so cadvisor recognizes the pod and can report networking metrics.

I think this is actually a bug in crio if we're not correctly finding the infra container. we should be adding it to the container server even though we've dropped it. Do you have a succinct reproducer @dsxing ?

I'm sorry for being away for so long. The initial issue was that after restarting crio, kubelet would print a large number of "Failed to create existing container" logs(see kubernetes/kubernetes#115609 ; kubernetes/kubernetes#120988). To reduce these log messages, I submitted this PR. However, after further analyzing the root cause, I discovered that it was an issue in crio. In the end, I submitted a PR(cri-o/cri-o#7726) that fundamentally resolved the problem, so I believe we should revert this PR. I apologize for any other issues this may have caused.