cadvisor interaction with CPU topology / SMT

[dghubble]

The very newest Kubelets (v1.19.0-alpha.1 - v1.19.0-beta.1, latest at time of writing) have trouble couting vCPUs on some OSes (incl. Fedora CoreOS), related to updates to cadvisor with CPU topology features. Within those versions, vCPU is detected as 0 (definitely wrong).

• General counting issues: Zero cpu's (cpu: 0) reported for nodes on K8s built on "master" kubernetes/kubernetes#89859
• Fedora CoreOS specific: Static pods never start w Kubelet v1.19.0-alpha.1/beta.2 on OSes with SMT disabled kubernetes/kubernetes#91795 (most relevant)
• Bump cadvisor: Update to latest cadvisor and containerd (v1.3.3) kubernetes/kubernetes#86975 (comment)
• Candidate fix: Fix #2566 warning instead of error when core_id or physical_package_id not available google/cadvisor#2567

A candidate fix detects the number of cores, unaware of disabled SMT, which I suspect is still wrong. I wanted to bubble this ongoing discussion to Fedora CoreOS folks in case someone has ideas on correctness.

State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200517.3.0 (2020-06-01T17:15:29Z)
                    Commit: 967b7b8d624e6d10ff51c2e81ef198fae966c567ac2e9b479771c693d0987949
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

[iwankgb]

I think that root-cause was a fact that cAdvisor has always been ignoring online/offline CPU information. It must be taken into account now and I believe that it should solve the problem. See: google/cadvisor#2567 (comment)

[dustymabe]

Considering the discussions going on in the upstream issues can someone summarize and let us know if there anything for FCOS to do in the context of this bug?

[dghubble]

Great info @iwankgb thanks. AFAIK, this has been narrowed down to cadvisor changes that detect effective vCPU count as 0, which seems to stem in whole or part from not handling OSes that disable simultaenous multi-threading (SMT) (Fedora CoreOS, Google's internal distro).

cadvisor currently reports the wrong CPU count on FCOS, they're discussing fixes, and that would eventually get merged and then included in a Kubernetes v1.19 beta or rc. I don't think FCOS would need to make any changes, I can close this out.