Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Collecting Filesystem metrics makes extremely high CPU #3082

Open
Shaye-man opened this issue Mar 23, 2022 · 5 comments · May be fixed by #3348
Open

Collecting Filesystem metrics makes extremely high CPU #3082

Shaye-man opened this issue Mar 23, 2022 · 5 comments · May be fixed by #3348

Comments

@Shaye-man
Copy link

PR #2768 fixed docker can collect FileSystem metrics. But in the scenario of multiple devices, per seconds(housekeeping time) collect FileSystem metrics causes kubelet CPU usage extremely high and impacts the application responsiveness.
This should be optimized or fixed in cadvisor.
CPU profiling data shows the hot code is here:
20220323-163351(WeLinkPC)

cadvisor/container/docker/handler.go

Lines 402 to 430 in dca17e6

var fsInfo *info.FsInfo
// Docker does not impose any filesystem limits for containers. So use capacity as limit.
for _, fs := range mi.Filesystems {
if fs.Device == device {
limit = fs.Capacity
fsType = fs.Type
fsInfo = &fs
break
}
}
fsStat := info.FsStats{Device: device, Type: fsType, Limit: limit}
usage := h.fsHandler.Usage()
fsStat.BaseUsage = usage.BaseUsageBytes
fsStat.Usage = usage.TotalUsageBytes
fsStat.Inodes = usage.InodeUsage
if fsInfo != nil {
fileSystems, err := h.fsInfo.GetGlobalFsInfo()
if err == nil {
addDiskStats(fileSystems, fsInfo, &fsStat)
} else {
klog.Errorf("Unable to obtain diskstats for filesystem %s: %v", fsStat.Device, err)
}
}
stats.Filesystem = append(stats.Filesystem, fsStat)

cadvisor/fs/fs.go

Lines 451 to 472 in dca17e6

func getDiskStatsMap(diskStatsFile string) (map[string]DiskStats, error) {
diskStatsMap := make(map[string]DiskStats)
file, err := os.Open(diskStatsFile)
if err != nil {
if os.IsNotExist(err) {
klog.Warningf("Not collecting filesystem statistics because file %q was not found", diskStatsFile)
return diskStatsMap, nil
}
return nil, err
}
defer file.Close()
scanner := bufio.NewScanner(file)
for scanner.Scan() {
line := scanner.Text()
words := strings.Fields(line)
if !partitionRegex.MatchString(words[2]) {
continue
}
// 8 50 sdd2 40 0 280 223 7 0 22 108 0 330 330
deviceName := path.Join("/dev", words[2])

With kubelet (version 1.22.1) and about 202 block device, Probably consumed 40%CPU
image
image

When I try to disable collecting FileSystem metrics, like this, only consumed 20%CPU, kubelet restore normal consumption.

 // Docker does not impose any filesystem limits for containers. So use capacity as limit. 
 for _, fs := range mi.Filesystems { 
 	if fs.Device == device { 
 		limit = fs.Capacity 
 		fsType = fs.Type 
 		fsInfo = nil 
 		break 
 	} 
 }

image
And normal pprof like this:
normal
@Shaye-man
Copy link
Author

@JohnnyG235711 @iwankgb cc

@hedinasr
Copy link

@Shaye-man how did you managed to disable collecting filesystem metrics ?

@iwankgb
Copy link
Collaborator

iwankgb commented Jul 13, 2023 •
edited
Loading

@Shaye-man, I hope that #3257 will pave way towards improved performance.

EDIT: I know it looks to be abandoned.

@iwankgb iwankgb linked a pull request Jul 20, 2023 that will close this issue
Open
@chenk008
Copy link

It causes this problem #3349

@iwankgb
Copy link
Collaborator

iwankgb commented Jul 25, 2023

@chenk008 #3349 has been merged already - it was pretty neat fix to perfectly valid problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Bringing number of syscalls down in fs.getDistStatsMap() iwankgb/cadvisor
4 participants
@iwankgb @chenk008 @hedinasr @Shaye-man