Support CgroupsV2 in processors and Metricbeat

[andrewkroh]

Some Linux distros are starting to adopt CgroupsV2 and the container runtimes are being updated as well. Beats should be updated to support reading data from CgroupsV2. This affects Metricbeat's system/process metricset and some of the add_*_metadata processors that try to associate processes to container IDs.

• https://fedoraproject.org/wiki/Changes/CGroupsV2
• libcontainer: initial support for cgroups v2 opencontainers/runc#2113
• Tracker for cgroups v2 opencontainers/runtime-spec#1002

[fearful-symmetry]

Fedora has adopted it, as of F31: https://fedoraproject.org/wiki/Changes/CGroupsV2

I'll spin a VM sometime and test out what happens.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[fearful-symmetry]

After a quick test, I don't see any errors being thrown with the system/process or system/process_summary metricsets.

[andrewkroh]

@fearful-symmetry Did you get any cgroup metrics from system/process for processes that have cgroups applied?

[fearful-symmetry]

@andrewkroh I'm mostly looking for errors at this point. Testing this with docker metadata is functionally moot, as the only way to get docker working is to enable enable cgroupv1 backwards compatibility. What other use cases are there? Should I try manually adding PIDs to cgroups or something?

[andrewkroh]

Should I try manually adding PIDs to cgroups or something?

Yeah, either do that or use systemd to do it for you. systemd supports v2 on fedora31 from what I've read. So maybe install a beat RPM (or some other service) then edit the service unit file to add some resource constraint.

[exekias]

For inventory:

add_docker_metadata will also be affected if Docker adds support for v2, check

beats/libbeat/processors/add_docker_metadata/add_docker_metadata.go

Lines 219 to 266 in 1d36da7

	func (d *addDockerMetadata) lookupContainerIDByPID(event *beat.Event) string {
	var cgroups map[string]string
	for _, field := range d.pidFields {
	v, err := event.GetValue(field)
	if err != nil {
	continue
	}
	pid, ok := common.TryToInt(v)
	if !ok {
	d.log.Debugf("field %v is not a PID (type=%T, value=%v)", field, v, v)
	continue
	}
	cgroups, err = d.getProcessCgroups(pid)
	if err != nil && os.IsNotExist(errors.Cause(err)) {
	continue
	}
	if err != nil {
	d.log.Debugf("failed to get cgroups for pid=%v: %v", pid, err)
	}
	break
	}
	return getContainerIDFromCgroups(cgroups)
	}
	// getProcessCgroups returns a mapping of cgroup subsystem name to path. It
	// returns an error if it failed to retrieve the cgroup info.
	func (d *addDockerMetadata) getProcessCgroups(pid int) (map[string]string, error) {
	// Initialize at time of first use.
	lazyCgroupCacheInit(d)
	cgroups, ok := d.cgroups.Get(pid).(map[string]string)
	if ok {
	d.log.Debugf("Using cached cgroups for pid=%v", pid)
	return cgroups, nil
	}
	cgroups, err := processCgroupPaths(d.hostFS, pid)
	if err != nil {
	return nil, errors.Wrapf(err, "failed to read cgroups for pid=%v", pid)
	}
	d.cgroups.Put(pid, cgroups)
	return cgroups, nil
	}

.

Also add_process_metadata after this pr: #15947

[eyalkoren]

For the record - cgroup (v1 and v2) memory metrics issue in APM - elastic/apm#291.
See if you can use anything when implementing and let the APM team know if you think something can be done better.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[fearful-symmetry]

This is a good reminder, this is on the board for 7.15.

[graphaelli]

@fearful-symmetry is this one done now? It seems to have been moved on the project board without closing the issue. If it is complete, can you point us at the implementation?

[fearful-symmetry]

@graphaelli , yep, must have forgotten. Implementation is here: https://github.com/elastic/beats/tree/master/libbeat/metric/system/cgroup/cgv2

[fearful-symmetry]

Just going to close this, since it looks like the issue itself was limited to the basic groupsv2.