[Infra Monitoring UI] Add module filter to metrics tables

[miltonhultgren]

With the infrastructure node metrics tables we've seen that sometimes a filter can cause documents to be included that don't have the relevant metric fields but still contain for example host.name.

An example is a filter on container.id might match a Kubernetes pod document, which does have host.name but it means the name of the Pod, not the machine the Pod/Container runs on. So we'd end up showing a Pod as a host and with no data.

While the main issue is that we filter on container.id when we want only hosts, it might be wise to safeguard against unintended filters by adding a filter clause for the module we expect to pull metrics from.

For host, this is system. For Pods and Containers we'd have to check which module those specific fields are pulled from.

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)

[crespocarlos]

On the scenario below, the host doesn't have metricbeat installed, but the data about it exists on APM indices and that's why we can see it here.

The APM indices do have system related info, eg: system.memory.actual, system.memory.total and system.process.cpu.total.norm.pct.

Should we ignore APM indices info and only show hosts that have metricbeat data, by filtering by event.dataset for example? @miltonhultgren @jasonrhodes

[crespocarlos]

Actually, when there is no metricbeat data available, there won't be any details to be presented:

[miltonhultgren]

Follow up to this based on Slack discussion with Carlos, to what extent are documents from the Agent system integration the same "value"?

[crespocarlos]

For this ticket I'd say that we should focus on what is the main ask - to add a new filter. Let's maybe raise another one to understand how Agents impact this. wdyt?

[jasonrhodes]

@crespocarlos I agree on focusing in here. What we want to avoid is assuming that because a document has a certain value (e.g. x.y.z), we can assume things about other values in that document. By using a filter on the type of document it is, we can avoid that. That may prevent us from sort of accidentally benefiting from other documents that provided relevant information by happenstance, but I think that's still a good thing.

If we want to take advantage of APM documents that have metric data, we should explicitly opt into them with another similar filter. We can look into that in a separate ticket for sure. This has a lot of overlap with the entity model work I'm doing so I would love to be involved there.

Thanks for digging into this, everyone!

[miltonhultgren]

What we want to avoid is assuming that because a document has a certain value (e.g. x.y.z), we can assume things about other values in that document.

This is in essence the original issue, we assumed that any document with host.name is a host metrics document which isn't true at all.

If we want to take advantage of APM documents that have metric data, we should explicitly opt into them with another similar filter.

💯