Fix k8s metadata issue

[ChrsMark]

What does this PR do?

This PR fixes labels in k8s metadata. Issues found in 7.6 version: https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/6

The proposed solution is to add kubernetes.labels.* in all resources (node, namespace etc) so as to cover cases like node metricset. Also when node, namespace, are part of other resources like pod then their labels are part of the sub-resource and not the main resource.

The proposed solution is to add kubernetes.labels.* in node resources so as to cover cases like node metricset. Also when node, is part of other resources like pod then their labels are part of the sub-resource and not the main resource.

Example with node as main resource :

{
    "node": {
        "name": "testnode",
        "uid":  "uid",
    },
    "labels": {
        "nodekey": "nodevalue",
    }
}

Example with sub-resources:

{
    "pod": {
        "name": "obj",
        "uid":  "uid",
    },
    "namespace": "default",
    "namespace_uid": "uid",
    "namespace_labels": {
            "nskey": "nsvalue",
    },
    "node": {
        "name": "testnode",
        "uid":  "uid",
        "labels": {
            "nodekey": "nodevalue",
        },
    },
    "labels": {
        "foo": "bar",
    },
    "annotations": {
        "app": "production",
    }
}

Why is it important?

Currently the metadata are not populated properly.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

TBA

Related issues

• Relates Fix k8s pod labels tier in metadata #16480, Fix k8s pod annotations tier in metadata #16554, Define mapping for new k8s resources #16558

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[jsoriano]

I am fine with this change for 8.0, but I am not sure of seeing the advantages of backporting this to 7.x versions.

[josharrington]

Has there been any update on this issue? This is a significant breaking change moving to 7.6.x and it makes it impossible for us to group nodes or pods by kubernetes cluster.

[ChrsMark]

@jsoriano @exekias I did some tests with this approach and seems to fix the issue with labels in general while preserving namespace schema for now until we decide on a final schema along with a breaking change. Let me know what you think!

[blakerouse]

The approach seems acceptable to me, at least until 8.0.

[exekias]

LGTM!

[exekias]

I am fine with this change for 8.0, but I am not sure of seeing the advantages of backporting this to 7.x versions.

I understand that after the latest updates we are fine with backporting this, right @jsoriano?

[jsoriano]

I am fine with this change for 8.0, but I am not sure of seeing the advantages of backporting this to 7.x versions.

I understand that after the latest updates we are fine with backporting this, right @jsoriano?

Looks good now, yes. Thanks!