Use alias to report container image in k8s parts

[ChrsMark]

What does this PR do?

This PR leverages type :alias for kubernetes.container.image so as to copy the fields under ECS container.image.name.
This affects metadata collected by kubernetes autodiscover provider, add_kubernetes_metadata processor and state_container metricset.

Why is it important?

In order to have a persistent way to report ECS container fields, without the need to duplicate fields and store both which will impact ECS.

How to test this PR locally

1. Before all test verify that fields.yml on your testing env is properly updated with the alias change and manually setup and verify that mappings are properly loaded in ES.
2. Check that autodiscover meta work:

filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

Make sure that container.image.name exist in the final documents in ES and you able to search on based on it while kubernetes.container.image is not populated in the event but is searchable because of the alias.
3. Check that add_kubernetes_metadata processor works:

filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            indexers:
            - container:
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

Make sure that container.image.name exist in the final documents in ES and you able to search on based on it while kubernetes.container.image is not populated in the event but is searchable because of the alias.

4. Verify with state_container metricset enabled that, container.image.name exist in the final documents in ES and you able to search based on it while kubernetes.container.image is not populated in the event but is searchable because of the alias.

Related issues

• Closes Verify ECS alignment in container fields reported by kubernetes module and meta processor #23585
• Relates Add container ECS fields in kubernetes metadata #20984, Add container image and container name ECS fields for state_container #23802

Screenshots

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #24380 updated

• Start Time: 2021-03-10T10:38:51.383+0000

• Duration: 73 min 26 sec

• Commit: 67a4d9b

Test stats 🧪

Test	Results
Failed	0
Passed	45249
Skipped	4972
Total	50221

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	45249
Skipped	4972
Total	50221

[ChrsMark]

@jsoriano @kaiyan-sheng could you please take another look please? #24380 (comment) seems to be valid since code should set the values under ECS fields and not under kubernetes.container.*. kubernetes.container.* will be populated through the alias. I also added testing notes and some screenshots.

[kaiyan-sheng]

Change looks good to me, do you think it's worth a changelog?

[ChrsMark]

Change looks good to me, do you think it's worth a changelog?

Well if this change is correct it should be transparent to the end users. Otherwise it's a breaking change 😬 ?

[jsoriano]

A couple of minor comments but it looks good.

[jsoriano]

Move these deletions to the previous ifs where it is checked if these fields exist?

[jsoriano]

This comment looks a bit confusing to me. Not all these kubernetes.* meta fields are moved, right?

[jsoriano]

Change looks good to me, do you think it's worth a changelog?

Well if this change is correct it should be transparent to the end users. Otherwise it's a breaking change 😬 ?

I think that it can be a good idea to add a changelog entry for this change, in the breaking changes section, even if it should be transparent for most users. Even if not recommended, there can be users using custom mappings or no mapping at all and this change can be breaking for them. The entry could say something as "kubernetes.container.name and kubernetes.container.image fields are now aliases to their ECS equivalents".

Btw, it could be also breaking in the Metrics UI if it uses the old kubernetes fields and it doesn't handle aliases. We should check this, preferably before merging. @ChrsMark could you confirm this?

[jsoriano]

Sorry, I have just thought on something that can be important. kubernetes.container.name and container.name are actually not the same thing, and maybe this was the reason why this field was not filled here.
kubernetes.container.name is the name of a container in its pod, this is a kubernetes-specific concept and can make sense in a kubernetes-specific field.
container.name would be the container name in the runtime, that will be probably different, but would be the same concept in multiple orchestrators using the same container runtime.

[jsoriano]

On the other hand, the image reported by kubernetes.container.image is the image in the runtime, so it is probably ok if this is moved to a common field.

[ChrsMark]

Long story short, kubernates.container.name field should not be the same as container.name field since the former is the container name as defined in Pod's spec while the later is the (actual) name of the container as reported by the runtime.

In this, this PR will only apply the change on kubernetes.image.name -> container.image.name.

I updated the description with accordingly (with new screenshots). @kaiyan-sheng @jsoriano sorry for the back-and-forth on this.

[jsoriano]

Thanks!