K8s observer should emit Pod endpoints per container

[ChrsMark]

Component(s)

extension/observer/k8sobserver

Describe the issue you're reporting

Today the k8sobserver emits 1 Pod endpoint by default (if the Pod is running) and then multiple Port endpoints per container port that is mapped to a container.

opentelemetry-collector-contrib/extension/observer/k8sobserver/pod_endpoint.go

Line 17 in 5133f4c

func convertPodToEndpoints(idNamespace string, pod *v1.Pod) []observer.Endpoint {

Even though this logic can cover the metrics' use-cases since we can enable different receivers based on the different container ports, it cannot cover a logs' use-case where we want to handle each container with a separate parsing logic.

Given the following configuration:

    receivers:
      receiver_creator:
        watch_observers: [ k8s_observer ]
        receivers:
          filelog:
            rule: type == "pod" && labels["otel.logs"] == "true"
            config: 
              include:
              - /var/log/pods/`namespace`_`name`_`uid`/*/*.log
              include_file_path: true
              operators:

We would like to scope the log's file path more including the container's name to sth like /var/log/pods/`namespace`_`name`_`uid`/`container.name`/*.log

This would allow us to handle logs from different containers separately.
For example:

    receivers:
      receiver_creator:
        watch_observers: [ k8s_observer ]
        receivers:
          filelog/nginx:
            rule: type == "pod" && labels["otel.logs"] == "true" && container.name == "nginx"
            config: 
              include:
              - /var/log/pods/`namespace`_`name`_`uid`/`container.name`/*.log
              include_file_path: true
              operators:
              ### apply nginx logs parsing logic here
           filelog/apache:
            rule: type == "pod" && labels["otel.logs"] == "true" && container.name == "apache"
            config: 
              include:
              - /var/log/pods/`namespace`_`name`_`uid`/`container.name`/*.log
              include_file_path: true
              operators:
              ### apply apache logs parsing logic here

This might be a corner case but from what I can understand it's quite equivalent to the reason we emit multiple Port endpoints.

Maybe we could introduce a new type of Endpoint called pod.container and emit this additionally.

That's also related to #34427.

/cc @dmitryax

[github-actions]

Pinging code owners:

• extension/observer/k8sobserver: @rmfitzpatrick @dmitryax

See Adding Labels via Comments if you do not have permissions to add labels yourself.