Provide a way to add annotations to Pods

[indrekj]

Hi,

We're using DataDog to collect metrics. I've noticed that prometheus.io/scrape is added to Deployment. DD supports prometheus.io/scrape but it has to be in the Pod spec, not in the Deployment spec. Their current implementation is quite limited though and I'd prefer to use their OpenMetrics autodiscovery hook.

For that I'd need to add annotations to Pod, e.g.:

  annotations:
    ad.datadoghq.com/otc-container.check_names: '["openmetrics"]'
    ad.datadoghq.com/otc-container.init_configs: '[{}]'
    ad.datadoghq.com/otc-container.instances: |-
      [
        {
          "prometheus_url": "http://%%host%%:8888/metrics",
          "namespace": "opentelemetry.collector",
          "metrics": [
            {"otelcol_exporter_queue_size": "exporter.queue_size"}
          ]
        }
      ]

I noticed that it is possible to add annotations to OpenTelemetryCollector CRD, but then this annotation is propagated to everything (deployment, services, pods, etc). This causes error noise and I'd like to avoid it.

It would be good to be able to control Pod annotations.

[jpkrohling]

Is this about the sidecar that is injected? Or is it about the PodSpec of a collector instance as Deployment/Daemonset/Statefulset?

[indrekj]

The latter: PodSpec of a collector instance as deployment.

[jpkrohling]

The only problem I have with this is that it leaks the detail about the pod(s) that is being created by the operator. For instance, right now we create only one pod for most CRs. In the future, we might create more (see the TargetAllocator feature that was just merged). In that case, should the annotations specified here be applied to all pods created as a result of a CR?

[indrekj]

In our case, we would like to annotate all the pods. Then in DataDog we can aggregate them and see how things are performing. It also allows showing how one particular pod is performing.

[jpkrohling]

Just to double-check: if you are using the target allocator, you'd want the annotations to be present at both the pods for the target allocator and for the collector, right?

[indrekj]

I'm not that familiar with the target allocator yet so pardon my ignorance. I assume you mean collector pods that target allocator creates (agents I guess?) and collector (as in gateway?), then yes. I would like to annotate all of them. If I could provide different annotations for them, then that would be even better but both should work for us.

EDIT: Just to clarify. We're currently using just "Gateway" with deployment type. Soon we're also planning to add agents likely in a form of sidecars that proxy everything to the gateway pods.

[jpkrohling]

The target allocator is a different deployment that is created when Prometheus is used. Sounds good to me, I think we can start by adding the annotation to all pods and see how it goes.

Would you like to open a PR with that?

[indrekj]

Tried to put something together: #451