azure monitor sometimes sends 0 values

[arend-melissant-tnt]

Component(s)

receiver/azuremonitor

What happened?

Description

Sometimes the azure monitor get 0 values, this happens every few minutes

Steps to Reproduce

Expected Result

Retrieve azure monitor metric every minute

Actual Result

Some 25-50% of the time 0 values are received, other times >200 values

Collector version

v0.105

Environment information

Environment

Kubernetes

OpenTelemetry Collector configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: {{ tpl .Values.namespace . }}
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      redis:
        endpoint: "redis:6379"
        collection_interval: 60s
      azuremonitor:
        subscription_id: ${env:AZURE_SUBSCRIPTION_ID}
        tenant_id: ${env:AZURE_TENANT_ID}
        client_id: ${env:AZURE_CLIENT_ID}
        client_secret: ${env:AZURE_CLIENT_SECRET}
        resource_groups:
          - "{{ tpl .Values.resourcegroup . }}"
        collection_interval: 60s
        services:
          - "Microsoft.DBforPostgreSQL/flexibleServers"
          - "Microsoft.Storage/storageAccounts/blobServices"
          - "Microsoft.ServiceBus/Namespaces"
    processors:
      transform/ignore:
        error_mode: ignore
        log_statements:
          - context: log
            statements:
              - set(severity_text, "INFO2") where (severity_text == "INFO" and attributes["channel"] == "messenger")

      resource/tags:
        attributes:
          - key: deployment_environment
            value: {{ tpl .Values.namespace . }}
            action: upsert

      filter/no_info:
        error_mode: ignore
        logs:
          exclude:
            match_type: strict
            severity_texts:
              - "INFO"
    exporters:
      otlphttp:
        endpoint: https://{{ tpl .Values.uptraceurl . }}:443
        tls: { insecure: false }
        headers:
          uptrace-dsn: 'http://{{ tpl .Values.loggingsecret . }}@{{ tpl .Values.uptraceurl . }}:14318'
      debug:
        #verbosity: detailed
    service:
      telemetry:
        logs:
          level: "info"
      pipelines:
        traces:
          receivers: [otlp]
          processors: []
          exporters: [otlphttp, debug]
        metrics:
          receivers: [otlp, redis, azuremonitor]
          processors: [resource/tags]
          exporters: [otlphttp, debug]
        logs:
          receivers: [otlp]
          processors: [resource/tags, transform/ignore, filter/no_info]
          exporters: [otlphttp, debug]

Log output

│ 2024-07-26T11:57:51.806Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 115, "data points": 310}                              ││ 2024-07-26T11:58:48.589Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 0, "metrics": 0, "data points": 0}                                  ││ 2024-07-26T11:58:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T11:59:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T11:59:50.791Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 105, "data points": 275}                              ││ 2024-07-26T12:00:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 0, "metrics": 0, "data points": 0}                                  ││ 2024-07-26T12:00:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:01:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:01:50.401Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 145, "data points": 315}                              ││ 2024-07-26T12:02:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 0, "metrics": 0, "data points": 0}                                  ││ 2024-07-26T12:02:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:03:48.589Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:03:50.091Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 105, "data points": 275}                              ││ 2024-07-26T12:04:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:04:49.969Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 105, "data points": 275}                              ││ 2024-07-26T12:05:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:05:50.228Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 65, "data points": 65}                                ││ 2024-07-26T12:06:48.591Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:06:49.848Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 40, "data points": 210}                               ││ 2024-07-26T12:07:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:07:50.399Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 75, "data points": 75}                                ││ 2024-07-26T12:08:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:08:49.876Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 40, "data points": 210}                               ││ 2024-07-26T12:09:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:09:49.913Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 105, "data points": 275}                              ││ 2024-07-26T12:10:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:10:50.022Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 95, "data points": 265}                               ││ 2024-07-26T12:11:48.589Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:11:49.068Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 10, "data points": 10}                                ││ 2024-07-26T12:12:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:12:50.173Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 95, "data points": 265}                               ││ 2024-07-26T12:13:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:13:50.013Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 75, "data points": 75}                                ││ 2024-07-26T12:14:48.590Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}                                ││ 2024-07-26T12:14:49.761Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 40, "data points": 210}                               ││ 2024-07-26T12:15:48.589Z    info    MetricsExporter    {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 29, "data points": 32}

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/azuremonitor: @nslaughter @codeboten

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/azuremonitor: @nslaughter @codeboten

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

A code owner referenced this as being a valid issue in #35446, removing needs triage.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/azuremonitor: @nslaughter @codeboten

See Adding Labels via Comments if you do not have permissions to add labels yourself.