prometheusremotewrite context deadline exceeded #31910

frakev · 2024-03-22T15:49:12Z

Component(s)

exporter/prometheusremotewrite

What happened?

Description

If the endpoint is not reachable and OTEL can't send metrics, I get some error messages.

Steps to Reproduce

Configure a prometheus receiver which will scrap some metrics or itself.
Configure an exporter (prometheusremotewrite) with one endpoint.
Configure a processor. In my case "batch" processor.
And finally, configure a service.
Start your OTEL collector to scrape and send metrics.
After a few minutes, try to simulate an endpoint crash. See "Actual result".

Expected Result

No error messages, but an info log to let you know that the collector is queuing the metrics due to endpoint downtime.

Actual Result

2024-03-22T14:35:06.566Z error exporterhelper/queue_sender.go:97 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: context deadline exceeded", "dropped_items": 2353} go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1 go.opentelemetry.io/collector/exporter@v0.96.0/exporterhelper/queue_sender.go:97 go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume go.opentelemetry.io/collector/exporter@v0.96.0/internal/queue/bounded_memory_queue.go:57 go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1 go.opentelemetry.io/collector/exporter@v0.96.0/internal/queue/consumers.go:43 2024-03-22T14:35:12.008Z error exporterhelper/queue_sender.go:97 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: context deadline exceeded", "dropped_items": 24} go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1 go.opentelemetry.io/collector/exporter@v0.96.0/exporterhelper/queue_sender.go:97 go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume go.opentelemetry.io/collector/exporter@v0.96.0/internal/queue/bounded_memory_queue.go:57 go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1 go.opentelemetry.io/collector/exporter@v0.96.0/internal/queue/consumers.go:43

Collector version

0.96.0

Environment information

Environment

Docker image: otel/opentelemetry-collector:0.96.0

OpenTelemetry Collector configuration

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: node-exporter
          scrape_interval: 5s
          static_configs:
            - targets: [localhost:9100]
        - job_name: otel-collector
          scrape_interval: 5s
          static_configs:
            - targets: [localhost:8888]

exporters:
    prometheusremotewrite:
      endpoint: http://X.X.X.X/push
      headers:
        Authorization: "Bearer toto"
      remote_write_queue:
        enabled: True
        queue_size: 100000
        num_consumers: 5
    debug:
      verbosity: detailed

processors:
  batch:

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [prometheusremotewrite]
  telemetry:
    metrics:
      level: detailed
      address: 0.0.0.0:8888

Log output

2024-03-22T14:35:06.566Z	error	exporterhelper/queue_sender.go:97	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: context deadline exceeded", "dropped_items": 2353}
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
	go.opentelemetry.io/collector/exporter@v0.96.0/exporterhelper/queue_sender.go:97
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
	go.opentelemetry.io/collector/exporter@v0.96.0/internal/queue/bounded_memory_queue.go:57
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
	go.opentelemetry.io/collector/exporter@v0.96.0/internal/queue/consumers.go:43
2024-03-22T14:35:12.008Z	error	exporterhelper/queue_sender.go:97	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: context deadline exceeded", "dropped_items": 24}
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
	go.opentelemetry.io/collector/exporter@v0.96.0/exporterhelper/queue_sender.go:97
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
	go.opentelemetry.io/collector/exporter@v0.96.0/internal/queue/bounded_memory_queue.go:57
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
	go.opentelemetry.io/collector/exporter@v0.96.0/internal/queue/consumers.go:43

Additional context

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2024-03-22T15:49:27Z

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

martinohansen · 2024-04-10T07:58:39Z

I'm seeing something similar. I added some details over on this issue: open-telemetry/opentelemetry-collector#8217 (comment)

martinohansen · 2024-04-10T17:58:54Z

I managed to solve (or brute force?) the issue by setting this in the exporter, the default is 5 consumers.

remote_write_queue:
  num_consumers: 50

frakev · 2024-04-11T07:12:12Z

Thank you @martinohansen. I'll try, but it's just a workaround and maybe this issue needs to be fixed.

dhilgarth · 2024-04-14T21:49:10Z

I'm having a similar error, but increasing the num_consumers didn't help.
I'm running otel collector contrib 0.98.0 with global mode - so, 1 instance per node - in Docker Swarm.
My configuration is generally working, as in 4 out of 6 instances can properly send their metrics to the endpoint.

However, two instances can't do that and it fails with:

2024-04-14T21:40:38.279Z	error	exporterhelper/queue_sender.go:101	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite/metrics-infrastructure", "error": "Permanent error: Permanent error: context deadline exceeded", "dropped_items": 1425}
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
	go.opentelemetry.io/collector/exporter@v0.98.0/exporterhelper/queue_sender.go:101
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
	go.opentelemetry.io/collector/exporter@v0.98.0/internal/queue/bounded_memory_queue.go:57
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
	go.opentelemetry.io/collector/exporter@v0.98.0/internal/queue/consumers.go:43

No more info is available.

This is my config:

receivers:
  hostmetrics:
    collection_interval: 3s
    root_path: /hostfs
    scrapers:
      cpu:
        metrics:
          system.cpu.logical.count:
            enabled: true
          system.cpu.physical.count:
            enabled: true
          "system.cpu.frequency":
            enabled: true
          "system.cpu.utilization":
            enabled: true
      load: { }
      memory:
        metrics:
          "system.linux.memory.available":
            enabled: true
          "system.memory.limit":
            enabled: true
          "system.memory.utilization":
            enabled: true
      disk: { }
      filesystem:
        metrics:
          "system.filesystem.utilization":
            enabled: true
      paging:
        metrics:
          "system.paging.utilization":
            enabled: true
          "system.paging.usage":
            enabled: true
      network: { }
      process:
        mute_process_io_error: true
        mute_process_exe_error: true
        mute_process_user_error: true
        metrics:
          "process.cpu.utilization":
            enabled: true
          "process.memory.utilization":
            enabled: true
          "process.disk.io":
            enabled: true
          "process.disk.operations":
            enabled: true
          process.threads:
            enabled: true
          process.paging.faults:
            enabled: true
processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s
exporters:
  prometheusremotewrite/metrics-infrastructure:
    endpoint: http://mimir-lb:9010/api/v1/push
    tls:
      insecure: true
    headers:
      - "X-Scope-OrgID": "infrastructure"
    resource_to_telemetry_conversion:
      enabled: true
    remote_write_queue:
      enabled: true
      queue_size: 100000
      num_consumers: 50

service:
  telemetry:
    logs:
      level: debug
    metrics:
      level: detailed
      address: 0.0.0.0:8888
  pipelines:
    metrics/infrastructure:
      receivers: [ hostmetrics]
      processors: [ batch ]
      exporters: [ prometheusremotewrite/metrics-infrastructure ]

What can we do to further troubleshoot this issue?

dhilgarth · 2024-04-14T22:48:41Z

After 6 hours, I finally figured it out: An nginx config in mimir-lb that doesn't update the IP addresses of the upstream servers. One of the upstream containers restarted and got a new IP address, which wasn't reflected in nginx.
The two containers that exhibited this problem must've been routed to the restarted instance every single time.

github-actions · 2024-06-14T03:30:30Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

koenvandesande · 2024-06-18T13:45:44Z

I'm encountering the same error. From the error messages, it is not clear to me whether writing to the remote endpoint is failing (i.e. does Permanent error: context deadline exceeded come from that server?), or is some local endpoint not being scraped properly, i.e. timeouts?

juniorsdj · 2024-06-26T20:41:04Z

I am getting a similar issue here. I am using the opentelemetry-collector-contrib: 0.103.1
my collector is processing 200k span/min
processing on spanmetrics connector and finally sending to prometheus instance.
After a couple of hours running, the collector begin to show this error message.

2024-06-26T20:06:15.292972825Z 2024-06-26T20:06:15.292Z	error	exporterhelper/queue_sender.go:90	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded; Permanent error: Permanent error: context deadline exceeded", "errorCauses": [{"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadl
ine exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}, {"error": "Permanent error: Permanent error: context deadline exceeded"}], "dropped_items": 12404}
2024-06-26T20:06:15.293013725Z go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
2024-06-26T20:06:15.293019456Z 	go.opentelemetry.io/collector/exporter@v0.103.0/exporterhelper/queue_sender.go:90
2024-06-26T20:06:15.293023534Z go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
2024-06-26T20:06:15.293026425Z 	go.opentelemetry.io/collector/exporter@v0.103.0/internal/queue/bounded_memory_queue.go:52
2024-06-26T20:06:15.293029184Z go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
2024-06-26T20:06:15.293031423Z 	go.opentelemetry.io/collector/exporter@v0.103.0/internal/queue/consumers.go:43```

github-actions · 2024-08-26T03:32:45Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

dashpole · 2024-08-26T13:46:09Z

triage:

Context deadline exceeded means the client timed out when trying to send to the backend.
The error is generic, so it is likely that people posting above have different root causes.
One potential way to mitigate would be to increase the timeout of the exporter. If this is a common issue, we may want to increase the default timeout, which is 5 seconds.
For people that want to debug this, I would recommend using the http self-observability metrics, but note that those will change over time. The PRW exporter will currently produce metrics if the service metrics level is set to detailed: https://opentelemetry.io/docs/collector/internal-telemetry/#additional-detailed-level-metrics

cabrinha · 2024-08-27T23:57:27Z

I'm also getting this error when trying to use prometheusremotewrite... Can't figure out what the issue is. Error from prometheusremotewrite exporter:

2024-08-28T00:11:46.851Z        error   exporterhelper/queue_sender.go:92       Exporting failed. Dropping data.        {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: context deadline exceeded", "dropped_items": 636}
go.opentelemetry.io/collector/exporter/exporterhelper.newQueueSender.func1
        go.opentelemetry.io/collector/exporter@v0.106.1/exporterhelper/queue_sender.go:92
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
        go.opentelemetry.io/collector/exporter@v0.106.1/internal/queue/bounded_memory_queue.go:52
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
        go.opentelemetry.io/collector/exporter@v0.106.1/internal/queue/consumers.go:43

Config:

    exporters:
      prometheusremotewrite:
        add_metric_suffixes: false
        endpoint: http://mimir-.../api/v1/push
        headers:
          Authorization: Bearer my-token-here
          X-Scope-OrgID: my-org-id
        max_batch_size_bytes: 30000000
        tls:
          insecure_skip_verify: true

Also, when trying to use the otlphttp exporter I'm getting a 499.

I've tried both exporters to send into Mimir.

UPDATE

I have solved this for the prometheusremotewrite exporter by simplifying the config:

    exporters:
      prometheusremotewrite:
        endpoint: http://mimir-.../api/v1/push
        headers:
          Authorization: Bearer my-token-here
        tls:
          insecure_skip_verify: true

gauravphagrehpe · 2024-08-28T11:07:14Z

for me its worked by changing

from this:
`tls:

insecure: true`

to this:
`tls:

insecure_skip_verify: true`

Priyanka-src · 2024-09-05T15:34:50Z

I am having similar issue as:
error exporterhelper/common.go:95 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: context deadline exceeded", "dropped_items": 5}
I am using adot-collector as my pod, in AWS. Any idea what can be done. Any help would be appreciated.

Shindek77 · 2024-09-09T06:08:10Z

same for me..please help

frakev added bug Something isn't working needs triage New item requiring triage labels Mar 22, 2024

github-actions bot added the exporter/prometheusremotewrite label Mar 22, 2024

github-actions bot mentioned this issue Mar 26, 2024

Weekly Report: 2024-03-19 - 2024-03-26 #31947

Closed

This was referenced Apr 2, 2024

Weekly Report: 2024-03-26 - 2024-04-02 #32082

Closed

Weekly Report: 2024-04-02 - 2024-04-09 #32230

Closed

github-actions bot mentioned this issue Apr 16, 2024

Weekly Report: 2024-04-09 - 2024-04-16 #32407

Closed

roopeshsn mentioned this issue May 14, 2024

Added support for OTEL Collector l3af-project/l3af-arch#106

Merged

github-actions bot mentioned this issue May 28, 2024

Weekly Report: 2024-05-21 - 2024-05-28 #33243

Closed

github-actions bot added the Stale label Jun 14, 2024

github-actions bot removed the Stale label Jun 19, 2024

LucaLanziani mentioned this issue Jun 19, 2024

Weekly Report: 2024-06-12 - 2024-06-19 LucaLanziani/opentelemetry-collector-contrib#10

Closed

github-actions bot mentioned this issue Jul 2, 2024

Weekly Report: 2024-06-25 - 2024-07-02 #33839

Open

github-actions bot mentioned this issue Jul 9, 2024

Weekly Report: 2024-07-02 - 2024-07-09 #33962

Open

This was referenced Jul 16, 2024

Weekly Report: 2024-07-09 - 2024-07-16 #34087

Open

Weekly Report: 2024-07-16 - 2024-07-23 #34202

Open

This was referenced Jul 30, 2024

Weekly Report: 2024-07-23 - 2024-07-30 #34301

Open

Weekly Report: 2024-07-30 - 2024-08-06 #34410

Open

This was referenced Aug 13, 2024

Weekly Report: 2024-08-06 - 2024-08-13 #34626

Open

Weekly Report: 2024-08-13 - 2024-08-20 #34743

Open

github-actions bot added the Stale label Aug 26, 2024

dashpole removed Stale needs triage New item requiring triage labels Aug 26, 2024

dashpole mentioned this issue Aug 26, 2024

[exporter/prometheusremotewrite] Permanent error: Permanent error: context deadline exceeded #32511

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prometheusremotewrite context deadline exceeded #31910

prometheusremotewrite context deadline exceeded #31910

frakev commented Mar 22, 2024

github-actions bot commented Mar 22, 2024

martinohansen commented Apr 10, 2024

martinohansen commented Apr 10, 2024

frakev commented Apr 11, 2024

dhilgarth commented Apr 14, 2024

dhilgarth commented Apr 14, 2024

github-actions bot commented Jun 14, 2024

koenvandesande commented Jun 18, 2024

juniorsdj commented Jun 26, 2024

github-actions bot commented Aug 26, 2024

dashpole commented Aug 26, 2024

cabrinha commented Aug 27, 2024 •

edited

Loading

gauravphagrehpe commented Aug 28, 2024 •

edited

Loading

Priyanka-src commented Sep 5, 2024

Shindek77 commented Sep 9, 2024

prometheusremotewrite context deadline exceeded #31910

prometheusremotewrite context deadline exceeded #31910

Comments

frakev commented Mar 22, 2024

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Mar 22, 2024

martinohansen commented Apr 10, 2024

martinohansen commented Apr 10, 2024

frakev commented Apr 11, 2024

dhilgarth commented Apr 14, 2024

dhilgarth commented Apr 14, 2024

github-actions bot commented Jun 14, 2024

koenvandesande commented Jun 18, 2024

juniorsdj commented Jun 26, 2024

github-actions bot commented Aug 26, 2024

dashpole commented Aug 26, 2024

cabrinha commented Aug 27, 2024 • edited Loading

gauravphagrehpe commented Aug 28, 2024 • edited Loading

Priyanka-src commented Sep 5, 2024

Shindek77 commented Sep 9, 2024

cabrinha commented Aug 27, 2024 •

edited

Loading

gauravphagrehpe commented Aug 28, 2024 •

edited

Loading