Unable to send metrics using googlecloudexporter to custom endpoint with incorrect errors

[Mona3087]

Component(s)

exporter/googlecloud

What happened?

Description

We are using otel collector to collect custom metrics from our Kubernetes cluster and export it to Google Cloud Monitoring for alerts. When have lots of pods in our cluster, our clusters could be generating more than 200k time series per minute. Otel at that point starts logging error and is not able to push metrics in cloud. The error is talking about hitting 200 quota limit and it is misleading. Ref: https://cloud.google.com/monitoring/quotas

Steps to Reproduce

Create sample pod/app that pushes more than 200k time series on single monitored resource type.

Expected Result

Error should specify that which quota limit otel exporter is hitting.

Actual Result

2024-11-19T17:18:39.278Z error internal/queue_sender.go:92 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = ResourceExhausted desc = One or more TimeSeries could not be written: timeSeries[0-53]: write for resource=k8s_cluster{location:us-east1,cluster_name:ctl-gke-4501} failed with: Monitored resource has too many time series (custom metrics).\nerror details: name = Unknown desc = total_point_count:54 errors:{status:{code:8} point_count:54}", "dropped_items": 54}

Collector version

v0.114.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")
GKE 1.31.1-gke.2105000

OpenTelemetry Collector configuration

processors: 
  batch:       
    send_batch_size: 200       
    send_batch_max_size: 200
    timeout: 200ms
  memory_limiter:      
    check_interval: 1s       
    limit_mib: 1600       
    spike_limit_mib: 320
exporters:    
  googlecloud:      
    project: ""      
    metric:       
      prefix: "custom.googleapis.com"
service:  
  pipelines:    
    metrics:     
      recievers: []    
      processors: [batch, memory_limiter]     
      exporters: [googlecloud]

Log output

2024-11-19T17:18:39.278Z	error	internal/queue_sender.go:92	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = ResourceExhausted desc = One or more TimeSeries could not be written: timeSeries[0-53]: write for resource=k8s_cluster{location:us-east1,cluster_name:ctl-gke-4501} failed with: Monitored resource has too many time series (custom metrics).\nerror details: name = Unknown  desc = total_point_count:54  errors:{status:{code:8}  point_count:54}", "dropped_items": 54}

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/googlecloud: @aabmass @dashpole @jsuereth @punya @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[psx95]

Hi @Mona3087,

The error message you posted:
ResourceExhausted desc = One or more TimeSeries could not be written: timeSeries[0-53]: write for resource=k8s_cluster{location:us-east1,cluster_name:ctl-gke-4501} failed with: Monitored resource has too many time series (custom metrics)

Does inform that Active time series from custom metrics per monitored resources quota listed on the quotas page is the one that is being resource exhausted.

The error mentions that it is for custom metrics which distinguishes it from the other kinds of active TimeSeries quota.

Could you mention what was your expected output/error message ?