Send metrics from vector to prometheus hosted on cloud

[uni-pooja-laad]

Hi team,
I am trying to send metrics from vector to Prometheus which is on cloud. Below is the configuration I am using:
transforms:
aggregate_metrics:
type: aggregate
inputs: [datadog_agent.metrics]
interval_ms: 10000
sinks:
prom_metrics:
type: prometheus_remote_write
inputs: [aggregate_metrics]
endpoint: "https://prometheus-prod-.grafana.net"
auth:
strategy: basic
user: ""
password:
The error here is : 2024-11-08T06:59:19.740100Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Unexpected status: 405 Method Not Allowed component_kind="sink" component_type="prometheus_remote_write" component_id=prom_metrics
Even I tried by adding suffix like: /api/prom/push or /api/v1/write etc. but seems its either 400 or 405 error.

[jszwedko]

Hi @uni-pooja-laad !

I think you are hitting this bug: #8279. For now, I'd suggest disabling the healthcheck for that sink:

healthcheck:
  enabled: false

[uni-pooja-laad]

yes I am using correct URL which is provided by grafana integration page.
Above I might have cropped it.
But this is the exact URL I am using https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom.

[uni-pooja-laad]

I am getting metrics from datadog_agent and using vector as an aggregator. I want to send metrics to the prometheus endpoint which is hosted on cloud. I did same task to send logs using loki sink and it is working fine. But not sure if prometheus need any extra transform or exporter.

[jszwedko]

I think you want /api/prom/push based on https://grafana.com/docs/grafana-cloud/send-data/alloy/reference/components/prometheus/prometheus.remote_write/#send-metrics-to-a-managed-service

[uni-pooja-laad]

Tried that as well:
https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push
Error is :
2024-11-15T15:28:59.607707Z ERROR sink{component_kind="sink" component_id=prom_metrics component_type=prometheus_remote_write}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the re
quest. reason="Http status: 400 Bad Request" internal_log_rate_limit=true

[jszwedko]

Ah, strange. Do you see any logs in Grafana Cloud indicating why the incoming request was considered invalid? There could be some clues there. I believe the path you have is now correct.