gcp_stackdriver_metrics makes invalid calls to GCP monitoring API

[mloughran]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

The code (https://github.com/vectordotdev/vector/blob/master/src/sinks/gcp/stackdriver_metrics.rs) is posting to the timeSeries create endpoint (https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/create), which expects a MonitoredResource, like this:

{ "type": "gce_instance",
  "labels": { "project_id": "my-project",
              "instance_id": "12345678901234",
              "zone": "us-central1-a" }}

However the error response below suggests that project_id, instance_id, and zone is getting flattened by https://github.com/vectordotdev/vector/blob/master/src/sinks/gcp/stackdriver_metrics.rs#L222. My rust is nonexistent, but could this be the cause: https://github.com/vectordotdev/vector/blob/master/src/sinks/gcp/mod.rs#L28 ?

Configuration

[sources.statsd]
  type = "statsd"
  address = "0.0.0.0:8125"
  mode = "udp"

[sinks.gcp_metrics]
  type = "gcp_stackdriver_metrics"
  inputs = ["statsd"]
  project_id = "<gcp_project>"

[sinks.gcp_metrics.resource]
  type = "gce_instance"
  instance_id = "<instance_id>"
  zone = "<zone>"

Version

vector 0.24.2 (x86_64-unknown-linux-gnu 5e47b89 2022-10-13)

Debug Output

I get the following error:

2022-10-19T21:22:47.823868Z ERROR sink{component_kind="sink" component_id=gcp_metrics component_type=gcp_stackdriver_metrics component_name=gcp_metrics}:request{request_id=0}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 400 Bad Request"
2022-10-19T21:22:47.824090Z ERROR sink{component_kind="sink" component_id=gcp_metrics component_type=gcp_stackdriver_metrics component_name=gcp_metrics}:request{request_id=0}: vector::sinks::util::sink: Response failed. response=Response { status: 400, version: HTTP/1.1, headers: {"vary": "X-Origin", "vary": "Referer", "vary": "Origin,Accept-Encoding", "content-type": "application/json; charset=UTF-8", "date": "Wed, 19 Oct 2022 21:22:47 GMT", "server": "ESF", "cache-control": "private", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "accept-ranges": "none", "transfer-encoding": "chunked"}, body: b"{\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"Invalid JSON payload received. Unknown name \\\"project_id\\\" at 'time_series[0].resource': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"instance_id\\\" at 'time_series[0].resource': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"zone\\\" at 'time_series[0].resource': Cannot find field.\",\n    \"status\": \"INVALID_ARGUMENT\",\n    \"details\": [\n      {\n        \"@type\": \"type.googleapis.com/google.rpc.BadRequest\",\n        \"fieldViolations\": [\n          {\n            \"field\": \"time_series[0].resource\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"project_id\\\" at 'time_series[0].resource': Cannot find field.\"\n          },\n          {\n            \"field\": \"time_series[0].resource\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"instance_id\\\" at 'time_series[0].resource': Cannot find field.\"\n          },\n          {\n            \"field\": \"time_series[0].resource\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"zone\\\" at 'time_series[0].resource': Cannot find field.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n" }

Extracting the GCP error response from the above log:

{
  "error": {
    "code": 400,
    "message": "Invalid JSON payload received. Unknown name \"project_id\" at 'time_series[0].resource': Cannot find field.\nInvalid JSON payload received. Unknown name \"instance_id\" at 'time_series[0].resource': Cannot find field.\nInvalid JSON payload received. Unknown name \"zone\" at 'time_series[0].resource': Cannot find field.",
    "status": "INVALID_ARGUMENT",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.BadRequest",
        "fieldViolations": [
          {
            "field": "time_series[0].resource",
            "description": "Invalid JSON payload received. Unknown name \"project_id\" at 'time_series[0].resource': Cannot find field."
          },
          {
            "field": "time_series[0].resource",
            "description": "Invalid JSON payload received. Unknown name \"instance_id\" at 'time_series[0].resource': Cannot find field."
          },
          {
            "field": "time_series[0].resource",
            "description": "Invalid JSON payload received. Unknown name \"zone\" at 'time_series[0].resource': Cannot find field."
          }
        ]
      }
    ]
  }
}

Example Data

No response

Additional Context

No response

References

#14057

[neuronull]

This was fixed by #16394