"Response failed." with "Invalid JSON payload received." for gcp_stackdriver_metrics

[tiny-tinker]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm getting a "Response failed." message in my vector log when using the gcp_stackdriver_metrics sink. The source is just host_metrics and vector is running on an old laptop running debian stretch. (I also had the same issue running a similar config on a GCP compute instance.)

I can't tell what the body is in the log output above. Seems that even in debug and trace it's listed as body=[unknown]. The REST API docs don't seem talk about having the "host" or "collector" fields as inputs.

Help?

Configuration

my `vector.toml` file:


# Random Syslog-formatted logs
[sources.dummy_logs]
type = "demo_logs"
format = "syslog"
interval = 1

# Parse Syslog logs
# See the Vector Remap Language reference for more info: https://vrl.dev
[transforms.parse_logs]
type = "remap"
inputs = ["dummy_logs"]
source = '''
. = parse_syslog!(string!(.message))
'''

# Print parsed logs to stdout
[sinks.print]
type = "console"
inputs = ["parse_logs"]
encoding.codec = "json"


[sources.my_host_metrics]
type = "host_metrics"
collectors = [
  "cgroups",
  "cpu",
  "disk",
  "filesystem",
  "load",
  "host",
  "memory",
  "network"
]
namespace = "host"
scrape_interval_secs = 15

[sinks.my_gcp_snk]
type = "gcp_stackdriver_metrics"
inputs = [ "my_host_metrics" ]
credentials_path = "/etc/vector/my_vm_host-creds.json"
project_id = "my-sweet-prj"

  [sinks.my_gcp_snk.resource]
  type = "global"
  projectId = "my-other-prj"
  instanceId = "my_vm_host"
  zone = "us-central1-a"

Version

vector 0.23.3 (x86_64-unknown-linux-gnu af8c9e1 2022-08-10)

Debug Output

Full log here: https://gist.github.com/tiny-tinker/1337bcabc1e456c1f2e76ce4fde93769

Relevant snippet:

2022-08-23T03:14:11.959058Z ERROR sink{component_kind="sink" component_id=my_gcp_snk component_type=gcp_stackdriver_metrics component_name=my_gcp_snk}:request{request_id=0}: vector::sinks::util::sink: Response failed. response=Response { status: 400, version: HTTP/1.1, headers: {"vary": "X-Origin", "vary": "Referer", "vary": "Origin,Accept-Encoding", "content-type": "application/json; charset=UTF-8", "date": "Tue, 23 Aug 2022 03:14:11 GMT", "server": "ESF", "cache-control": "private", "x-xss-protection": "0", "x-frame-options": "SAMEORIGIN", "x-content-type-options": "nosniff", "alt-svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\"", "accept-ranges": "none", "transfer-encoding": "chunked"}, body: b"{\n  \"error\": {\n    \"code\": 400,\n    \"message\": \"Invalid JSON payload received. Unknown name \\\"cpu\\\" at 'time_series[0].metric': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"mode\\\" at 'time_series[0].metric': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"collector\\\" at 'time_series[0].metric': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"host\\\" at 'time_series[0].metric': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"instanceId\\\" at 'time_series[0].resource': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"projectId\\\" at 'time_series[0].resource': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"zone\\\" at 'time_series[0].resource': Cannot find field.\",\n    \"status\": \"INVALID_ARGUMENT\",\n    \"details\": [\n      {\n        \"@type\": \"type.googleapis.com/google.rpc.BadRequest\",\n        \"fieldViolations\": [\n          {\n            \"field\": \"time_series[0].metric\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"cpu\\\" at 'time_series[0].metric': Cannot find field.\"\n          },\n          {\n            \"field\": \"time_series[0].metric\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"mode\\\" at 'time_series[0].metric': Cannot find field.\"\n          },\n          {\n            \"field\": \"time_series[0].metric\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"collector\\\" at 'time_series[0].metric': Cannot find field.\"\n          },\n          {\n            \"field\": \"time_series[0].metric\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"host\\\" at 'time_series[0].metric': Cannot find field.\"\n          },\n          {\n            \"field\": \"time_series[0].resource\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"instanceId\\\" at 'time_series[0].resource': Cannot find field.\"\n          },\n          {\n            \"field\": \"time_series[0].resource\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"projectId\\\" at 'time_series[0].resource': Cannot find field.\"\n          },\n          {\n            \"field\": \"time_series[0].resource\",\n            \"description\": \"Invalid JSON payload received. Unknown name \\\"zone\\\" at 'time_series[0].resource': Cannot find field.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n" }

Example Data

I'm not sure. It seems to happen with any of the metric sources and the gcp_stackdriver_metrics sink.

Additional Context

Brand new install. The issue occurred on a debian bullseye gce instance as well as a debian stretch laptop.

References

No response

[jszwedko]

Combining this with #14890