Fix: avoid redundant logs on failures to export metrics

[jjti]

Description

I'm seeing redundant log lines on failures to push metrics to HCP:

2024-02-07T14:22:37.358Z [ERROR] agent.hcp.hcp_telemetry_client: request failed: error="Post "https://consul-telemetry.hcp.dev/otlp/v1/metrics\": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error" method=POST url=https://consul-telemetry.hcp.dev/otlp/v1/metrics

2024/02/07 14:22:37 failed to export metrics: failed to post metrics: POST https://consul-telemetry.hcp.dev/otlp/v1/metrics giving up after 1 attempt(s): Post "https://consul-telemetry.hcp.dev/otlp/v1/metrics": http2: Transport: cannot retry err [http2: Transport received Server's graceful shutdown GOAWAY] after Request.Body was written; define Request.GetBody to avoid this error

We log in two places:

• here in the go-retryablehttp client: https://github.com/hashicorp/go-retryablehttp/blob/4165cf8897205a879a06b20d1ed0a2a76fbb6a17/client.go#L669
• here in the metrics exporter:

  consul/agent/hcp/client/metrics_client.go

  Line 75 in 47c5c8b

  return fmt.Errorf("failed to post metrics: %w", err)

The fix here is to pass a NullLogger to the http client so:

• we do not get redundant log lines on errors
• we get the log lines with the name/group that we use elsewhere. After the other logging change this will be the hcp.telemetry group logs: Fix logging when we fail to export metrics to hcp #20514

An alternative approach is to combine this PR with the other logging-related PR: #20514. And:

• set the otel global logger to a no-op logger that does nothing
• also disable logging in the go-retryablehttp client (NullLogger as in this PR)
• only log in ExportMetrics

Testing & Reproduction steps

Links

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

[hc-github-team-consul-core]

Backport failed @jjti. Run: https://github.com/hashicorp/consul/actions/runs/7832377374

[hc-github-team-consul-core]

Backport failed @jjti. Run: https://github.com/hashicorp/consul/actions/runs/7832377374

[hc-github-team-consul-core]

@jjti, a backport is missing for this PR [20519] for versions [1.16,1.17] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

[hc-github-team-consul-core]

@jjti, a backport is missing for this PR [20519] for versions [1.16,1.17] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

[hc-github-team-consul-core]

@jjti, a backport is missing for this PR [20519] for versions [1.16,1.17] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

[hc-github-team-consul-core]

@jjti, a backport is missing for this PR [20519] for versions [1.16,1.17] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>