gcp_stackdriver_logs: 401 Unauthorised each hour

[garethpelly]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Each ~hour we are observing "Http status: 401 Unauthorized" in our vector logs coming from the gcp_stackdriver_logs sink.
Vector is running in GKE as a headless service and consuming logs from kafka, we do not utilise the credentials_path and instead rely on the Service Account to authenticate. The problem resolves itself after a 2/3 minute period however my understanding is that these logs are not retried and are therefore discarded.

Configuration

type = "gcp_stackdriver_logs"
inputs = ["final_cleanup"]
log_id = "{{ type }}"
project_id = "centralized-logging"
severity_key = "log_level"

batch.max_events = 1000
batch.max_bytes = 9900000

resource.type = "{{ resource_type }}"
resource.project_id = "{{ gcp_project_id }}"
resource.instance_id = "{{ hostname }}"

Version

0.34.1-distroless-libc

Debug Output

No response

Example Data

No response

Additional Context

No response

References

• gcp_stackdriver_logs: 401 Unauthorized (ACCESS_TOKEN_EXPIRED) #17559
• Vector agent stops watching logs from new pods #8616

[jszwedko]

You are correct, it seems like those requests are not retried. I'd argue they should be (per #10870), in addition to refreshing the token before it expires.

Retry logic:

vector/src/sinks/util/http.rs

Lines 517 to 531 in 131ab45

	fn should_retry_response(&self, response: &T) -> RetryAction {
	let status = (self.func)(response);
	match status {
	StatusCode::TOO_MANY_REQUESTS => RetryAction::Retry("too many requests".into()),
	StatusCode::NOT_IMPLEMENTED => {
	RetryAction::DontRetry("endpoint not implemented".into())
	}
	_ if status.is_server_error() => {
	RetryAction::Retry(format!("Http Status: {}", status).into())
	}
	_ if status.is_success() => RetryAction::Successful,
	_ => RetryAction::DontRetry(format!("Http status: {}", status).into()),
	}
	}

[garethpelly]

Coming back to this, it seems as though the root issue relates to running more than 1 gcp_stackdriver_logs sink (we had a separate sink sending a subset of logs to a different GCP project).
Vector's handling of the authentication token refreshes seems to (perhaps) have a timing/race issue when more than one sink is in play, when we removed the additional sink the 401s were no longer observed.

[garethpelly]

Update: The 401s have returned since we scaled back to a single gcp_stackdriver_logs sink.

[garethpelly]

@jszwedko I've taken a stab at changing how the token is refreshed in #20574.

[garethpelly]

Closing. Fixed in #20574