Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout error logs #545

Closed
atrinh9f opened this issue Aug 7, 2023 · 5 comments · Fixed by #638
Closed

Timeout error logs #545

atrinh9f opened this issue Aug 7, 2023 · 5 comments · Fixed by #638
Labels
custom-metrics-stackdriver-adapter Issues for custom-metrics-stackdriver-adapter

Comments

@atrinh9f
Copy link

atrinh9f commented Aug 7, 2023

Getting the following errors, but the service seems to be functional. Any idea on how I can fix/silence the logs?

`
[resource.labels.containerName: pod-custom-metrics-stackdriver-adapter] post-timeout activity - time-elapsed: 202.178365ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result:

[resource.labels.containerName: pod-custom-metrics-stackdriver-adapter] apiserver was unable to write a fallback JSON response: http: Handler timeout

[resource.labels.containerName: pod-custom-metrics-stackdriver-adapter] post-timeout activity - time-elapsed: 24.450736ms, GET "/apis/custom.metrics.k8s.io/v1beta2" result:
`

Running: v0.12.2-gke.0

If I upgrade to latest, the pod won't run / runs out of memory.

@hermanbanken
Copy link

We were also wondering if this is problematic. We see timeout errors too:

E0829 10:04:54.328637       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="77c939f4-30d8-4219-b8a6-2910bdb74f57"
E0829 10:04:54.328790       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta2" audit-ID="e1d09e99-fadc-4db0-9804-dfd1eec26aa3"
E0829 10:04:54.328828       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta1" audit-ID="3c7f8a5e-2505-4074-93cf-2b7d26e27777"
E0829 10:04:54.328863       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta2" audit-ID="b3224ed1-2f5a-46e7-a578-134157e08d95"
E0829 10:04:54.329106       1 writers.go:117] apiserver was unable to write a JSON response: http2: stream closed
E0829 10:04:54.329193       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta2" audit-ID="f42ca820-96c8-4261-acc0-644c2c22ce81"
E0829 10:04:54.329243       1 writers.go:111] apiserver was unable to close cleanly the response writer: http: Handler timeout
E0829 10:04:54.329751       1 writers.go:111] apiserver was unable to close cleanly the response writer: http: Handler timeout
E0829 10:04:54.329784       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0829 10:04:54.331263       1 writers.go:111] apiserver was unable to close cleanly the response writer: http: Handler timeout
E0829 10:04:54.331324       1 writers.go:130] apiserver was unable to write a fallback JSON response: http2: stream closed
E0829 10:04:54.332603       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0829 10:04:54.334141       1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/custom.metrics.k8s.io/v1beta2" audit-ID="f7580d3e-884a-4731-81d0-ac52aa3aa804"
E0829 10:04:54.334428       1 writers.go:111] apiserver was unable to close cleanly the response writer: http: Handler timeout
E0829 10:04:54.335826       1 writers.go:111] apiserver was unable to close cleanly the response writer: http: Handler timeout
E0829 10:04:54.340580       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}: http2: stream closed
E0829 10:04:54.342783       1 timeout.go:135] post-timeout activity - time-elapsed: 14.116501ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0829 10:04:54.343848       1 timeout.go:135] post-timeout activity - time-elapsed: 15.001601ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>
E0829 10:04:54.344944       1 timeout.go:135] post-timeout activity - time-elapsed: 16.130327ms, GET "/apis/custom.metrics.k8s.io/v1beta2" result: <nil>
E0829 10:04:54.346118       1 timeout.go:135] post-timeout activity - time-elapsed: 71.758936ms, GET "/apis/custom.metrics.k8s.io/v1beta2" result: <nil>
E0829 10:04:54.347092       1 writers.go:130] apiserver was unable to write a fallback JSON response: http2: stream closed
E0829 10:04:54.349250       1 timeout.go:135] post-timeout activity - time-elapsed: 15.089807ms, GET "/apis/custom.metrics.k8s.io/v1beta2" result: <nil>
E0829 10:04:54.350332       1 timeout.go:135] post-timeout activity - time-elapsed: 21.444076ms, GET "/apis/custom.metrics.k8s.io/v1beta2" result: <nil>
E0829 10:04:54.351409       1 writers.go:130] apiserver was unable to write a fallback JSON response: http2: stream closed
E0829 10:04:54.357014       1 timeout.go:135] post-timeout activity - time-elapsed: 82.632572ms, GET "/apis/custom.metrics.k8s.io/v1beta1" result: <nil>

@hermanbanken
Copy link

--v=-1 does not seem to mute anything 😞😢

@strimpak
Copy link

strimpak commented Nov 2, 2023

I am seeing the same issue with v.0.13.1

@CatherineF-dev
Copy link
Contributor

CatherineF-dev commented Jan 5, 2024

Updated: For OOM in latest version, could you try v0.14.2?

@CatherineF-dev
Copy link
Contributor

Could you try v0.14.2?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
custom-metrics-stackdriver-adapter Issues for custom-metrics-stackdriver-adapter
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants