-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
logSnippet error reading termination message from logs #15878
Comments
can you share one of your build json objects? |
I tested this with "oc cluster up --version=v3.6.0" and created a build w/ a non-existent repository name and it seems to work ok:
so i'd like to know your exact cluster version and also the full build yaml so i can see why your build failed in case it failed in a way that we'd be unable to get a log snippet for. |
I've seen this happen as well, could be a race condition between pod shutdown and container log cleanup. Or the container exits so fast no logs are reported. |
if it's that one it sounds like something that should be an upstream k8s issue since it means the requested termination logs api isn't really being respected, no?
@andrewklau can you tell us if you're able to manually retrieve logs for the build or build pod when this happens? |
yeah I was able to get the logs of the failed build, just not from the snippet. I'll get you the build yaml when I'm back. |
I think this one failed due to oom, but hard to tell. Increasing the mem did fix it though.
|
but you were able to get the build/pod logs for it? can you share the build pod json? in particular i'm interested in the pod.Status.ContainerStatuses.[container].State.Terminated.Message That is the field we are retrieving. If it's not set correctly, then it seems like a k8s bug. |
Yeah the error is in the pod json too. I was able to get the logs at the time of the error. However the logs are now gone since the host has already cleaned up the container.
|
On the node /var/log/pods is empty although the folders exist:
|
ok, given that this appears in the pod itself, this is a k8s issue w/ the last termination log feature and needs to be opened as an upstream issue. |
Upstream issue filed here: |
Automatic merge from submit-queue. UPSTREAM: 52503: Get fallback termination msg from docker when using journald log driver xref kubernetes/kubernetes#52503 From the commit message: > When using the legacy docker container runtime and when a container has > terminationMessagePolicy=FallbackToLogsOnError and when docker is > configured with a log driver other than json-log (such as journald), > the kubelet should not try to get the container's log from the > json log file (since it's not there) but should instead ask docker for > the logs. fixes #15878
None of my failed builds ever seem to get this new logSnippet, it always returns:
Version
3.6
The text was updated successfully, but these errors were encountered: