DEVOPS-7803 restore log stream #2903
Conversation
Re-establish connection while the pod is still running
| plo := &corev1.PodLogOptions{ | ||
| Follow: true, | ||
| Container: containerName, | ||
| } | ||
| if sinceNow { |
There was a problem hiding this comment.
Should it be sinceNow? Maybe we can somehow figure out the time when previous log ended?
I'm thinking if we can lose some logs this way. Probably having duplicate logs and deduplicating on log parsing is better than potentially losing some. WDYT?
There was a problem hiding this comment.
Even though this solution will work in some (maybe even in most of the cases), it would be better to have something more robust.
The io.Reader returned by StreamPodLogs could be wrapped, and the last X chunks of data read, could be stored in buffer, with timestamp when it was read.
Then, when reader will be closed with EOF, streaming could be restarted since last seen data (maybe even 10 seconds before). Then everything up to seen data (including it) needs to be skipped.
In such case we will have confidence that we won't skip any output line.
There was a problem hiding this comment.
Sorry for closing, I had a misunderstanding in my organization. I opened a new PR for follow-up action.
In the new PR, I have already taken this comment into account. We store the time of the last successful read, and after EOF, we request logs from that time. During tests, it shows that kubelet will always return logs starting with the last read line (even after a few hours). This always leads to duplication, never a loss of information.
Change Overview
Unfortunately, there is a hardcoded timeout in the kubelet server. If we want to support long-lasting phases, we need to work around it.
This PR introduces re-establishing the log stream connection while the pod is still running.
Pull request type
Please check the type of change your PR introduces:
Issues
Test Plan
I created a phase executing function
KubeTask, with a long-lasting command (sleep). I ensured that the logs were read until the end of its execution and that the output was properly gathered.