-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
origin-node excessive logging unable to clean up volumes / orphaned pods #13111
Comments
I was able to stop the overly excessive logging by the origin-node following the comment by @gnufied kubernetes/kubernetes#38498 (comment)
In my case, the volumes dir had contents and moving it and creating an empty volumes dir was the missing key. Just an after thought just being done recovering a couple of our infra-nodes after Docker stopped responding. Appreciate everyones' hard work and passion from the community. I look forward putting this one behind us. It's a tough one to deal with. Cheers! |
Good afternoon @gnufied! Would you be able to provide any suggestions by means to manually clean up orphaned pods and stop the repeated logs? Earlier I reported that manually moving the volumes directory inside the pod volume and creating an empty volume would clean up orphaned pods. This approach does not appear to be the fix for every single occurrence. Sincerely, |
@canit00 thanks for bug report. I am evaluating this from bunch of angles. Just to confirm - the directories which are not getting cleaned up, are they mounted volumes like one of the commenter reported in kubernetes bug or are they plain directories? If they are mounted directories we have to reconsider our approach a bit because it will require unmounting the volume first |
…rphanedPodDirs This is basically a cherry-pick of: 1. kubernetes/kubernetes#38909 2. https://github.com/kubernetes/kubernetes/pull/41626/files But those commits can't be directly cherry picked because of whole lot of dependencies. Fixes openshift#13111
Issues go stale after 90d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle stale |
Stale issues rot after 30d of inactivity. Mark the issue as fresh by commenting If this issue is safe to close now please do so with /lifecycle rotten |
Rotten issues close after 30d of inactivity. Reopen the issue by commenting /close |
Troubleshooting a separate issue we noticed excessive logging on three particular nodes that have been designated to run infra services.
Excessive logging is causing minor issues such as inability to capture logging via sosreport, however more concerning is not being able to address the root of these events.
Version
oc v1.4.1+3f9807a
kubernetes v1.4.0+776c994
openshift v1.4.1+3f9807a
kubernetes v1.4.0+776c994
Steps To Reproduce
We have an rsyslog daemon set and ipf-routers running on these hosts. Simply rebooting
the systems causes this to occur.
Current Result
Excessive logging from origin-node from what appears to be unable to remove volumes left behind by orphan pods.
Example log entry:
Volumes can be seen not mounted and still exist in on disk.
Expected Result
Additional Information
New Containerized Origin cluster running on Atomic Host 7.3 7.3.2 receiving approximately 2k messages a minute on three designated infra nodes.
Others are experiencing the same issue and been reported kubernetes/kubernetes#38498
I tried exporting journald logs for origin-node just for today and it exceeded a couple of gigs.
Looking for guidance how to go about normalizing this situation and how to provide more detailed logs.
logs.pdf
The text was updated successfully, but these errors were encountered: