Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect ip to namespace table for pods in a failed state. #95

Closed
wants to merge 1 commit into from

Conversation

corlettb
Copy link

@corlettb corlettb commented Sep 7, 2017

We run a number of cronjobs in our environment.

These pods finish running in a Complete or Failed state and linger for sometime. Unfortunately kubernetes leaves the PodIP intact that was used for these pods.

If docker reuses these ips then we have a conflict and this leads to ips being flagged against the wrong namespace. This breaks the restricted namespace functionality.

This change restricts the namespace tracking to pods in a Running phase.

This might not be a complete solution to this issue but maybe it can start a dialogue.

These pods finish running in a Complete or Failed state and linger for sometime. Unfortunately kubernetes leaves the PodIP intact that was used for these pods.

If docker reuses these ips then we have a conflict and this leads to ips being flagged against the wrong namespace.

This change restricts the namespace tracking to pods in a Running phase.

This might not be a complete solution to this issue.
@coveralls
Copy link

Coverage Status

Changes Unknown when pulling cf7e76e on corlettb:master into ** on jtblin:master**.

@SleepyBrett
Copy link
Contributor

I think I might be seeing this even with cronjob pods that aren't failed but completed. I'm going to evaluate this as a possible solution.

@jrnt30
Copy link
Collaborator

jrnt30 commented Sep 9, 2017

@SleepyBrett That does happen. We have a working set that changes the indexing pattern that is build with the :dev tag. @jtblin has been looking for individuals to test it.

We have been running it for several weeks now without these issues.

@jtblin
Copy link
Owner

jtblin commented Sep 11, 2017

Yes #92 should fix these issues so I don't think we're going to need this PR. Going to release #92 soon now.

@jtblin
Copy link
Owner

jtblin commented Sep 18, 2017

#92 is now released as 0.8.0 and should solve this issue so closing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants