Skip to content

Commit

Permalink
fix agent stuck on drain bug
Browse files Browse the repository at this point in the history
Some customers are using StatefulSet which makes pod's name unchangeable,
and we use the name of the evicted pod and query the k8s API on wait_for_deletion
fucntion, waiting for it to 404 before declaring a pod deleted.
Therefore, current logic can't handle this sepcial case. We add a new if
statement condition to compare two pods by uid to make sure if they are the
same pod. If they are same pod, brupop should wait until it be deleted,
otherwise we should recognize the old pod has already been deleted and
re-lives on other node.
  • Loading branch information
gthao313 committed Apr 5, 2022
1 parent 327c868 commit fbcea3d
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions models/src/node/drain.rs
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,12 @@ async fn wait_for_deletion(k8s_client: &kube::Client, pod: &Pod) -> Result<(), e
event!(Level::INFO, "Pod {} deleted.", pod.name(),);
break;
}

Ok(p) if p.uid() != pod.uid() => {
event!(Level::INFO, "Pod {} deleted.", p.name(),);
break;
}

Ok(_) => {
event!(
Level::DEBUG,
Expand Down

0 comments on commit fbcea3d

Please sign in to comment.