fix agent stuck on drain bug

[gthao313]

Issue number:
#168

Description of changes:

    fix agent stuck on drain bug

    Some customers are using StatefulSet which makes pod's name unchangeable,
    and we use the name of the evicted pod and query the k8s API on wait_for_deletion
    fucntion, waiting for it to 404 before declaring a pod deleted.
    Therefore, current logic can't handle this sepcial case. We add a new if
    statement condition to compare two pods by uid to make sure if they are the
    same pod. If they are same pod, brupop should wait until it be deleted,
    otherwise we should recognize the old pod has already been deleted and
    re-lives on other node.

Testing done:

1. Apply StatefulSet to cluster

NAME    READY   STATUS    RESTARTS   AGE
web-0   1/1     Running   0          5m32s
web-1   1/1     Running   0          10m
web-2   1/1     Running   0          5m22s


Name:         web-0
Namespace:    default
Priority:     0
Node:         ip-192-168-156-161.us-west-2.compute.internal/192.168.156.161
....

2. Run brupop integration test and monitor update

NAME                                                STATE   VERSION   TARGET STATE   TARGET VERSION   CRASH COUNT
brs-ip-192-168-128-161.us-west-2.compute.internal   Idle    1.7.0     Idle                            0
brs-ip-192-168-144-77.us-west-2.compute.internal    Idle    1.7.0     Idle                            0
brs-ip-192-168-156-161.us-west-2.compute.internal   Idle    1.7.0     Idle                            0

apiserver log

[EVICT_POD - EVENT] Attempting to evict pod web-0",
[EVICT_POD - EVENT] Successfully evicted Pod 'web-0'"
[WAIT_FOR_DELETION - EVENT] Pod web-0 deleted.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[cbgbt]

nit:

I'd suggest using "match guards" here to keep the indentation level as before. e.g.

Err(kube::Error::Api(e)) if e.code == 404 => {...},
Ok(p) if p.uid() != pod.uid() => {...}
Ok(p) => {...},