Replicator crashes when watches expire

[Bowbaq]

When watches expire (and possibly at other times), the pod crashes because it cannot write to a log file. I believe this is related to kubernetes/kubernetes#61006. The fix for this appears to have gone in to 1.13, and this repository appears to depend on the 1.14 client, so I'm not sure what's going on there.

Name:               replicator-deployment-766c46874f-54bs2
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               ip-172-31-15-37.us-west-2.compute.internal/172.31.15.37
Start Time:         Fri, 14 Feb 2020 13:37:22 -0800
Labels:             app=replicator
                    pod-template-hash=766c46874f
Annotations:        <none>
Status:             Running
IP:                 100.96.6.2
Controlled By:      ReplicaSet/replicator-deployment-766c46874f
Containers:
  replicator:
    Container ID:   docker://54fa9bce17bea730e50713ab35a0ad9de93b4321ac17f4d598870fa5c4995ec8
    Image:          quay.io/mittwald/kubernetes-replicator:latest
    Image ID:       docker-pullable://quay.io/mittwald/kubernetes-replicator@sha256:9cd515802fee4859d1978a39e2a2e4278b691c54a4236df1f5e663d9c9b37c2a
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 14 Feb 2020 14:35:05 -0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Fri, 14 Feb 2020 14:22:34 -0800
      Finished:     Fri, 14 Feb 2020 14:35:03 -0800
    Ready:          True
    Restart Count:  3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from replicator-token-xz6mg (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  replicator-token-xz6mg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  replicator-token-xz6mg
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason   Age                From                                                 Message
  ----    ------   ----               ----                                                 -------
  Normal  Pulling  28m (x4 over 86m)  kubelet, ip-172-31-15-37.us-west-2.compute.internal  Pulling image "quay.io/mittwald/kubernetes-replicator:latest"
  Normal  Pulled   28m (x4 over 86m)  kubelet, ip-172-31-15-37.us-west-2.compute.internal  Successfully pulled image "quay.io/mittwald/kubernetes-replicator:latest"
  Normal  Created  28m (x4 over 86m)  kubelet, ip-172-31-15-37.us-west-2.compute.internal  Created container replicator
  Normal  Started  28m (x4 over 86m)  kubelet, ip-172-31-15-37.us-west-2.compute.internal  Started container replicator

^ Note how the pod restarted 4 times

-> % kubectl -n kube-system logs replicator-deployment-766c46874f-54bs2 --previous --tail=10
2020/02/14 22:35:03 updating secret default/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 updating dependent secret kube-system/better-tls-3.dev.source.ai-tls -> jenkins/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 updating secret jenkins/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 secret jenkins/better-tls-3.dev.source.ai-tls is replicated from kube-system/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 updating secret jenkins/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 secret kube-system/better-tls-3.dev.source.ai-tls has 2 dependents
2020/02/14 22:35:03 updating dependent secret kube-system/better-tls-3.dev.source.ai-tls -> default/better-tls-3.dev.source.ai-tls
2020/02/14 22:35:03 updating secret default/better-tls-3.dev.source.ai-tls
W0214 22:35:03.908603       1 reflector.go:289] pkg/mod/k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:94: watch of *v1.Secret ended with: too old resource version: 36417 (40656)
log: exiting because of error: log: cannot create log: open /tmp/replicator.replicator-deployment-766c46874f-54bs2.unknownuser.log.WARNING.20200214-223503.1: no such file or directory

[ltagliamonte-dd]

facing the same issue.
It looks like it's an issue with glog. For this an other reason the kubernetes project moved away from it.
Some dependence may use it?

[ltagliamonte-dd]

I've fixed the issue updating the go-client library to v0.17.3
Seem working for me, will open a PR upstream once i'm sure I don't get the issue anymore.
I've built an publicly available image ltagliamonte/kubernetes-replicator:client-v0.17.3
@Bowbaq if you want to test it feel free.

[martin-helmich]

Fixed in #35.