Pepr Watch is not Responding to Changes after 90 mins

[cmwylie19]

Environment

Device and OS: UDS Core
App version:

• UDS Core v0.19.0 (Pepr v0.29.0) Reported by Jordan
• UDS Core v0.18.0 Pepr 0.28.6 Reported by Wayne

Watch controller started responding after rolling. Only pod logs were health checks

Kubernetes distro being used:
Other: EKS Kubernetes 1.29

Steps to reproduce

1. Working on reproducing soak

Expected result

Upon disconnection, it reconnects

Actual Result

Visual Proof (screenshots, videos, text, etc)

Severity/Priority

Additional Context

Add any other context or screenshots about the technical debt here.

Well Known Bug:
kubernetes-client/csharp#533
kubernetes-client/javascript#596

[cmwylie19]

here is a temporary workaround that should force the watcher pod to reconnect. https://gist.github.com/cmwylie19/2c07e0e6f0962b8f18999488d1646a4a

[cmwylie19]

something with the AWS VPC CNI could have issues and is sometimes dropping policy. It could have something to do with it.

Related Issue - aws/amazon-vpc-cni-k8s#2103

[cmwylie19]

Fixed by #766. If you experience this, use the environment variable PEPR_RESYNCINTERVALSECONDS and set it to something very low like 23 seconds. We soaked a module that was effected for 19 hours twice and did not see the problem

[marshall007]

Some seemingly related issues:

• Admission webhooks affected by dead tcp connections kubernetes/kubernetes#80313
• Internal Kubernetes API Calls Blocked by Istio istio/istio#8696
• Istio proxy sidecar interferes with Kubernetes API calls for the first seconds. istio/istio#12187
• In-cluster applications cannot access kube-apiserver istio/istio#12005
• Connection error when calling kubernetes-api-server (since 1.1.0-rc.0) istio/istio#12121

Possible solution/workaround in istio/istio#8696 (comment) by creating a ServiceEntry to treat connectino to k8s api-server as MESH_EXTERNAL.

[jeff-mccoy]

This is not technically resolved and setting a very low resync threshold like this is a really bad practice in production. Agree we either need to look at dropping Istio for this or looking at MESH_EXTERNAL

[cmwylie19]

This should be solved based on c0d3aaa and the KFC informer pattern. All soak tests have passed