Pepr Watch is not Responding to Changes after 90 mins

[cmwylie19]

Environment

Device and OS: UDS Core
App version:

• UDS Core v0.19.0 (Pepr v0.29.0) Reported by Jordan
• UDS Core v0.18.0 Pepr 0.28.6 Reported by Wayne

Watch controller started responding after rolling. Only pod logs were health checks

Kubernetes distro being used:
Other: EKS Kubernetes 1.29

Steps to reproduce

1. Working on reproducing soak

Expected result

Upon disconnection, it reconnects

Actual Result

Visual Proof (screenshots, videos, text, etc)

Severity/Priority

Additional Context

Add any other context or screenshots about the technical debt here.

Well Known Bug:
kubernetes-client/csharp#533
kubernetes-client/javascript#596

[cmwylie19]

here is a temporary workaround that should force the watcher pod to reconnect. https://gist.github.com/cmwylie19/2c07e0e6f0962b8f18999488d1646a4a

[cmwylie19]

something with the AWS VPC CNI could have issues and is sometimes dropping policy. It could have something to do with it.

Related Issue - aws/amazon-vpc-cni-k8s#2103

[cmwylie19]

Fixed by #766. If you experience this, use the environment variable PEPR_RESYNCINTERVALSECONDS and set it to something very low like 23 seconds. We soaked a module that was effected for 19 hours twice and did not see the problem

[marshall007]

Some seemingly related issues:

• Admission webhooks affected by dead tcp connections kubernetes/kubernetes#80313
• Internal Kubernetes API Calls Blocked by Istio istio/istio#8696
• Istio proxy sidecar interferes with Kubernetes API calls for the first seconds. istio/istio#12187
• In-cluster applications cannot access kube-apiserver istio/istio#12005
• Connection error when calling kubernetes-api-server (since 1.1.0-rc.0) istio/istio#12121

Possible solution/workaround in istio/istio#8696 (comment) by creating a ServiceEntry to treat connectino to k8s api-server as MESH_EXTERNAL.

[jeff-mccoy]

This is not technically resolved and setting a very low resync threshold like this is a really bad practice in production. Agree we either need to look at dropping Istio for this or looking at MESH_EXTERNAL

[cmwylie19]

This should be solved based on c0d3aaa and the KFC informer pattern. All soak tests have passed

[joelmccoy]

Ran into this the other day using uds-core 0.26.1 (pepr v0.36.0) in our internal environments with PEPR_LAST_SEEN_LIMIT_SECONDS set to 60. Didn't have the chance to catch the watch metrics, but will capture those if it occurs again.

[robsulllly]

Just ran through a UDS Core and bundling demo with Raytheon and ran into the problem where I had to restart the pepr-uds-core-watcher pod in order to get a successful deployment of mattermost, postgres operator, minio, etc. It looks like pepr blocks some of the secrets from being created, which then prevents mattermost from coming up. Killing the pepr watcher pod and then re-doing the deployment fixes the problem.

This can happen regardless of whether the cluster/platform has been up for hours or just came up a couple minutes ago. Does not happen every time though.

UDS Core 0.27.2 - k3d-core-demo:0.27.2 (also existed on 0.26.1)
ARM64
ghcr.io/defenseunicorns/packages/uds/dev-minio:0.0.2
ghcr.io/defenseunicorns/packages/uds/postgres-operator:1.12.2-uds.2-upstream
ghcr.io/defenseunicorns/packages/uds/mattermost:9.10.1-uds.0-upstream
Unfortunately, I was in the middle of a demo and had to press forward, kill the pepr watcher on the side, and re-deploy so I don't have the previous logs.

[JoeHCQ1]

Just got it today, happened while deploying core into a cluster, by the time I got to Neuvector, Pepr was AWOL. Had to kill the watcher, and bounce all affected Neuvector pods.

Core version 0.27.2.

[cmwylie19]

fixed in udici defenseunicorns/kubernetes-fluent-client#399, we have not heard any more complaints. Feel free to open it back up if it occurs again.