feat: watchcfg is configurable for KFC from Pepr (#766)

## Description

Under specific conditions the Kubernetes Watch will stop sending events
to the client even though the connection seems healthy. This issue is
well known. After researching, we found that tuning the resync interval
is the best way to ensure the connection stays healthy.

Note that this issue is very rare and is not likely to occur but if your
environment has a lot of network pressure and constant churn it is a
possibility.

## Related Issue

Fixes #765 
<!-- or -->
Relates to # kubernetes-client/javascript#596

## Type of change

- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Other (security config, docs update, etc)

## Checklist before merging

- [x] Test, docs, adr added or updated as needed
- [x] [Contributor Guide
Steps](https://docs.pepr.dev/main/contribute/contributor-guide/#submitting-a-pull-request)
followed

---------

Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>

docs/030_user-guide/120_customization.md

@@ -2,6 +2,18 @@
 
 This document outlines how to customize the build output through Helm overrides and `package.json` configurations.
 
+## Customizing Watch Configuration 
+
+The Watch configuration is a part of the Pepr module that allows you to watch for specific resources in the Kubernetes cluster. The Watch configuration can be customized by specific enviroment variables of the Watcher Deployment and can be set in the field in the `package.json` or in the helm `values.yaml` file.
+
+| Field                        | Description                                                                                                      | Example Values                  |
+|------------------------------|------------------------------------------------------------------------------------------------------------------|---------------------------------|
+| `PEPR_RETRYMAX`              | The maximum number of times to retry the watch, the retry count is reset on success.                             | default: `"5"`                  |
+| `PEPR_RETRYDELAYSECONDS`     | The delay between retries in seconds.                                                                            | default: `"10"`                 |
+| `PEPR_RESYNCINTERVALSECONDS` | Amount of seconds to wait before a forced-resyncing of the watch list                                            | default: `"300"` (5 mins)       |
+| `PEPR_ALLOWWATCHBOOKMARKS`   | Whether to allow [watch bookmarks](https://kubernetes.io/docs/reference/using-api/api-concepts/#watch-bookmarks).| default: `"true"`  or `"false"` |
+
+
 ## Customizing with Helm
 
 Below are the available Helm override configurations after you have built your Pepr module that you can put in the `values.yaml`.

src/lib/watch-processor.ts

@@ -10,8 +10,12 @@ import { Binding, Event } from "./types";
 
 // Watch configuration
 const watchCfg: WatchCfg = {
-  retryMax: 5,
-  retryDelaySec: 5,
+  retryMax: process.env.PEPR_RETRYMAX ? parseInt(process.env.PEPR_RETRYMAX, 10) : 5,
+  retryDelaySec: process.env.PEPR_RETRYDELAYSECONDS ? parseInt(process.env.PEPR_RETRYDELAYSECONDS, 10) : 5,
+  resyncIntervalSec: process.env.PEPR_RESYNCINTERVALSECONDS
+    ? parseInt(process.env.PEPR_RESYNCINTERVALSECONDS, 10)
+    : 300,
+  allowWatchBookmarks: process.env.PEPR_ALLOWWATCHBOOKMARKS ? process.env.PEPR_ALLOWWATCHBOOKMARKS === "true" : false,
 };
 
 // Map the event to the watch phase
@@ -47,6 +51,8 @@ async function runBinding(binding: Binding, capabilityNamespaces: string[]) {
   const phaseMatch: WatchPhase[] = eventToPhaseMap[binding.event] || eventToPhaseMap[Event.Any];
 
   // The watch callback is run when an object is received or dequeued
+
+  Log.debug({ watchCfg }, "Effective WatchConfig");
   const watchCallback = async (obj: KubernetesObject, type: WatchPhase) => {
     // First, filter the object based on the phase
     if (phaseMatch.includes(type)) {