Clear Cache for Pepr Store on Failure

[cmwylie19]

Environment

If you try to remove an item from the Pepr store that is not there, it will always give a 422 error.

      try {
        // Send the patch to the cluster
        await K8s(PeprStore, { namespace, name: this.#name }).Patch(payload);
      } catch (err) {
        Log.error(err, "Pepr store update failure");

        // On failure to update, re-add the operations to the cache to be retried
        for (const idx of indexes) {
          sendCache[idx] = payload[Number(idx)];
        }
      }

Steps to reproduce

import { Capability, Log, a } from "pepr";

export const HelloPepr = new Capability({
  name: "hello-pepr",
  description: "A simple example capability to show how things work.",
});

const { When, Store } = HelloPepr;

When(a.Pod)
  .IsCreatedOrUpdated()
  .WithLabel("removeItem")
  .Watch(() => {
    Log.info("Pod with timeout label");
    for (let i = 0; i < 20; i++) {
      Store.removeItem(`element-${i}`);
    }
  });


  When(a.Pod)
  .IsCreatedOrUpdated()
  .WithLabel("a")
  .Watch(() => {
    Log.info("Pod with timeout label");
    Store.setItem("a","a")
  });

First, create a pod that gets the store's cache out of whack: k run a --image=nginx -l removeItem=2

Notice the 422 in the logs

Now try to update the store: k run b --image=nginx -l a=a

Notice you still get a 422 bc the cache still contains items that do not exist from the store

Expected result

Actual Result

Visual Proof (screenshots, videos, text, etc)

Severity/Priority

Additional Context

Add any other context or screenshots about the technical debt here.