How might we control for logic that needs to be processed before the Pepr Module comes up

[cmwylie19]

Background:: 📖

UDS Core is creating a new ClusterConfig CRD, instances of said CRD will be hold information that is pertinent to reconciling UDS Packages and Exemptions.

Another similar use-case is how they process Exemptions. Exemptions must be processed before admission so the admission controller knows if there are any specific exceptions it must make based on the Exemptions CR.

They currently are doing this through a IIFE that processes the exemptions before the Pepr Module.

Problem:: 🍖

When Pepr is deployed, the AdmissionWebhooks are deployed, failurePolicy is set to Fail, and if a resource is not processed within the webhookTimeout then that resource will be rejected from the cluster.

Discussion:: 💡

Pepr could expand the API to new top level keywork, similar to When(<resource>), but something along the lines of WaitFor(<resource>). The code in this action when build would run in an initContainer, and exit successfully once the resource has been seen. This would allow UDS Core not have to worry about IIFEs and not have to worry about race conditions.

Really rough idea.

Example:

// initContainer 1
WaitFor(a.CustomResourceDefinition)
.WithName("clusterconfig.uds.dev")
.toBeCreated(crd => {
  Log.info("ClusterConfig CRD registered.")
})

// initContainer 2
WaitFor(ClusterConfig)
.WithName("initial")
.toBeCreated(async clusterConfig => {
  Log.info("ClusterConfig created.")
  Store.setItemAndWait("initialClusterConfig", clusterConfig)
})

// initContainer 3
WaitFor(a.CustomResourceDefinition)
.WithName("exemptions.uds.dev")
.toBeCreated(async crd => {
  Log.info("Exemption CRD  registered, fetching initial Exemptions)
  const initialExemptions = await fetchExemptions();
  Store.setItemAndWait("initialExemptions", initialExemptions)
})

// .. Pepr comes up, deploys webhooks, reads data from the store and continues.

The implementation behind the initContainer logic would not require webhooks but disposable watches from the informer

[TristanHoladay]

This is a cool idea!

Just a bit of clarification regarding the Exemptions (though it doesn't change much, if anything, about what you're proposing). The exemption logic that runs in the IIFE before the Pepr module is not necessarily for processing Exemptions before Packages or actual pods, though that is a nice benefit on upgrades or cycling of the Pepr pods. It mainly serves to set up the in-memory exemption store in the Admission controller and runs the watch from there so updates to the store happen as fast as possible. The control for making sure Exemptions exist before pods are created is by deploying those templates as a separate chart before the application chart (e.g. uds-neuvector-config > upstream chart). Even then, sometimes the application chart starts deploying before the Exemption is fully "reconciled", but the timing difference is so small that it's resolved on helm retries.

[btlghrants]

Some more possibilities to poke at:

To wait for a desired resource to match a desired state:

WaitFor(a.CustomResourceDefinition)
.WithName("clusterconfig.uds.dev")
.toMatch(crd => {
  crd.sub.data.key.status.thing === "whatIwanted" ? true : false;
})

To wait for a given custom lookup function (to call remote APIs..?) to return an object that matches the desired state:

Sample( () => { /* KFC.Get()..? */ }, every.seconds(30))
.toMatch(async crd => {
  crd.sub.data.key.status.thing === "whatIwanted" ? true : false;
})

To allow specifying the order of initContainer executions:

// "phases" (i.e. initContainers) run until success, in order of 0-n..?
WaitFor(a.Pod).InPhase(1).toXyz()
WaitFor(a.Pod).InPhase(2).toXyz()
WaitFor(a.Pod).toXyz() // no phase spec means "run in phase 0" or "phase 999" (or something).

Also, since we're lining these up as "guard"-kinda things, perhaps there's some way for us to end "a phase" once all conditions succeed / return true once, or... like, expose some sort of sdk/endPhase() call something something somewhere..? How would we want to allow Module Authors to signal that any given "phase" / initContainer should consider itself done in the case that an error occurs, or the check isn't just that something is existing?

[soltysh]

They currently are doing this through a IIFE that processes the exemptions before the Pepr Module.

What's currently wrong with this kind of approach? Is it only the fact that they are blocking starting the entire pepr module, and we have an explicit desire to init and perform some operations? I mean if you implement it using initContainers, the main container will not run until after initContainers successfully complete in order. Which means you're basically getting the exact same behavior like the IIEF + pepr module. Unless you're not talking about k8s initContainers.

[cmwylie19]

The current approach is a very solid approach. The reason that this discussion based on a statement that Tristan made and later wrote on the comment.

Even then, sometimes the application chart starts deploying before the Exemption is fully "reconciled", but the timing difference is so small that it's resolved on helm retries. (@TristanHoladay, comment)

The team was asking if it is a Pepr responsibility be able to run logic before the Webhooks start blocking admission due to failurePolicy = Fail and the webhookTimeout reaching the limit (In the event that we are waiting for a ClusterConfig CRD that is not present).

I don't necessarily think this is needed right now but we wanted to have the conversation documented as a possible alternative should we need to do this in the future.

[cmwylie19]

Also (for historical tracking), chatted briefly with @jeff-mccoy yesterday and he described a scenario where they mount a secret that is guaranteed to exist at pod startup. When the ClusterConfig CR is is created it the values are merged with the defaults from the secret. (Which was also talked about during the sync with Tristan)