Delete temporary resources when agent completes

[carolynvs]

When the agent completes, delete any temporary resources created by the operator such as: jobs, pvcs, secrets.

[carolynvs]

See https://sdk.operatorframework.io/docs/best-practices/resource-pruning/

[carolynvs]

It would be nice to have some configuration options:

• feature flag to disable pruning
• TTL or some other way to say how long to keep them before deleting
• Possibly keep failed runs longer or treat differently than passing jobs

[carolynvs]

We could use the built-in cleanup offered in k8s (https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/), but it will delete the resource based on the configured TTL when the job finishes(the job status is completed or failed).
What we can do is keep the finalizer on the resource longer and change the configured TTL if we want to hold off on cleanup until some other condition (like giving a failed run an hour for troubleshooting for example).

[carolynvs]

The plan is to add TTLSecondsAfterFinished to AgentConfig. This controls how an AgentAction and associated resources (job, pv, pvc, secrets, etc) should remain after the action finishes (success or failure). We will apply that value to the Agent Job and watch for its deletion. When it's deleted, we will remove our finalizer to allow it to delete, and then delete the parent AgentAction.

Resources created by the operator

• Agent Job
• Agent PVC
• Agent PV (share bundle outputs and files between porter and the bundle)
• Porter Config Secret
• Porter WorkDir Secret

Resources created by the agent (k8s driver)

• Bundle Job
• Bundle Secret (inject env vars into bundle)

I have created cnabio/cnab-go#300 to track supporting owner references in the k8s driver

[carolynvs]

I chose to not go with the operator framework pruning library because it would be on us to schedule and call it regularly. There aren't any documented best practices on how to run that in an operator at this time and I didn't find any use of the library in existing projects to look for ideas.

The simplicity of pawning off the pruning trigger onto k8s instead seems more reliable to me.