Spike: Pod Failure Policy (K8 1.25 alpha feature) Can we utilize this in Armada?

[kannon92]

In 1.25, the wg-batch/sig-scheduling has added an ability to retry Pods/Jobs based on error codes.

kubernetes/enhancements#3329

Docs: https://kubernetes.io/docs/tasks/job/pod-failure-policy/

We should investigate if we can use this to clean up our logic for executor pod retries.

Currently, we have code in our executor that does this in a non standard way. https://github.com/G-Research/armada/blob/master/internal/executor/configuration/podchecks/types.go

We should investigate if we can map some of this functionality to use Pod Failures.

[dave-gantenbein]

Hey team! Please add your planning poker estimate with Zenhub @dejanzele @jayofdoom @kannon92 @richscott @suprjinx

[dave-gantenbein]

Hey team! Please add your planning poker estimate with Zenhub @ClifHouck @dejanzele @richscott @Sharpz7 @suprjinx

[dave-gantenbein]

Hey team! Please add your planning poker estimate with Zenhub @ClifHouck @dejanzele @headphonejames @kannon92 @richscott @Sharpz7 @suprjinx

[kannon92]

I don't think this KEP will be helpful to Armada Executor as is.

In the KEP, their non goal is to support pending phase of pods. This means they are not targeting image pull issues or any pending issue that could cause the pod to not be scheduled.

I created an issue in the Kubernetes repo to start this conversation.

kubernetes/kubernetes#113211

The only outstanding question is if GR would be interesting in retrying based on exit codes for the containers?

@JamesMurkin any thoughts on this last point?

[kannon92]

@JamesMurkin mentions that the interest in retrying is usually based on Pending. There hasn't been much request on exit code retrying.