Bug: unable to setup indexes

[danielvegamyhre]

Users are reporting an issue when they apply their JobSet, the JobSet object is created but it never starts running. After examining the logs we found the following error:

2023-08-22T21:19:59Z    ERROR    Reconciler error    {"controller": "jobset", "controllerGroup": "jobset.x-k8s.io", "controllerKind": "JobSet", ....  "error": "Index with name field:.metadata.controller does not exist"}

The index on .metadata.controller is created in SetupIndexes which is called from main.go here. If the indexes cannot be built, we would expect to see this error logged. Looking further in the logs, we do indeed see this error:

2023-08-21T14:37:20Z    ERROR    setup    unable to setup indexes    {"error": "failed to get API group resources: unable to retrieve the complete list of server APIs: batch/v1: Get \"https://10.72.0.1:443/apis/batch/v1\": dial tcp 10.72.0.1:443: i/o timeout"}

/kind bug

[danielvegamyhre]

Perhaps adding retry logic with some kind of exponential backoff policy here would be sufficient

[danielvegamyhre]

@kannon92 I notice Kueue has a helper utilruntime.Must(...) in which they call SetupIndexes. Perhaps we should implement something similar? This bug has been popping up a lot lately so I want to fix it asap.

[kannon92]

Yea I think that is a good idea. I think @vsoch ran into this actually but wasn't sure how to reproduce it.

[alculquicondor]

Interesting. I wouldn't have expected SetupIndexes to fail.
For whatever reason, we haven't hit these issues. From the message, it looks like the apiserver was down.

Indeed it seems best to exit.

[vsoch]

Correct - I've seen this several times, usually when I install / uninstall something and the only fix is to delete and re-create the entire cluster.