Support startup probes

[ramperher]

Feature Request

Describe the problem you need a feature to resolve.

Currently, working with operators created with operator-sdk, we could see that liveness and readiness probe are availables in the controller-manager pods (as a result of #1234), but startup probes, which are also documented in k8s documentation, they're not present.

To bypass this, a hotfix for this is to create a startupProbe that is consuming one of the endpoints created by any of the liveness and readiness probes:

...
          livenessProbe:
            httpGet:
              path: /healthz
              port: 6789
            initialDelaySeconds: 15
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /readyz
              port: 6789
            initialDelaySeconds: 5
            periodSeconds: 10
          startupProbe:
            httpGet:
              path: /healthz
              port: 6789
            initialDelaySeconds: 30
            periodSeconds: 10
...

Describe the solution you'd like.

Ideally, the startup probe should have its own endpoint.

...
          livenessProbe:
            httpGet:
              path: /healthz
              port: 6789
            initialDelaySeconds: 15
            periodSeconds: 20
          readinessProbe:
            httpGet:
              path: /readyz
              port: 6789
            initialDelaySeconds: 5
            periodSeconds: 10
          startupProbe:
            httpGet:
              path: /startz
              port: 6789
            initialDelaySeconds: 30
            periodSeconds: 10
...

Probably something similar to what was implemented in this PR, could be done in this case: #4326. However, for this, controller-manager must include the support of startup probes, and now it isn't. I have opened this issue for that: kubernetes-sigs/controller-runtime#2644

After having that enabled, then I suppose everything should be in place to make this work in operator-sdk.

[jberkhahn]

So this looks like it's getting worked on in controller runtime. Once it's in, it'll get pulled in when we bump the controller runtime version, so putting this is the backlog.

[ramperher]

Issue in controller-runtime repo was finally closed as won't fix, so I'll close this one too.