Basic auth disabled for /status endpoint

[bu3]

Hi,

I'm having an issue deploying bazel-remote in a K8s cluster using /status as liveness probe with basic authentication enabled.
I cannot enable allow_unauthenticated_reads because of some internal security requirement but that means that /status returns 401 and K8s kills my deployment.

Is there any way to disable authentication for /status only?

[mostynb]

Have you considered using the /grpc.health.v1.Health/Check (gRPC) service for this?

[bu3]

No. Sorry I didn't. I don't get it though. You mean to use it for a liveness probe over gRPC?

Something like this?

  livenessProbe:
    grpc:
      port: 9092
      service: /grpc.health.v1.Health/Check
  readinessProbe:
    grpc:
      port: 9092
      service: /grpc.health.v1.Health/Check

[mostynb]

Yes. I don't have a great knowledge of kubernetes so I can't offer much help with that side of things, but maybe the discussion in #522 is helpful? This gRPC service is supposed to always be reachable (as long as you haven't disabled gRPC support in bazel-remote) regardless of authentication status.

[bu3]

That's great. Thanks for getting back so fast

[mostynb]

No problem. There's also a brief mention of this in the kubernetes notes section of the README.md - suggestions for how to improve that are weclome, if you have any.

[bu3]

Hi @mostynb,

just to help anyone else dealing with the same issue, this deployment works fine with k8s >= 1.24:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: bazel-remote-cache
        image: buchgr/bazel-remote-cache:latest
        ports:
          - containerPort: 7442
            name: http
            protocol: TCP
          - containerPort: 8080
            name: http-default
            protocol: TCP
        livenessProbe:
            grpc:
              service: /grpc.health.v1.Health/Check
              port: 9092
            failureThreshold: 3
            initialDelaySeconds: 3
            periodSeconds: 1
            successThreshold: 1
            timeoutSeconds: 1
        readinessProbe:
            grpc:
              service: /grpc.health.v1.Health/Check
              port: 9092
            failureThreshold: 3
            periodSeconds: 3
            successThreshold: 1
            timeoutSeconds: 10

[mostynb]

Thanks for following up- I'll close this issue now, and we can move documentation discussion into #653.

[CareF]

This config seems to be buggy?

Deployment.apps "test-bazel-cache" is invalid: [spec.template.spec.containers[0].livenessProbe: Required value: must specify a handler type, spec.template.spec.containers[0].readinessProbe: Required value: must specify a handler type]

[bu3]

Hi @CareF, I've just tried to deploy it again and it seems to be working fine.

From my running pod description:

bazel-remote-cache:                                                                                                                                                 │
│     Container ID:   docker://f76ece540f53b6b2e8d8f58149109272374d162f9ebce27006e3ead9b55b61b9                                                                         │
│     Image:          buchgr/bazel-remote-cache:latest                                                                                                                  │
│     Image ID:       docker-pullable://buchgr/bazel-remote-cache@sha256:a334bb87e5059f1ea1a68a856b25a110dd09c538329303ccba90263ea664c750                               │
│     Ports:          7442/TCP, 8080/TCP                                                                                                                                │
│     Host Ports:     0/TCP, 0/TCP                                                                                                                                      │
│     State:          Running                                                                                                                                           │
│       Started:      Mon, 11 Sep 2023 09:44:54 +0100                                                                                                                   │
│     Ready:          True                                                                                                                                              │
│     Restart Count:  0                                                                                                                                                 │
│     Liveness:       grpc <pod>:9092 /grpc.health.v1.Health/Check delay=3s timeout=1s period=1s #success=1 #failure=3                                                  │
│     Readiness:      grpc <pod>:9092 /grpc.health.v1.Health/Check delay=0s timeout=10s period=3s #success=1 #failure=3

Tested with K8s 1.25

[CareF]

Hi @CareF, I've just tried to deploy it again and it seems to be working fine.

From my running pod description:

bazel-remote-cache:                                                                                                                                                 │
│     Container ID:   docker://f76ece540f53b6b2e8d8f58149109272374d162f9ebce27006e3ead9b55b61b9                                                                         │
│     Image:          buchgr/bazel-remote-cache:latest                                                                                                                  │
│     Image ID:       docker-pullable://buchgr/bazel-remote-cache@sha256:a334bb87e5059f1ea1a68a856b25a110dd09c538329303ccba90263ea664c750                               │
│     Ports:          7442/TCP, 8080/TCP                                                                                                                                │
│     Host Ports:     0/TCP, 0/TCP                                                                                                                                      │
│     State:          Running                                                                                                                                           │
│       Started:      Mon, 11 Sep 2023 09:44:54 +0100                                                                                                                   │
│     Ready:          True                                                                                                                                              │
│     Restart Count:  0                                                                                                                                                 │
│     Liveness:       grpc <pod>:9092 /grpc.health.v1.Health/Check delay=3s timeout=1s period=1s #success=1 #failure=3                                                  │
│     Readiness:      grpc <pod>:9092 /grpc.health.v1.Health/Check delay=0s timeout=10s period=3s #success=1 #failure=3

Tested with K8s 1.25

Thank you for the reply. My fault. Our k8s cluster is too old for grpc handler.

[pddg]

kubernetes does not support TLS in grpcProbe.
https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe

• The probes run against the pod IP address or its hostname. Be sure to configure your gRPC endpoint to listen on the Pod's IP address.
• The probes do not support any authentication parameters (like -tls).

Therefore, grpcProbe cannot be used in bazel-remote with TLS enabled.
If you enable TLS and also enable BASIC authentication, you must enable allow_unauthenticated_reads to set liveness/readiness probe.

Is there any good workaround?

[mostynb]

That's weird. I see that someone filed a kubernetes issue recently about not supporting gRPC health checks with TLS: kubernetes/kubernetes#128365

As a workaround maybe you could define a command probe and use something like https://github.com/grpc-ecosystem/grpc-health-probe which does support TLS.