GitLab OIDC provider results in ext_authz_denied

[karlschriek]

Is this a bug report or feature request?

• Bug Report

Describe the bug
Following the instruction in the readme (and also piecing together examples for a few different repos) I am unable to get the OIDC authservice to work. (I am doing this with Kubeflow 1.3 and am using GitLab to test the functionality).

How to Reproduce
My understanding is that the following should work. However, I get an ext_authz_denied error in the Istio Ingressgateway logs.

1. I start from this base: https://github.com/kubeflow/manifests/tree/master/common/oidc-authservice/base
2. params.env I update as follows:

OIDC_PROVIDER=https://gitlab.my-domain.com
USERID_HEADER=kubeflow-userid
USERID_CLAIM=email
OIDC_SCOPES=read_user profile email #comma separated seems to be wrong
OIDC_AUTH_URL=
AUTHSERVICE_URL_PREFIX=https://kubeflow.my-other-domain.com/authservice/
REDIRECT_URL=https://kubeflow.my-other-domain.com/authservice/oidc/callback #leaving this blank appears to not work as expected
CLIENT_NAME=Kubeflow
TEMPLATE_PATH=web/templates/gitlab/auto_logout
STRICT_SESSION_VALIDATION=true
STORE_PATH=/var/lib/authservice/data.db

3. A virtualservice is created as follows:

kind: VirtualService
metadata:
  name: authservice-web
spec:
  gateways:
    - kubeflow/kubeflow-gateway
  hosts:
    - '*'
  http:
    - match:
        - uri:
            prefix: /authservice/
      rewrite:
        uri: /
      route:
        - destination:
            host: authservice.istio-system.svc.cluster.local
            port:
              number: 8082

4. The authservice svc is patched to add port 8082, as follows:

apiVersion: v1
kind: Service
metadata:
  name: authservice
spec:
  ports:
  - port: 8082
    name: http-web
    targetPort: http-web

Expected behavior
I would expect the above to result in a successful authentication

Logs

authservice:

time="2021-05-11T10:47:37Z" level=info msg="Starting readiness probe at 8081"
time="2021-05-11T10:47:37Z" level=info msg="No USERID_TOKEN_HEADER specified, using 'kubeflow-userid-token' as default."
time="2021-05-11T10:47:37Z" level=info msg="No USERID_PREFIX specified, using '' as default."
time="2021-05-11T10:47:37Z" level=info msg="No SERVER_HOSTNAME specified, using '' as default."
time="2021-05-11T10:47:37Z" level=info msg="No SERVER_PORT specified, using '8080' as default."
time="2021-05-11T10:47:37Z" level=info msg="No SESSION_MAX_AGE specified, using '86400' as default."
time="2021-05-11T10:47:37Z" level=info msg="Starting web server at :8080"

istio ingress gateway:

[2021-05-11T10:52:21.016Z] "GET / HTTP/2" 302 - ext_authz_denied - "-" 0 418 3 2 "95.114.180.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" "2981ed16-3c5a-4cd6-94c3-5497a7582b80" "kubeflow.my-other-domain.com" "-" - - 10.10.0.215:443 95.114.180.246:55847 kubeflow.my-other-domain.com -
[2021-05-11T10:52:28.662Z] "GET /authservice/oidc/callback?code=5d830eaac4a01156871cf71e2f390cda1d2c86424d2eb6bbc9d3c054bc760c5e&state=MTYyMDczMDM0MXxFd3dBRUZkRU9FWXljVTVtU0VzMVlUZzBhbW89fKi3BLU4OfYGwR5zXD3xJyaDWjL83k-hhJvdVVo6T2EE HTTP/2" 302 - ext_authz_denied - "-" 0 418 3 3 "95.114.180.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" "5de49224-8295-414d-b7a2-ca93cdd78dfe" "kubeflow.my-other-domain.com" "-" - - 10.10.0.215:443 95.114.180.246:55847 kubeflow.my-other-domain.com

Environment:

• AuthService version: (found in image tag)
• Platform: EKS
• Kubernetes version: 1.18

Additional context
Add any other context about the problem here.

[karlschriek]

Some additional info:

Basing my distribution on https://github.com/arrikto/oidc-authservice/tree/master/e2e/manifests/authservice/base passes me through to the Kubeflow Dashboard without any authentication happening. The EnvoyFilter is still using the deprecated v2 though, so this might be the reason.

[karlschriek]

After updating the EnvoyFilter to v3 as follows:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: authn-filter
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
      patch:
        # For some reason, INSERT_FIRST doesn't work
        operation: INSERT_BEFORE
        value:
          name: "envoy.filters.http.ext_authz"
          config:
            http_service:
              server_uri:
                uri: http://authservice.$(NAMESPACE).svc.cluster.local
                cluster: outbound|8080||authservice.$(NAMESPACE).svc.cluster.local
                timeout: 10s
              authorization_request:
                allowed_headers:
                  patterns:
                    # XXX: MUST be lowercase!
                    - exact: "authorization"
                    - exact: "cookie"
              authorization_response:
                allowed_upstream_headers:
                  patterns:
                    - exact: "kubeflow-userid"

I now get the following error.

[2021-05-11T14:17:50.549Z] "GET / HTTP/2" 403 UAEX ext_authz_error - "-" 0 0 0 - "95.114.180.246" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" "a747bab4-920c-4810-a1b0-22e5b361f453" "kubeflow.my-other-domain.com" "-" - - 10.10.0.215:443 95.114.180.246:57885 kubeflow.my-other-domain.com -
2021-05-11T14:17:50.564166Z error envoy envoy_bug envoy bug failure: !state_.local_complete_ || status == FilterHeadersStatus::StopIteration. Details: Filters should return FilterHeadersStatus::StopIteration after sending a local reply.

[karlschriek]

After some more debugging I noticed that $(NAMESPACE) wasn't being replaced by Kustomize correctly (since the path to those vars were changed in the EnvoyFilter above). As a result, the FilterHeadersStatus::StopIteration no longer occurs, but I am now back to:

[2021-05-12T08:58:44.942Z] "GET / HTTP/2" 302 - ext_authz_denied - "-" 0 365 3 2 "95.117.25.93" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" "8c801b72-e886-4582-994e-5c29e3a36878" "kubeflow.my-other-domain.com" "-" - - 10.10.1.203:443 95.117.25.93:60032 kubeflow.my-other-domain.com -
[2021-05-12T08:58:45.323Z] "GET /robots.txt?1620809925246 HTTP/2" 302 - ext_authz_denied - "-" 0 365 3 2 "95.117.25.93" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36" "d67677b2-7d9d-447c-a33d-604b4a106d4b" "kubeflow.my-other-domain.com" "-" - - 10.10.1.203:443 95.117.25.93:60034 kubeflow.my-other-domain.com -

[yanniszark]

Hi Karl! Thanks for updating on your progress. Could you please provide more info on what your browser sees?
OIDC performs a specific set of actions to authenticate you, which we have documented in a diagram in the README: https://github.com/arrikto/oidc-authservice#readme
Can you provide more information about each step and clarify which step fails?

[psheorangithub]

I am also seeing the same issue recently.We are running KF 1.3.1 with Dev OIDC.This is occurring intermittently. Does anyone found a solution.

[2021-12-06T21:52:08.367Z] "GET /jupyter/api/namespaces/NS/notebooks HTTP/1.1" 403 UAEX ext_authz_error - "-" 0 0 9773 - "100.64.128.33,10.42.139.195" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" "eb3375d8-12ce-9765-8d53-73eed360187d" "<CLUSTER_URL>t" "-" - - 10.42.139.220:8080 10.42.139.195:34142 - -

[celiawa]

We're also seeing this same issue several times. We are using KF1.3.1.

# kubectl logs istio-ingressgateway-56447fcfb4-v7cgc -n istio-system
[2022-02-08T02:40:04.254Z] "GET / HTTP/1.1" 403 UAEX ext_authz_error - "-" 0 0 9998 - "10.210.152.75,10.42.245.128"
"python-requests/2.24.0" "e3dc78b8-10fd-995d-a9ad-4a6cb31067d4" "<cluster_url>" "-" - -
10.42.30.137:8080 10.42.245.128:40742 - -

Found that when with 403 error, the authservice ip:8080 not response. While ip:8081 is responsing.

#  k get po -n istio-system -o wide
NAME                                     READY   STATUS    RESTARTS   AGE   IP              NODE                 NOMINATED NODE   READINESS GATES
authservice-0                            1/1     Running   0          64d   10.42.245.145   node-10-101-101-10   <none>           <none>
#
# ping 10.42.245.145
PING 10.42.245.145 (10.42.245.145) 56(84) bytes of data.
64 bytes from 10.42.245.145: icmp_seq=1 ttl=63 time=0.446 ms
64 bytes from 10.42.245.145: icmp_seq=2 ttl=63 time=0.434 ms
64 bytes from 10.42.245.145: icmp_seq=3 ttl=63 time=0.454 ms
^C
--- 10.42.245.145 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2048ms
rtt min/avg/max/mdev = 0.434/0.444/0.454/0.025 ms
 #
# curl 10.42.245.145:8080 -v  --max-time 5
* Rebuilt URL to: 10.42.245.145:8080/
*   Trying 10.42.245.145...
* TCP_NODELAY set
* Connected to 10.42.245.145 (10.42.245.145) port 8080 (#0)
> GET / HTTP/1.1
> Host: 10.42.245.145:8080
> User-Agent: curl/7.58.0
> Accept: */*
>
* Operation timed out after 5000 milliseconds with 0 bytes received
* stopped the pause stream!
* Closing connection 0
curl: (28) Operation timed out after 5000 milliseconds with 0 bytes received
#
# curl 10.42.245.145:8081 -v  --max-time 5
* Rebuilt URL to: 10.42.245.145:8081/
*   Trying 10.42.245.145...
* TCP_NODELAY set
* Connected to 10.42.245.145 (10.42.245.145) port 8081 (#0)
> GET / HTTP/1.1
> Host: 10.42.245.145:8081
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 08 Feb 2022 07:41:43 GMT
< Content-Length: 0

The authservice pod manifest as below:

    image: gcr.io/arrikto/kubeflow/oidc-authservice:28c59ef
    imagePullPolicy: Always
    name: authservice
    ports:
    - containerPort: 8080
      name: http-api
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /
        port: 8081
        scheme: HTTP
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 1

@ryandawsonuk @yanniszark, could you please help to look on this.

[romanzdk]

Hi all, I am facing the same issue, any progress?