Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Security policy from different route interferes authentication #2507

Closed
sadovnikov opened this issue Jan 26, 2024 · 7 comments · Fixed by #2586
Closed

Security policy from different route interferes authentication #2507

sadovnikov opened this issue Jan 26, 2024 · 7 comments · Fixed by #2586
Assignees
@zhaohuabing
Labels
kind/bug Something isn't working road-to-ga
Milestone
v1.0.0-rc1

Comments

@sadovnikov
Copy link
Contributor

Description:

Reusing the same Gateway, we created two pairs of HTTPRoute with matching SecurityPolicy, which use identical IDP settings for OIDC. The HTTP Routes are on different hosts. Both hosts are CNAMEs to the same A-record, with the IP address of the Gateway service

hostname path http route security policy
reference-apps-twin.platform-staging.gcp.internal.xxx.org /cbdp-k8s-sample-eg cbdp-k8s-sample-web-eg cbdp-k8s-sample-web-eg
reference-apps.platform-staging.gcp.internal.xxx.org /cbdp-k8s-sample-pr-195 cbdp-k8s-sample-web-pr-195 cbdp-k8s-sample-web-pr-195

Opening of https://reference-apps.platform-staging.gcp.internal.xxx.org/cbdp-k8s-sample-pr-195 in an incognito tab goes through the login process and the application page gets opened.

However, the opening of https://reference-apps-twin.platform-staging.gcp.internal.xxx.org/cbdp-k8s-sample-eg/ in a new incognito tab gives "too many redirects" after the login page. This link starts working when the cbdp-k8s-sample-web-pr-195 Security Policy, which belongs to a different HTTP Route, is deleted.

Repro steps:

  • the envoy-proxy-min-err.json is the Envoy configuration with both Security Policies present
  • the envoy-proxy-min-ok.json is the Envoy configuration without the cbdp-k8s-sample-web-pr-195 Security Policy
  • YAML of HTTP Routes and Security Policies will be added in the issue comment

envoy-proxy-min-err.json
envoy-proxy-min-ok.json

Environment:

Envoy Gateway is deployed using the v0.0.0-latest Helm chart.

dependencies:
- name: gateway-helm
  version: v0.0.0-latest
  repository: oci://docker.io/envoyproxy
  alias: eg

Container

Containers:
  envoy:
    Container ID:  containerd://b8648147d345bcf50d6f68aab09149ece2737c29d72fb0c6fb18115cb946699d
    Image:         envoyproxy/envoy-dev:latest
    Image ID:      docker.io/envoyproxy/envoy-dev@sha256:9d9493167f9b35e0047ce1463df8b679bb5847f1d2b7bbb9d8fb432b08e97bab

Logs:

  • attached envoy-log.txt is the envoy proxy log from "too many redirects"

envoy-log.txt
@sadovnikov
Copy link
Contributor Author

HTTP Routes

apiVersion: v1
items:
- apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata:
    labels:
      app.kubernetes.io/instance: reference-apps-spring-boot-eg
    name: cbdp-k8s-sample-web-eg
    namespace: reference-apps
  spec:
    hostnames:
    - reference-apps-twin.platform-staging.gcp.internal.xxx.org
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: shared-host-default
      namespace: envoy-gateway-system
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: cbdp-k8s-sample-web-eg
        port: 80
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /cbdp-k8s-sample-eg
      timeouts:
        request: 10s
- apiVersion: gateway.networking.k8s.io/v1
  kind: HTTPRoute
  metadata:
    annotations:
      meta.helm.sh/release-name: cbdp-k8s-sample-pr-195-preemptible
      meta.helm.sh/release-namespace: reference-apps
    labels:
      app.kubernetes.io/managed-by: Helm
    name: cbdp-k8s-sample-web-pr-195
    namespace: reference-apps
  spec:
    hostnames:
    - reference-apps.platform-staging.gcp.internal.xxx.org
    parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: shared-host-default
      namespace: envoy-gateway-system
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: cbdp-k8s-sample-web-pr-195
        port: 80
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /cbdp-k8s-sample-pr-195
      timeouts:
        request: 10s
kind: List

Security Policies

apiVersion: v1
items:
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    labels:
      app.kubernetes.io/instance: reference-apps-spring-boot-eg
    name: cbdp-k8s-sample-web-eg
    namespace: reference-apps
  spec:
    oidc:
      clientID: d8c0c5a5-5b45-4566-9fb2-367e62c397ee
      clientSecret:
        group: ""
        kind: Secret
        name: oauth2-secret-eg
      provider:
        issuer: https://login.microsoftonline.com/a87b6d3d-d85e-4d9b-8704-6aed76a49444/v2.0
      scopes:
      - api://d8c0c5a5-5b45-4566-9fb2-367e62c397ee/oidc
      - offline_access
    targetRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: cbdp-k8s-sample-web-eg
      namespace: reference-apps
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    annotations:
      meta.helm.sh/release-name: cbdp-k8s-sample-pr-195-preemptible
      meta.helm.sh/release-namespace: reference-apps
    labels:
      app.kubernetes.io/managed-by: Helm
    name: cbdp-k8s-sample-web-pr-195
    namespace: reference-apps
  spec:
    oidc:
      clientID: d8c0c5a5-5b45-4566-9fb2-367e62c397ee
      clientSecret:
        group: ""
        kind: Secret
        name: oauth2-secret-eg
      provider:
        issuer: https://login.microsoftonline.com/a87b6d3d-d85e-4d9b-8704-6aed76a49444/v2.0
      scopes:
      - api://d8c0c5a5-5b45-4566-9fb2-367e62c397ee/oidc
      - offline_access
    targetRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: cbdp-k8s-sample-web-pr-195
      namespace: reference-apps
kind: List

@akhenakh
Copy link
Contributor

akhenakh commented Feb 7, 2024 •
edited

I can confirm it happened on our installations as well, while applying an ODIC security policy to an isolated httproute (hostname + path prefix)

  hostnames:
  - xxxxx.aaaa.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: envoy-gateway-system
  rules:
  - backendRefs:
    - kind: Service
      name: auth-generator-api
      port: 8085
    matches:
    - path:
        type: PathPrefix
        value: /

the OIDC redirection triggered on some others HTTPRoute

for example a route supposed to match /v1/users triggered the OIDC redirect with the url /v1/users/ (trailing slash)rather than the usual 404.

When removing the security policy on xxxxx.aaaa.com, the expected 404 on zzzzzz.aaaa.com reappeared.

Maybe it should be another ticket, since the title is matching but not the description, the security policy with the OIDC has been applied to another route that had nothing to do with it, not protected by OIDC at all.

@arkodg
Copy link
Contributor

arkodg commented Feb 8, 2024

assigning this to @zhaohuabing since he's been debugging this and has a solution, most likely the API will need to be revisited to include a user defined hmac secret (similar to #2567)

@arkodg arkodg added kind/bug Something isn't working road-to-ga and removed triage labels Feb 8, 2024
@arkodg arkodg added this to the v1.0.0-rc1 milestone Feb 8, 2024
@zhaohuabing
Copy link
Member

zhaohuabing commented Feb 9, 2024 •
edited

Reusing the same Gateway, we created two pairs of HTTPRoute with matching SecurityPolicy, which use identical IDP settings for OIDC. The HTTP Routes are on different hosts.

@sadovnikov I believe you can't use identical OIDC settings for two different HTTPRoutes because the callback url on the IDP side can only point to one HTTPRoute, and you want to make sure the callback url is caught by the targeting HTTPRoute.

Instead, you should have something like the below:

apiVersion: v1
items:
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    labels:
      app.kubernetes.io/instance: reference-apps-spring-boot-eg
    name: cbdp-k8s-sample-web-eg
    namespace: reference-apps
  spec:
    oidc:
      clientID: ${client-id-1}
      clientSecret:
        group: ""
        kind: Secret
        name: oauth2-secret-eg-1
      provider:
        issuer: https://login.microsoftonline.com/a87b6d3d-d85e-4d9b-8704-6aed76a49444/v2.0
      scopes:
      - api://d8c0c5a5-5b45-4566-9fb2-367e62c397ee/oidc
      - offline_access
      redirectURL: https://reference-apps-twin.platform-staging.gcp.internal.xxx.org/cbdp-k8s-sample-eg/oauth2/callback
    targetRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: cbdp-k8s-sample-web-eg
      namespace: reference-apps
- apiVersion: gateway.envoyproxy.io/v1alpha1
  kind: SecurityPolicy
  metadata:
    annotations:
      meta.helm.sh/release-name: cbdp-k8s-sample-pr-195-preemptible
      meta.helm.sh/release-namespace: reference-apps
    labels:
      app.kubernetes.io/managed-by: Helm
    name: cbdp-k8s-sample-web-pr-195
    namespace: reference-apps
  spec:
    oidc:
      clientID: ${client-id-2}
      clientSecret:
        group: ""
        kind: Secret
        name: oauth2-secret-eg-2
      provider:
        issuer: https://login.microsoftonline.com/a87b6d3d-d85e-4d9b-8704-6aed76a49444/v2.0
      scopes:
      - api://d8c0c5a5-5b45-4566-9fb2-367e62c397ee/oidc
      - offline_access
      redirectURL: https://reference-apps.platform-staging.gcp.internal.xxx.org/cbdp-k8s-sample-pr-195/oauth2/callback
    targetRef:
      group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: cbdp-k8s-sample-web-pr-195
      namespace: reference-apps

@zhaohuabing
Copy link
Member

zhaohuabing commented Feb 10, 2024 •
edited

I can confirm it happened on our installations as well, while applying an ODIC security policy to an isolated httproute (hostname + path prefix)

  hostnames:
  - xxxxx.aaaa.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: envoy-gateway-system
  rules:
  - backendRefs:
    - kind: Service
      name: auth-generator-api
      port: 8085
    matches:
    - path:
        type: PathPrefix
        value: /

the OIDC redirection triggered on some others HTTPRoute

for example a route supposed to match /v1/users triggered the OIDC redirect with the url /v1/users/ (trailing slash)rather than the usual 404.

When removing the security policy on xxxxx.aaaa.com, the expected 404 on zzzzzz.aaaa.com reappeared.

Maybe it should be another ticket, since the title is matching but not the description, the security policy with the OIDC has been applied to another route that had nothing to do with it, not protected by OIDC at all.

I reproduced this in my test env. Somehow a request that misses all the routes still hits the filters in the HCM filter chain even though these filters are disabled in the routeConfig. It seems a bug in the Envoy upstream. A workaround is adding a catching-all route at the bottom of the routes and returning a 404 direct response.

https://github.com/zhaohuabing/playground/blob/8e73a6573922aeac91550574da91e62d0708c61c/envoy/per-route-oauth2-oidc/envoy.yaml#L109-L142

@zhaohuabing zhaohuabing mentioned this issue Feb 12, 2024
Merged
@zhaohuabing zhaohuabing reopened this Feb 15, 2024
@zhaohuabing
Copy link
Member

This has been fixed in EG with #2586. Reopen it to track the upstream issue (which I haven't created yet).

@arkodg
Copy link
Contributor

arkodg commented Feb 27, 2024

fixed with #2663

@arkodg arkodg closed this as completed Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zhaohuabing zhaohuabing

Labels
kind/bug Something isn't working road-to-ga
Projects
Envoy Gateway: The Road to GA
Status: Done
Milestone
v1.0.0-rc1
Development

Successfully merging a pull request may close this issue.

add a catch-all route if needed zhaohuabing/gateway
4 participants
@akhenakh @zhaohuabing @sadovnikov @arkodg