token not valid for running mode after SSO authentication

[Deepa050]

Summary

Argo UI comes back to login page with error {"code":16,"message":"token not valid for running mode”} after completing SSO user authentication.

Diagnostics

argo version:
argo: v2.12.3
  BuildDate: 2021-01-05T00:49:48Z
  GitCommit: 93ee530126cc1fc154ada84d5656ca82d491dc7f
  GitTreeState: clean
  GitTag: v2.12.3
  GoVersion: go1.13
  Compiler: gc
  Platform: darwin/amd64

I have added sso configuration in the config map "workflow-controller-configmap"

sso: |
      issuer: https://cloudsso-test.xxx.com
      clientId:
        name: argo-server-sso
        key: clientID
      clientSecret:
        name: argo-server-sso
        key: clientSecret
      redirectUrl: http://argo-code.xxx.com/workflows/argo
      scopes:
       - groups
       - email
      rbac:
        enabled: false

I have added the required secrets for SSO and also starting argo-server in "--auth-mode sso"

argo-server logs :

time="2021-02-02T07:34:01.198Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2021-02-02T07:34:01Z" grpc.time_ms=0.34 span.kind=server system=grpc
time="2021-02-02T07:34:01.218Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = token not valid for running mode" grpc.code=Unauthenticated grpc.method=ListWorkflowTemplates grpc.service=workflowtemplate.WorkflowTemplateService grpc.start_time="2021-02-02T07:34:01Z" grpc.time_ms=0.065 span.kind=server system=grpc

Please note we use JWT token for SSO and not JWE.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

Did you also add —auth-mode=sso?

[Deepa050]

Yes I did. Below is the yaml for argo-server. Application goes through the entire SSO authentication successfully but fails to run after that. Can you also confirm if using JWT token doesnt impact in anyway since argo document says it uses JWE.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argo-server
spec:
  selector:
    matchLabels:
      app: argo-server
  template:
    metadata:
      labels:
        app: argo-server
    spec:
      containers:
      - args:
        - server
        - --namespaced
        - --auth-mode
        - sso
        image: argoproj/argocli:v2.12.3
        name: argo-server
        ports:
        - containerPort: 2746
          name: web
        readinessProbe:
          httpGet:
            path: /
            port: 2746
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 20
      nodeSelector:
        kubernetes.io/os: linux
      serviceAccountName: argo-server

[alexec]

It does not make sense to have groups without RBAC enabled. Groups are only used for RBAC.

     scopes:
       - groups
       - email
      rbac:
        enabled: true

[Deepa050]

I tried this but same issue.

[simster7]

Possibly fixed by: #4735

Please try upgrading

[Deepa050]

Upgrded to 2.12.8 but that didnt work

argo: v2.12.8
  BuildDate: 2021-02-08T23:25:36Z
  GitCommit: d19d4eeed3224ea7e854c658d3544663e86cd509
  GitTreeState: clean
  GitTag: v2.12.8
  GoVersion: go1.13
  Compiler: gc
  Platform: darwin/amd64

I tried --auth-mode sso --auth-mode server and that's when the SSO login went through. May be because of the bearer token in server mode. But how to make it work solely on --auth-mode sso

[alexec]

Have you tried private browsing mode yet?

[Deepa050]

Yes I have. same issue.

[arnoldrw]

Any chance you are using istio with argo? I saw this same behavior when running argo with istio. The fix was to add an annotation for the argo server deployment spec to exclude the kubernetes api IP range from being handled by istio. EG

spec:
  selector:
    matchLabels:
      app: argo-server
  template:
    metadata:
      labels:
        app: argo-server
      annotations:
        sidecar.istio.io/inject: "true"
        traffic.sidecar.istio.io/excludeOutboundIPRanges: yourk8api/32

The idea for this came from experience with other istio race condition issues outlined in 1282.

If you are using ssl termination this may help eliminate the login issues.

@simster7 this might be of interest.

[Deepa050]

We are not using istio. @arnoldrw
@simster7 It seems to be related to token. I am guessing since the sso login worked with --auth-mode sso --auth-mode server
Are the bearer token not same for the two auth mode ? Does it work differently.

[paveq]

I'm also having problems getting service account bearier tokens working with --auth-mode sso

Behaviour I would like is real users using SSO to login, but also supporting ServiceAccounts with Bearer tokens for service-to-service API calls.

[Deepa050]

Is there any solution to this ?

[alexec]

I think there are mixed problem in this issue. Maybe raise a new issue if you’re having problems. You should be able to use multiple auth modes.

[mruoss]

@Deepa050 Did you find a solution to this? We're facing the same problem on version 3.0.1 with SSO via argocd/dex.

[mruoss]

Alright - I think I just got lucky and found the problem. I assume you copied the credentials from the "Using Your Login With The CLI" dialogue in the UI. If I do that, for the token it does this:

export ARGO_TOKEN='"Bearer ..."'

if I remove the inner quotes it works:

export ARGO_TOKEN='Bearer ...'

[alexec]

@mruoss would you like to sumbit a PR to fix this mistake?

[rchojn]

@Deepa050 any updates? Have you resolved the issue? Even with --auth-mode server I have the same error

[mruoss]

@gardnerdev Did you try using the button on "Using Your Login With The CLI" on the user page in the web UI? It copies all the necessary commands to your clipboard.

[rchojn]

@mruoss the issue is about:

Argo UI comes back to login page with error {"code":16,"message":"token not valid for running mode”} after completing SSO user

why should I use CLI?

[mruoss]

oh right. I thought this was about using the CLI. Sorry.

[rchojn]

@alexec any advise how to debug this issue?

[mrsiesta]

Just FYI, the solution for us was to ensure we were specifying both SSO and Client with the server configuration:
#7490

Hopefully this helps someone else :)

[agilgur5]

missing Bearer in our case

Ran into this when switching from --auth-mode=server to a combination of --auth-mode=sso for UI usage and --auth-mode=client for programmatic API access.

In our case, --auth-mode=server was allowing an incorrect SA token access to the API, and we did not realize this until we removed --auth-mode=server (previously we had --auth-mode=server for UI access with an NGINX OAuth redirect + --auth-mode=client for programmatic API access).

The problem was we were missing Bearer in front of the token entirely. Even when logging in the UI with a token, you have to input Bearer insertmytokenhere and it will not work without that. Bearer is also case-sensitive.

If you run into this, I strongly recommend testing the token in the UI -- that was a very helpful validation step for us to make sure we were on the right track.

DX improvements to error messages etc

This was a misconfiguration on our side, but I think the error message that Argo gives back could be a lot more detailed. Many people (including various teams I work with) were confused by token not valid for running mode and thought that --auth-mode=client was incompatible with --auth-mode=sso or something, rather than the token just being incorrect.

--auth-mode=server is also, just, well, incredibly permissive even when used in combination with other modes, which can cause some confusion. Perhaps disallowing other modes when server is used might make sense? Not sure of the proper fix there, but I think some DX improvements could be made there as well.

[agilgur5]

Improved the docs and error message in #11300