Quickstart: transport: authentication handshake failed

[jmclean-starburst]

Summary

What happened/what you expected to happen?
When connecting to the UI of Argo Workflows (after port forwarding), I receive this message and virtually no components load:

Service Unavailable: connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"

What version of Argo Workflows are you running?
latest (ie 3.2.6)

Diagnostics

Quickstart Guide - follow steps in guide to reproduce

What Kubernetes provider are you using?
EKS/Minikube

Configuration:

    spec:
      containers:
      - args:
        - server
        - --namespaced
        - --auth-mode
        - server
        - --auth-mode
        - client
        - --insecure-skip-verify
        - "true"

kustomization.yaml

resources:
  - namespace.yml
  - https://raw.githubusercontent.com/argoproj/argo-workflows/v3.2.6/manifests/quick-start-minimal.yaml

namespace: argo-workflows

Note: This is reproducible w/ the minimal quickstart via minikube as well

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[materkey]

I also have this issue

[materkey]

argo-server seems to respond with 503 but can't see it in argo-server pod logs even with -v flag

[materkey]

UPD:
After @jmclean-starburst comment #7632 (comment)
I'm using v3.2.6 for quay.io/argoproj/argocli container image and for my kustomization.yaml to solve this issue

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/argoproj/argo-workflows/manifests/base?ref=v3.2.6
- github.com/argoproj/argo-workflows/manifests/cluster-install/workflow-controller-rbac?ref=v3.2.6
- github.com/argoproj/argo-workflows/manifests/cluster-install/argo-server-rbac?ref=v3.2.6
...

Old comment text

I've ended up by using --secure=false and scheme: HTTP for argo-server probe as a workaround. Also deleted nginx.ingress.kubernetes.io/backend-protocol: https for ingress.
Https for argo workflows worked for months and now it is broken for an unknown reason.

[jmclean-starburst]

@materkey I have done this in the past...its just kind of annoying to update the readiness probe, add a new flag to server start, update ingress; This definitely needs to be resolve by the Argo team bc any new user of workflows is going to be put off by these errors. I have used the quickstart in the past without any issues...so this seems a bit wonky.

[jmclean-starburst]

I believe I found the issue; the argo CLI is using latest for the image tag, instead of specifying the version tag (ie v3.2.6). I have tested this w/ v3.1.15 and after changing the image tag, it works. I will now validate same for version 3.2.6

[jmclean-starburst]

I was able to fix this by patching the image off of latest

resources:
  - namespace.yml
  - https://raw.githubusercontent.com/argoproj/argo-workflows/v3.2.6/manifests/quick-start-minimal.yaml

images:
- name: quay.io/argoproj/argocli
  newTag: v3.2.6

namespace: argo-workflows

[alexec]

Please, you must use the manifests attached to the release:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.2.6/install.yaml

[jmclean-starburst]

@alexec I believe the quickstart guide is broken then, as the latest tag on argocli will spawn the issue mentioned by this bug

[alexec]

I now think this might be bug.

[alexec]

Maybe

3614db6 feat: adding support for getting tls certificates from kubernetes secret (e.g. (#7621)

[alexec]

@ChaosInTheCRD do you think your commit could have introduced this bug please?

[tachyus-ryan]

I ran into this error after updating one of my clusters from GKE 1.19 to 1.20. I cannot seem to connect to argo-server since restarting deployments, no matter what I try. I was on v3.2.4 and tried updating to v3.2.6 to no avail.

[tachyus-ryan]

I was able to fix this by patching the image off of latest

This worked for me, too.

[plaicebo]

Please, you must use the manifests attached to the release:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.2.6/install.yaml

This did it for me. I was using the quick start manifest instead of the release specific. That will have to be updated.

[andrein]

Please, you must use the manifests attached to the release:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.2.6/install.yaml

For kustomize users, it's much easier to reference the remote base from git. We tried to do this for the 3.2.6 tag and experienced the same issue.

We've been doing the same with argo-cd for a while without issues.

It look like they're overriding the images with the correct version when the git tag is created:

• https://github.com/argoproj/argo-cd/blob/v2.2.3/hack/update-manifests.sh
• https://github.com/argoproj/argo-cd/blob/v2.2.3/manifests/ha/base/kustomization.yaml

I found #2715 that seems related, @alexec can you reopen it to continue the kustomize discussion there?

[ChaosInTheCRD]

@alexec it seems that yes, the problem has come from my PR, apologies for that. My suspicion is that it is related to the removal of this line, which I overlooked as I guess some process (I guess the UI) can't trust the self signed cert generated. I will investigate this further.

I guess though, the main thing to resolve right now is to ensure that this quickstart does not use the latest image. This way it can be ensured that prerelease bugs like this don't hinder users.

As for the error itself, can I get some clarity on what is resolving the error? Is it directly from the ui or is there something else involved (e.g. API).

[alexec]

This issues is a blocker or v3.3, which we'd hoped to release this week. If you don't have time to look into this, I suggest we revert the commit.

[tachyus-ryan]

As for the error itself, can I get some clarity on what is resolving the error? Is it directly from the ui or is there something else involved (e.g. API).

I found that I could not submit workflows to the API, so this does not appear to be a UI-only issue. Ensuring latest was not used was the means of resolving the error for me.