Unable to setup Letsencrypt/SSL with Loadbalancer

[abhishek-ch]

I am trying to follow the Readme for Letsencrypt but I am not able to get https, by following the exact steps.

I have following errors when tryinh to add tls cert in the ingress

1. cert-manager-webhook pod throws TLS error http: TLS handshake error from 10.244.0.118:35840: EOF
2. cert-manager pod throws `-1 sync.go:380] "error waiting for authorization" err="acme: authorization error for test.mysite.com: 403 urn:ietf:params:acme:error:unauthorized: 2a02:4780:11:1740:0:1698:11a3:2: Invalid response from http://test.mysite.com/.well-known/acme-challenge/uhGWYpz66H5MScdOXNXaCukWG4UK5kkG9ImFG-V-aL8: 404" logger="cert-manager.controller.acceptChallenge" resource_name="test.mysite.com-tls-1-2029938892-612110412" resource_namespace="test" resource_kind="Challenge" resource_version="v1" dnsName="test.mysite.com" type="HTTP-01"

• 1 controller.go:98] ingress 'test/cm-acme-http-solver-7fck8' in work queue no longer exists

• 1 conditions.go:192] Found status change for Certificate "test.mysite.com-tls" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2024-08-13 13:40:22.166178033 +0000 UTC m=+3031.711949782

• 1 trigger_controller.go:194] "Backing off from issuance due to previously failed issuance(s). Issuance will next be attempted at 2024-08-13 14:40:22.00000148 +0000 UTC m=+6631.545773269" logger="cert-manager.controller" key="test/test.mysite.com-tls"`

Any suggestion

[vitobotta]

How are you creating the certificate? With a certificate resource or with the hooks e.g. in nginx?

[abhishek-ch]

I am simply using ingress and passing nginx classname , adding the annotation and ssl

  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"    # <<<--- Add annotation
    kubernetes.io/tls-acme: "true"

  tls:
  - hosts:
    - test.mysite.com
    secretName: test.mysite.com-tls

[vitobotta]

Can you share the full ingress yaml and the cluster issuer yaml minus secrets? Also have you waited for DNS propagation for the domain?

[abhishek-ch]

1. DNS lookup is showing Hetzner lb's Public IP for the subdomain I am using

2. My ingress yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: web-app
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web-app-service
spec:
  selector:
    app: web-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-app-ingress
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"    # <<<--- Add annotation
    kubernetes.io/tls-acme: "true"

spec:
  ingressClassName: nginx
  rules:
  - host: abc.mysite.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-app-service
            port:
              number: 80
  tls:
  - hosts:
    - abc.mysite.com
    secretName: tls-hello-mysite-prod-abc

3. Cluster Issuer yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: abc@test.net
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          class: nginx
---
apiVersion: cert-manager.io/v1
kind: "ClusterIssuer"
metadata:
  name: "letsencrypt-staging"
  namespace: cert-manager
spec:
  acme:
    email: "abc@test.net"
    server: "https://acme-staging-v02.api.letsencrypt.org/directory"
    privateKeySecretRef:
      name: "letsencrypt-staging-issuer-account-key"
    solvers:
      - http01:
          ingress:
            class: "nginx"

[abhishek-ch]

browser is showing k8s fake cert

[vitobotta]

Do you have proxy protocol enabled in the load balancer?

[abhishek-ch]

sorry, that's a typo, actually its abc.mysite.com

[vitobotta]

OK. I am not sure of what the problem could be. I have set up Nginx ingress with LetsEncrypt on these clusters a gazillion times and I have never had a problem (apart in the beginning when I wasn't aware of the proxy protocol thing).

[abhishek-ch]

And that's the feedback I got from many others, I will restart again and follow the readme exactly step by step ,

1 question regarding load balancer hostname, if I am using subdomain abc.mysite.com, will the load balancer hostname be mysite.com

[vitobotta]

The hostname is what you specify in the ingress, so if the ingress resource has abc.mysite.com that's what will be used. The load balancer just sends all the traffic it receives to the ingress so it doesn't directly know about the hostnames involved.

[vitobotta]

Please keep me posted if you find the problem since it may be something useful to add to the wiki for others in the future, although it's not directly a problem with my tool :)

[abhishek-ch]

So I finally managed to get the ssl working but Cert Manager/Let's Encrypt didnt work. So I generated the certificate and created a secret

kubectl create secret generic abc-mysite-tls --from-file=tls.crt=./tls.crt --from-file=tls.key=./tls.key

And Added the following in the ingress.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-world
  annotations: # <<<--- Add annotation
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - abc.mysite.com
    secretName: abc-mysite-tls
  rules:
  - host: abc.mysite.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-world
            port:
              number: 80

[vitobotta]

Still, it would be interesting to know what the problem was. Like I mentioned I have never had problems with cert-manager so I am kinda surprised.

[abhishek-ch]

I think I now know the issue, I am using hostinger and it doesnt support authentication via httpauth. It works fine with DNS but it doesnt provide any API endpoint to communicate and authorize the dns acme file. So the last option is manually uploading the cert.

[vitobotta]

Super weird. HTTP verification only visits a path on your site, while would that not work? Unless ... do they also offer LetsEncrypt verification? If yes, perhaps when the challenge is made from your cluster it is possible that their own .well-known path was receiving the request and failing the authorization.

[vitobotta]

Glad that you solved it with the DNS verification at least :)

[abhishek-ch]

I moved my domain to cloudflare and made it worked using dnsauth. Now cert manager + letsencrypt working perfectly as per the doc

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cloudflare-issuer
  namespace: cert-manager
spec:
  acme:
    email: <some email>
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - dns01:
        cloudflare:
          email: <some email>
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-app
  annotations:
    cert-manager.io/cluster-issuer: "cloudflare-issuer"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - abc.mysite.com
    secretName: abc-mysite-dns-tls