propagation: time limit exceeded: when using either lego CLI or docker image

[kevdogg]

I'm not exactly sure how to start debugging this error. I've build lego from source and also tried using the docker image however I'm getting the same result.

My host system is fedora
Lego version built from source: lego version 96bb0ba linux/amd64
Lego docker version: lego version 4.17.3 linux/amd64

I'm using a CSR in attempts to obtain LE certs. The reason for using this method is that ipa has a specific private key and I'm using the FreeIPA httpd private key to generate the CSR in attempts of then obtaining a LE SSL certificate. Other methods (https://github.com/freeipa/freeipa-letsencrypt/tree/master) use http webroot challenge and I'm basically trying to get around opening a port and use DNS challenge to accomplish the same task.

Here are my two scenarios:

1. CLI:

# LEGO_DEBUG_CLIENT_VERBOSE_ERROR=1 \
CF_API_EMAIL=myemail@gmail.com \
CF_API_KEY=<API_KEY> \
CLOUDFLARE_HTTP_TIMEOUT=30 \
/usr/bin/lego --email "myemail@gmail.com" --accept-tos \
--key-type RSA2048 \
--dns cloudflare --dns.resolvers "1.1.1.1:53" --csr httpd-csr.der run

2. Docker:

# docker run --env-file ./.env -it \
--mount type=bind,source="$(pwd)"/httpd-csr.der,target=/httpd-csr.der,readonly goacme/lego \
--email "myemail@gmail.com" --accept-tos \
--key-type RSA2048 \
--dns cloudflare --dns.resolvers "1.1.1.1:53" --csr httpd-csr.der run

When running both scenarios I get a similar output to the following:

2024/06/01 13:34:47 [INFO] [domain.gohilton.com] acme: Obtaining bundled SAN certificate given a CSR
2024/06/01 13:34:47 [INFO] [idomain.gohilton.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/358282955762
2024/06/01 13:34:47 [INFO] [domain.gohilton.com] acme: Could not find solver for: tls-alpn-01
2024/06/01 13:34:47 [INFO] [domain.gohilton.com] acme: Could not find solver for: http-01
2024/06/01 13:34:47 [INFO] [domain.gohilton.com] acme: use dns-01 solver
2024/06/01 13:34:47 [INFO] [domain.gohilton.com] acme: Preparing to solve DNS-01
2024/06/01 13:34:47 [INFO] cloudflare: new record for domain.gohilton.com, ID 64a9ae14897250a5fe29abe8cd1914e0
2024/06/01 13:34:47 [INFO] [domain.gohilton.com] acme: Trying to solve DNS-01
2024/06/01 13:34:47 [INFO] [domain.gohilton.com] acme: Checking DNS record propagation. [nameservers=1.1.1.1:53]
2024/06/01 13:34:49 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2024/06/01 13:34:49 [INFO] [domain.gohilton.com] acme: Waiting for DNS record propagation.
2024/06/01 13:34:51 [INFO] [domain.gohilton.com] acme: Waiting for DNS record propagation.
2024/06/01 13:34:53 [INFO] [domain.gohilton.com] acme: Waiting for DNS record propagation.
2024/06/01 13:34:55 [INFO] [domain.gohilton.com] acme: Waiting for DNS record propagation.
2024/06/01 13:34:57 [INFO] [domain.gohilton.com] acme: Waiting for DNS record propagation.
2024/06/01 13:34:59 [INFO] [domain.gohilton.com] acme: Waiting for DNS record propagation.

...Repeated for 30 seconds...
2024/06/01 13:34:59 [INFO] [domain.gohilton.com] acme: Waiting for DNS record propagation.
2024/06/01 13:36:49 [INFO] [domain.gohilton.com] acme: Cleaning DNS-01 challenge
2024/06/01 13:36:50 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/358282955762
2024/06/01 13:36:50 Could not obtain certificates:
	error: one or more domains had a problem:
[domain.gohilton.com] propagation: time limit exceeded: last error: NS jerry.ns.cloudflare.com. returned REFUSED for _acme-challenge.domain.gohilton.com.

Am I not passing the appropriate parameters to lego to accomplish this task??

[ldez]

Hello,

the most important part is last error: NS jerry.ns.cloudflare.com. returned REFUSED for _acme-challenge.domain.gohilton.com.

Your DNS (maybe there is a firewall) refused the DNS calls.

[kevdogg]

Hey I appreciate the response. I'm not sure how to troubleshoot the error as you stated -- however using acme.sh command line with same CF email and API key, I didn't have an issue -- I don't think however I can use a csr with acme.sh. With DNS challenge using cloudflare -- not sure there is actually a firewall involved here.

[ldez]

acme.sh doesn't check the propagation and doesn't detect the zone, so the difference is expected.

To troubleshoot, you should use dig/drill to check the DNS calls:

drill _acme-challenge.domain.gohilton.com. TXT
drill _acme-challenge.domain.gohilton.com. SOA
drill domain.gohilton.com. SOA
drill gohilton.com. SOA

Note, the problem is not related to CSR, but to DNS.

[kevdogg]

A little over my skis with checking DNS records since I don't know exactly what I'm supposed to be looking for, however looking at the output below I suspect there is a problem with the drill domain.gohilton.com. SOA query since it's basically returning empty

# drill _acme-challenge.domain.gohilton.com. TXT
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 47388
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; _acme-challenge.domain.gohilton.com.	IN	TXT

;; ANSWER SECTION:
_acme-challenge.domain.gohilton.com.	120	IN	TXT	"B2-OBnVeMeo8TK1EVXVQUge4P31WiwZb-CIdXJgUm2Q"

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 70 msec
;; SERVER: 127.0.0.53
;; WHEN: Sat Jun  1 09:45:52 2024
;; MSG SIZE  rcvd: 113

#drill _acme-challenge.domain.gohilton.com. SOA
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 57498
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; QUESTION SECTION:
;; _acme-challenge.domain.gohilton.com.	IN	SOA

;; ANSWER SECTION:

;; AUTHORITY SECTION:
gohilton.com.	1800	IN	SOA	connie.ns.cloudflare.com. dns.cloudflare.com. 2342736759 10000 2400 604800 1800

;; ADDITIONAL SECTION:

;; Query time: 71 msec
;; SERVER: 127.0.0.53
;; WHEN: Sat Jun  1 09:46:02 2024
;; MSG SIZE  rcvd: 118

# drill domain.gohilton.com. SOA
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 63436
;; flags: qr rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; domain.gohilton.com.	IN	SOA

;; ANSWER SECTION:

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 42 msec
;; SERVER: 127.0.0.53
;; WHEN: Sat Jun  1 09:52:17 2024
;; MSG SIZE  rcvd: 41

# drill gohilton.com. SOA
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 870
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; gohilton.com.	IN	SOA

;; ANSWER SECTION:
gohilton.com.	254	IN	SOA	connie.ns.cloudflare.com. dns.cloudflare.com. 2342736759 10000 2400 604800 1800

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 0 msec
;; SERVER: 127.0.0.53
;; WHEN: Sat Jun  1 09:53:55 2024
;; MSG SIZE  rcvd: 91

Local DNS resolution is through pfsense:

# resolvectl status
Global
           Protocols: -LLMNR -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
    resolv.conf mode: stub
  Current DNS Server: 10.1.0.1#pfsense-quincy.gohilton.com
         DNS Servers: 10.1.0.1#pfsense-quincy.gohilton.com 9.9.9.9#dns.quad9.net
Fallback DNS Servers: 127.0.0.1 10.0.1.1 9.9.9.9 1.1.1.2 1.1.1.3
          DNS Domain: gohilton.com ~.

Link 3 (eth1.0)
    Current Scopes: DNS
         Protocols: -DefaultRoute -LLMNR -mDNS DNSOverTLS=opportunistic DNSSEC=no/unsupported
Current DNS Server: 10.1.0.1
       DNS Servers: 10.1.0.1
        DNS Domain: ~gohilton.com

[kevdogg]

Thanks for pointing me to the right direction with the query. I had NAT firewall rule redirecting port 53 requests to the pfsense firewall itself -- this was suggested in the pfsense documentation. Anyway, I disabled the NAT firewall rule and was able to obtain a certificate. Thanks a lot for pointing me to the correct direction -- fairly certain I would have never figured the problem out without some useful guidance.