Feature request: randomize TLS Let's Encrypt renewal logic

[nodesocket]

I propose changing the Let's Encrypt logic checking for TLS renewals instead of hard coding 12 hours to use a random value between 12 hours and 16 hours instead. This helps mitigate rate limit issues when running Caddy in a clustered environment and creating instances with automation (Terraform) all having nearly the same exact time for checking for TLS Let's Encrypt renewals.

The logic could be as simple as:

[0] - 12 hours
[1] - 12.5 hours
[2] - 13 hours
[3] - 13.5 hours
[4] - 14 hours
[5] - 14.5 hours
[6] - 15 hours
[7] - 15.5 hours
[8] - 16 hours

Then randomly select an integer between 0 and 8 and that's the duration to next check for Let's Encrypt renewals.

[mholt]

Hi -- thanks for the feature request.

I'm not sure that randomizing the timing of the renewal checks will really mitigate rate limit issues, since the typical relevant rate limit has a window of 7 days; varying it by a few hours won't help much.

Caddy is not currently designed to share certificates with other instances. However, when #2015 is merged in hopefully just a few days, Caddy will be more compatible with sharing certificates on disk used by other Caddy instances. One thing that PR does is cause Caddy to check on disk for an already-renewed certificate before trying to renew one itself. If the cert on disk is found to have a later expiry date, it simply loads it up and doesn't renew again. Even though this check is not (yet) synchronized between multiple Caddy instances, staggering the deployment of Caddy instances by a few seconds or minutes will mostly ensure in the vast majority of cases that rate limits aren't hit, and Caddy can nearly safely share the same certificates in the same ~/.caddy/acme folder. If timing of renewal checks is randomized, it is less certain that renewals won't step on each other. But with the PR as-is, I'm pretty confident in deploying many Caddy instances that share the same certificates on disk in step will work together well.

This is a stepping stone to truly modular, synchronized cert sharing, which is on the roadmap for future.

UPDATE: The timing is no longer an issue; Caddy instances will synchronize renewals.

[nodesocket]

Thanks for reply @mholt. To confirm, does a check to determine if a cert is expired (basically the check that runs every 12 hours) count against any Let's Encrypt rate limits?

[mholt]

does a check to determine if a cert is expired (basically the check that runs every 12 hours) count against any Let's Encrypt rate limits?

Not at all. Caddy only iterates an in-memory map of certificates to determine if they need to be renewed.

[nodesocket]

Suppose we create 10 Caddy instances automatically with Terraform and they all boot within in the same couple of seconds. They share a $CADDYPATH using Amazon Elastic Filesystem. Do all 10 instances fire off requests to create Let's Encrypt certificates at boot? That would cause a LE rate limit block right?

[mholt]

Suppose we create 10 Caddy instances automatically with Terraform and they all boot within in the same couple of seconds. They share a $CADDYPATH using Amazon Elastic Filesystem. Do all 10 instances fire off requests to create Let's Encrypt certificates at boot? That would cause a LE rate limit block right?

Depends. For the same hostname? And for certificates expiring soon? Certificates should be renewed unless Caddy hasn't been serving that hostname for some good amount of time (2+ months).

If you stagger the execution of Caddy instances by 5+ seconds, I think you should be good. (With that PR.) UPDATE: No longer necessary. It should just work.