Renew LetsEncrypt certificates #641

iulianbarbu · 2023-02-19T21:47:08Z

Closes #621 .

Description

This PR adds functionality that enables the periodical renewal of LetsEncrypt certificates. These certificates have a validity window of 90 days and is recommended to be renewed 30 days before expiration. The functionality will cover renewal for:

certificates emitted for a custom domain project when requested through the admin gateway API "/admin/acme/request/:project_name/:fqdn" or through shuttle-admin CLI.
the gateway certificate, which is loaded when starting the gateway service or created based on a set of ACME credentials present on the local filesystem.

WIP testing

End-to-end manual test of the certificate renewal with all shuttle components deployed locally. I have some issues with the shuttle local deployment that blocks me while testing. I need some more time to understand how to enable shuttle locally, with pebble deployed locally as CA.
Unit testing for the introduced business logic where it makes sense and is possible. Spent lots of time on 1) and I'll need to come back at this.

I welcome any suggestions on steps/guides/docs for how to enable the shuttle components locally to test the certificates renewal.

Documentation

Haven't seen much documentation on how to test the admin functionality related to the certificates requesting, so I hope I'll be able to compile something worth mentioning after finishing this.

Review wise

Besides code review, I am curious at the beginning to understand if this PR covers what's asked from #621 functionality-wise. I would be happy to follow up with adjustments code-wise if there are suggestions for improvement. Also, in parallel, I'll try to come back with information on the testing aspect.

iulianbarbu · 2023-02-20T16:32:58Z

Testing update

I am blocked when requesting a certificate by the fact that my pebble local deployment finalizes an order but it's not returning the certificate url, needed by the ACME client to download the certificate. Can't say what's the order status after finalizing, I'll need to debug by working through the instance-acme dependency code.

So far, the pebble CA log looks like this:

Pebble 2023/02/20 17:55:06 ACME directory available at: https://0.0.0.0:14000/dir
Pebble 2023/02/20 17:56:55 GET /dir -> calling handler()
Pebble 2023/02/20 17:56:55 HEAD /nonce-plz -> calling handler()
Pebble 2023/02/20 17:56:55 POST /sign-me-up -> calling handler()
Pebble 2023/02/20 17:56:55 There are now 1 accounts in memory
Pebble 2023/02/20 17:57:19 HEAD /nonce-plz -> calling handler()
Pebble 2023/02/20 17:57:19 POST /order-plz -> calling handler()
Pebble 2023/02/20 17:57:19 There are now 1 authorizations in the db
Pebble 2023/02/20 17:57:19 Added order "niAlrdD77-VhHeU6raADNxxxYuZSjCqQujLmUTCmirI" to the db
Pebble 2023/02/20 17:57:19 There are now 1 orders in the db
Pebble 2023/02/20 17:57:19 POST /authZ/ -> calling handler()
Pebble 2023/02/20 17:57:19 POST /chalZ/ -> calling handler()
Pebble 2023/02/20 17:57:19 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"hello-world-rocket-app.unstable.shuttleapp.rs"}, Challenge:(*core.Challenge)(0xc0001c8280), Account:(*core.Account)(0xc0000785a0)}
Pebble 2023/02/20 17:57:19 Starting 3 validations.
Pebble 2023/02/20 17:57:19 Sleeping for 3s seconds before validating
Pebble 2023/02/20 17:57:19 Sleeping for 2s seconds before validating
Pebble 2023/02/20 17:57:19 Sleeping for 2s seconds before validating
Pebble 2023/02/20 17:57:20 POST /my-order/ -> calling handler()
Pebble 2023/02/20 17:57:20 POST /my-order/ -> calling handler()
Pebble 2023/02/20 17:57:21 POST /my-order/ -> calling handler()
Pebble 2023/02/20 17:57:21 Attempting to validate w/ HTTP: http://hello-world-rocket-app.unstable.shuttleapp.rs:7999/.well-known/acme-challenge/trr79hstEGoLoSkdJfdQWxAZq7Ufl1BDG4PmKzsMVks
Pebble 2023/02/20 17:57:21 Attempting to validate w/ HTTP: http://hello-world-rocket-app.unstable.shuttleapp.rs:7999/.well-known/acme-challenge/trr79hstEGoLoSkdJfdQWxAZq7Ufl1BDG4PmKzsMVks
Pebble 2023/02/20 17:57:22 Attempting to validate w/ HTTP: http://hello-world-rocket-app.unstable.shuttleapp.rs:7999/.well-known/acme-challenge/trr79hstEGoLoSkdJfdQWxAZq7Ufl1BDG4PmKzsMVks
Pebble 2023/02/20 17:57:22 authz UFddOjhDm8lo3tgfjdAvU4pJYo8l8iIgTYFrIE4IsvE set VALID by completed challenge -lhkvnOJCIG4rRoDj6b-ijmsfUD_GiKAXKkl1ekwmo8
Pebble 2023/02/20 17:57:23 POST /my-order/ -> calling handler()
**Pebble 2023/02/20 17:57:23 POST /finalize-order/ -> calling handler()
Pebble 2023/02/20 17:57:23 Order niAlrdD77-VhHeU6raADNxxxYuZSjCqQujLmUTCmirI is fully authorized. Processing finalization
Pebble 2023/02/20 17:57:23 Issued certificate serial 05cf8f20a311617a for order niAlrdD77-VhHeU6raADNxxxYuZSjCqQujLmUTCmirI**

It seems that the order finalization succeeded with a certificate issued, but the finalization response doesn't contain the certificate URL, or the logs don't mention it sadly.

The gateway logs look like this:

The place where this fails is in instant-acme finalize order method:

gateway/src/main.rs

gateway/src/api/latest.rs

chesedo · 2023-02-21T07:29:49Z

Hey @iulianbarbu, thanks for the PR 😄

It seems that the order finalization succeeded with a certificate issued, but the finalization response doesn't contain the certificate URL, or the logs don't mention it sadly.

Gateway should create a trace event with "got authorization" (a few lines earlier). Iirc, this log will contain some challenge URL that can help with the debugging

iulianbarbu · 2023-02-21T08:08:31Z

Hey @chesedo ! Thanks for taking a look. On the pebble side, the above log I sent contains this at the end, but haven't checked the corresponding trace events on the gateway side. I'll try again.

Pebble 2023/02/20 17:57:23 POST /my-order/ -> calling handler()
Pebble 2023/02/20 17:57:23 POST /finalize-order/ -> calling handler()
Pebble 2023/02/20 17:57:23 Order niAlrdD77-VhHeU6raADNxxxYuZSjCqQujLmUTCmirI is fully authorized. Processing finalization
Pebble 2023/02/20 17:57:23 Issued certificate serial 05cf8f20a311617a for order niAlrdD77-VhHeU6raADNxxxYuZSjCqQujLmUTCmirI

iulianbarbu · 2023-02-21T12:52:53Z

@brokad , I've addressed your comments here: 8b9f484. Let me know in case you have other concerns.

Other than this, I will take another stab at the end-to-end manual test and then unit testing + docs.

chesedo

Thanks for the updates @iulianbarbu

admin/src/args.rs

chesedo

LGTM

Containerfile

gateway/src/acme.rs

oddgrd

Thanks for this Iulian, LGTM to me too! I left a few comments but mostly just nice things and a question or two 😄

Makefile

admin/README.md

gateway/Cargo.toml

gateway/src/acme.rs

gateway/src/main.rs

...and gateway certificates renewal. * added a new `request-gateway-certificate` admin command to renew the gateway certificate on-deman. * added gateway admin APIs for custom domains and gateway certificates renewal. Note: requesting renewal for the gateway certificate requires inserting manually a DNS TXT record to complete the ACME DNS-01 challenge. Signed-off-by: Iulian Barbu <iulianbarbu2@gmail.com>

...and removed Makefile containters additional build args for panamax and postgres. It seems there is a 'PLATFORMS' env variable in the Makefile which can be set before building the images and used by `buildx` to build against the mentioned target platforms. The `PLATFORMS` env variable is not needed on my machine because I use Docker Desktop, which supports running containers built for other arch using behind the scenes a LinuxKit VM that intercepts [1][2] the binaries and runs them using binfmt_misc, through QEMU [1]. [1] https://www.docker.com/blog/the-magic-behind-the-scenes-of-docker-desktop/ [2] https://stackoverflow.com/questions/72444103/what-does-running-the-multiarch-qemu-user-static-does-before-building-a-containe Signed-off-by: Iulian Barbu <iulianbarbu2@gmail.com>

Simplified the functions' signatures to use only the `AccountCredentials`, which are received from the API requests and read as well from the gateway state when needed.

...when needing to request/renew custom-domain and gateway certificates on the local environment.

Strings that are returned by gateway API requests must be wrapped within double quotes to be deserialized with serde_json.

chesedo

Thanks for all the edits @iulianbarbu. LGTM

Containerfile

...of pem files.

iulianbarbu force-pushed the renew_lets_encrypt_certificates_621 branch 4 times, most recently from 4dd4956 to a6a669e Compare February 19, 2023 21:55

brokad self-requested a review February 20, 2023 08:53

iulianbarbu force-pushed the renew_lets_encrypt_certificates_621 branch from a6a669e to 861b868 Compare February 20, 2023 09:08

brokad reviewed Feb 20, 2023

View reviewed changes

gateway/src/main.rs Outdated Show resolved Hide resolved

gateway/src/main.rs Outdated Show resolved Hide resolved

gateway/src/api/latest.rs Outdated Show resolved Hide resolved

brokad mentioned this pull request Feb 20, 2023

Renew LetsEncrypt certificates arriving at expiration #621

Closed

iulianbarbu force-pushed the renew_lets_encrypt_certificates_621 branch 2 times, most recently from 7af6922 to 5ad20c0 Compare February 21, 2023 12:49

iulianbarbu force-pushed the renew_lets_encrypt_certificates_621 branch 3 times, most recently from 8b9f484 to 2d81d54 Compare February 22, 2023 17:52

iulianbarbu force-pushed the renew_lets_encrypt_certificates_621 branch 4 times, most recently from 31e7d4a to 87afe4f Compare March 3, 2023 18:22

chesedo approved these changes Mar 7, 2023

View reviewed changes

admin/src/args.rs Show resolved Hide resolved

iulianbarbu force-pushed the renew_lets_encrypt_certificates_621 branch 5 times, most recently from 03dfb60 to 6bbc4da Compare March 21, 2023 13:16

iulianbarbu changed the title ~~[RFC] Renew LetsEncrypt certificates~~ Renew LetsEncrypt certificates Mar 21, 2023

chesedo approved these changes Mar 21, 2023

View reviewed changes

Containerfile Outdated Show resolved Hide resolved

gateway/src/acme.rs Outdated Show resolved Hide resolved

oddgrd approved these changes Mar 21, 2023

View reviewed changes

Makefile Outdated Show resolved Hide resolved

admin/README.md Outdated Show resolved Hide resolved

gateway/Cargo.toml Show resolved Hide resolved

gateway/src/acme.rs Outdated Show resolved Hide resolved

gateway/src/main.rs Outdated Show resolved Hide resolved

iulianbarbu added 14 commits March 21, 2023 21:06

review-feedback: refactored credentials sharing

cf948a3

Simplified the functions' signatures to use only the `AccountCredentials`, which are received from the API requests and read as well from the gateway state when needed.

shuttle-dev*/Containerfile: copy pem files as well

f494f71

auth/admin: authorize renew certs APIs

7cf7430

gateway: fixed x509 certificates parsing

68b0126

admin/docs: added notes for Pebble setup...

a39420a

...when needing to request/renew custom-domain and gateway certificates on the local environment.

gateway/acme: bumped instant-acme version

56a2632

Containerfile: arch agnostic protoc setup

4e45ff9

admin: fix gateway string responses

ad21636

Strings that are returned by gateway API requests must be wrapped within double quotes to be deserialized with serde_json.

Makefile: keep *.pem files for dev images

15482cb

gateway: simplify certificate polling duration

17c51e3

Makefile: remove comment

3aeb06e

gateway: revert the type coercion for Arc::clone

b177ac1

iulianbarbu force-pushed the renew_lets_encrypt_certificates_621 branch from 6bbc4da to b177ac1 Compare March 21, 2023 22:02

chesedo approved these changes Mar 22, 2023

View reviewed changes

Containerfile Show resolved Hide resolved

Containerfile: added comment to explain the addition...

b2f6e60

...of pem files.

iulianbarbu merged commit 6843874 into shuttle-hq:main Mar 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Renew LetsEncrypt certificates #641

Renew LetsEncrypt certificates #641

iulianbarbu commented Feb 19, 2023 •

edited

Loading

iulianbarbu commented Feb 20, 2023 •

edited

Loading

chesedo commented Feb 21, 2023

iulianbarbu commented Feb 21, 2023

iulianbarbu commented Feb 21, 2023 •

edited

Loading

chesedo left a comment

chesedo left a comment

oddgrd left a comment

chesedo left a comment

Renew LetsEncrypt certificates #641

Renew LetsEncrypt certificates #641

Conversation

iulianbarbu commented Feb 19, 2023 • edited Loading

Description

WIP testing

Documentation

Review wise

iulianbarbu commented Feb 20, 2023 • edited Loading

Testing update

chesedo commented Feb 21, 2023

iulianbarbu commented Feb 21, 2023

iulianbarbu commented Feb 21, 2023 • edited Loading

chesedo left a comment

Choose a reason for hiding this comment

chesedo left a comment

Choose a reason for hiding this comment

oddgrd left a comment

Choose a reason for hiding this comment

chesedo left a comment

Choose a reason for hiding this comment

iulianbarbu commented Feb 19, 2023 •

edited

Loading

iulianbarbu commented Feb 20, 2023 •

edited

Loading

iulianbarbu commented Feb 21, 2023 •

edited

Loading