Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Certificate expiry 18 January 2022 #2811

Closed
8 tasks done
richardlau opened this issue Nov 19, 2021 · 16 comments · Fixed by #2932
Closed
8 tasks done

Certificate expiry 18 January 2022 #2811

richardlau opened this issue Nov 19, 2021 · 16 comments · Fixed by #2932

Comments

@richardlau
Copy link
Member

richardlau commented Nov 19, 2021

The Sectigo certificate we use on nodejs.org (including the two Jenkins servers) is due to expire on 18 January 2022.

image

From the previous renewal (#1901) I guess this is one for @brianwarner?

cc @nodejs/build-infra

(It's on my radar to get some sort of automated checking/warning for this (and the code signing certs) but that shouldn't block getting the certificates renewed.)


Tasks

  • Obtain new certificate
  • update www / direct.nodejs.org
  • update unencrypted.nodejs.org (this is the failover/load balanced website)
  • update ci.nodejs.org
  • update ci-release.nodejs.org
  • update CloudFlare
  • Update certificate and private key in secrets
  • Add documentation on how to renew/update the certificate doc: add certificate update notes #2932
@rvagg
Copy link
Member

rvagg commented Nov 22, 2021

This is via gogetssl.com btw. I've just logged in and there's no option to renew, says it's got 58 days remaining and there's only an option to "Reissue". I think maybe we're too early—perhaps it'll start warning us at 40 or 30 days. But good to keep an eye on!

@brianwarner
Copy link

brianwarner commented Nov 22, 2021 via email

@rvagg
Copy link
Member

rvagg commented Nov 23, 2021

No specific need for gogetssl, it's just the provider we ended up at because they were a good broker doing discounts for major providers.
We're using letsencrypt on all our other sites, but nodejs.org is a little more complicated where reissuing every 3 months isn't an option—although IIRC that's only because of our use of cloudflare, who issue their own certificates these days, so it may be possible for us to use CF for the main nodejs.org certificates and letsencrypt for the other subdomains that we don't shunt through CF (like ci and ci-release). But, that's yet more work!
So yes @brianwarner, a new wildcard from a provider that gives us a long expiry would be nice. LF IT may have some difficulty validating ownership of the domain since we do DNS through CloudFlare, so it might end up being easier to use gogetssl for renewal. I think I gave you the login for that last time we renewed and I just did the initial setup, so we could just do that again.

@brianwarner
Copy link

brianwarner commented Nov 23, 2021 via email

@rvagg
Copy link
Member

rvagg commented Nov 23, 2021

I'm not so worried about nodejs.org, but all the other sites we use the wildcard for. Just requires a bit of work to get through everything and make sure we have them set up properly. But we'd have to manually replace the cert anyway so we have work ahead of us regardless!

@brianwarner
Copy link

brianwarner commented Nov 23, 2021 via email

@mhdawson
Copy link
Member

@rvagg can you see the renewal option now?

@richardlau
Copy link
Member Author

richardlau commented Jan 11, 2022

@mhdawson and I have spent my afternoon (re)figuring out what needs to be done, writing stuff up as we go (once we're done we'll add the instructions either here or add to secrets).

I've done https://unencrypted.nodejs.org/ (we picked a low-volume site that we felt confident we could restore (i.e. backup the existing files on the server so they could be copied back) if anything went wrong) to do first, which now reports certificate valid until February 2023:

image

It appears that even though we went for 60 months (5 years) the certificates are issued for 13 months but can be reissued unlimited times during the ordered period (i.e. the 60 months) so we'll still need to update the machines yearly: https://www.gogetssl.com/wiki/general/multi-year-subscription-ssl/

@richardlau
Copy link
Member Author

Updated https://ci.nodejs.org/ and https://direct.nodejs.org/

image
image

@mhdawson It looks like https://nodejs.org/ is still showing the old certificate which suggests that is what the Cloudflare setting we found affects (i.e. it's used for the sites that are proxied/load balanced):

image

I'm leaving updating https://ci-release.nodejs.org/ and updating Cloudflare to @mhdawson (we agreed beforehand -- https://ci-release.nodejs.org/ so that he can run through some instruction I have written to validate).

@richardlau
Copy link
Member Author

hmm. Maybe something isn't right... although the updated sites work in my web browser trying to use curl is throwing an unable to get local issuer certificate error:

$ curl -v https://direct.nodejs.org
* Rebuilt URL to: https://direct.nodejs.org/
*   Trying 138.197.224.240...
* TCP_NODELAY set
* Connected to direct.nodejs.org (138.197.224.240) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
$

@richardlau
Copy link
Member Author

Okay... it looks like one of the emails we got containing the certificate has three BEGIN CERTIFICATE/END CERTIFICATE blocks while the one @mhdawson downloaded from the order page only contained one. If I replace the .crt file with the three blocks then the above curl error goes away.

@rvagg
Copy link
Member

rvagg commented Jan 12, 2022

Right, so this comes down to what's in the chained cert file .. it'd be worth checking the star chained that's in https://github.com/nodejs-private/secrets/pull/198 to see whether the other certs match what we have now and whether there's enough of them (or too many?). We'd had a bunch of problems with either having too many or not enough in our chained cert file! Some clients need more, some complain at more.
Also use the https://www.ssllabs.com/ssltest/ to see what it says about the current setup, it'll also look at the chaining and report what it thinks about it.

@mhdawson
Copy link
Member

@richardlau we should probably add @rvagg 's suggestions to the documentation as additional verification steps after an update.

Running https://www.ssllabs.com/ssltest/ the major complaint is that we allow TLS 1.0 and 1.1 which caps the grade at B. I can' remember any discussions around whether we should force TLS 1.2 or higher in the past?

It also shows as not having CAA as an issue we could probably address - https://blog.qualys.com/product-tech/2017/03/13/caa-mandated-by-cabrowser-forum?_ga=2.90215386.175395475.1642194769-1593607923.1642194769

It also shows a number of TLS 1.2 protocols that we allows which are weak as well although they look to be symmetric algs and do we have anything that is actually confidential on the site?

The only other thing that shows as orange is Session resumption (caching) No (IDs assigned but not accepted)

I guess the thing that suggested running in the first place was to look at the certificates and that shows as 100% on the report.

`

@andig
Copy link

andig commented Jan 25, 2022

Cert is expired now: nodejs/help#3686 (comment)

@richardlau
Copy link
Member Author

@andig Everything I have access to is successfully validating the new certificate (with curl) so we'll need help identifying failing environments.

As @mhdawson mentioned, https://www.ssllabs.com/ssltest/ which is reporting 100 on certificates, including trusted certification paths: https://www.ssllabs.com/ssltest/analyze.html?d=nodejs.org

@andig
Copy link

andig commented Jan 25, 2022

@richardlau sorry for the confusion, please mark my comment as OT. For what it's worth, getting this response on fully patch OSX Monterey- seems the Root CA is missing in OSX?

❯ wget -v https://nodejs.org/download/release/v16.13.2/node-v16.13.2-headers.tar.gz
--2022-01-25 13:30:05--  https://nodejs.org/download/release/v16.13.2/node-v16.13.2-headers.tar.gz
Resolving nodejs.org (nodejs.org)... 2606:4700:10::6814:172e, 2606:4700:10::6814:162e, 104.20.23.46, ...
Connecting to nodejs.org (nodejs.org)|2606:4700:10::6814:172e|:443... connected.
ERROR: cannot verify nodejs.org's certificate, issued by ‘CN=Sectigo RSA Domain Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB’:
  Unable to locally verify the issuer's authority.
To connect to nodejs.org insecurely, use `--no-check-certificate'.

richardlau added a commit to richardlau/build that referenced this issue Apr 25, 2022
Add notes on website certificate renewal based on what was done
the last time we updated in January 2022.

Refs: nodejs#2811
richardlau added a commit to richardlau/build that referenced this issue Apr 25, 2022
Add notes on website certificate renewal based on what was done
the last time we updated in January 2022.

Refs: nodejs#2811
@richardlau richardlau linked a pull request Apr 25, 2022 that will close this issue
richardlau added a commit that referenced this issue Apr 28, 2022
Add notes on website certificate renewal based on what was done
the last time we updated in January 2022.

Refs: #2811
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants