Add Let's Encrypt R3 intermediate certificate to server

[tobykurien]

Bridgy server is currently failing to connect to sites using Let's Encrypt R3 certificates (I suspect), so I cannot syndicate from my personal website, I get:

{
  "error": "Could not fetch source URL https://tobykurien.com/post-1670753167/"
}

To fix this, on the bridgy server:

• download the intermediate certificate from https://letsencrypt.org/certs/lets-encrypt-r3.pem
• add this to the trusted CA bundle on the server: cat lets-encrypt-r3.pem >> /etc/ssl/certs/ca-certificates.crt

[tobykurien]

PS, I think this log error confirms the SSL problem: https://brid.gy/log?start_time=1670753224&key=agdicmlkLWd5ck4LEg1QdWJsaXNoZWRQYWdlIidodHRwczovL3RvYnlrdXJpZW4uY29tL3Bvc3QtMTY3MDc1MzE2Ny8MCxIHUHVibGlzaBiAgJjj0PbUCww&module=default

[snarfed]

Ruh roh. Thank you for reporting and details, and sorry for the trouble!

Bridgy is serverless, on https://cloud.google.com/appengine/ , so sadly I can't directly control the CA bundles on the host(s). Its outbound HTTP requests use https://requests.readthedocs.io/ though, which I think uses https://github.com/certifi/python-certifi as its trust store instead of the host's? I'll look.

[tobykurien]

I haven't tested, but this probably also affects ZeroSSL (maybe others?) A better long-term solution might be to regularly download and use a browser CA bundle, than waiting for python-certifi to update. I mean, the former R3 intermediate cert was deprecated over a year ago - https://blog.dnsimple.com/2021/09/letsencrypt-intermediate-expiration/ (https://blog.dnsimple.com/2021/09/letsencrypt-intermediate-expiration/)

[snarfed]

A year is definitely a long time, agreed! I don't plan to start maintaining my own cert bundle though, that's a bridge too far. I'll see what I can come up with.

[snarfed]

Looks like the root cause here is that Mozilla's CA bundle, which certifi is based on, doesn't include R3? That's odd, since Firefox 108 loads https://tobykurien.com/ fine and doesn't complain.

Background:

• requests uses certifi by default
• Bridgy currently uses python-certifi 2022.12.07
• I've reproduced this failure locally with requests.get('https://tobykurien.com/'), on requests 2.28.1 + certifi 2022.12.07 + Python 3.10.9 (from Homebrew) + macOS 11.6.7, so I know it's not doing something different on App Engine
• certifi uses Mozilla's CA cert bundle
• Mozilla CA bundle docs and full contents
  • I see Root X1 and X2 in that list, but not R3?
  • ...oh, because X1 and X2 are root certs but R3 isn't, so it wouldn't be in a CA bundle directly anyway?

I may be hitting the limit of my understanding of CA root vs intermediate certs and chains and CA bundles. Open to ideas! Except for maintaining my own bundle. 😎

[tobykurien]

I did some digging, and this may be related to "AIA fetching" as described here. A possible solution might be to use aia Python package.

[snarfed]

Oops, forgot to hit submit on this yesterday!

---

@tobykurien one workaround you could try is bundling the ISRG Root X1 R3 (?) cert with your site's certificate. I do that on eg snarfed.org, which lets Bridgy connect to it ok. I expect most other Bridgy users with LE certs currently do the same thing.

$ openssl s_client -connect snarfed.org:443
...
Certificate chain
 0 s:/CN=snarfed.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

vs.

$ openssl s_client -connect tobykurien.com:443
...
subject=/CN=cloud.tobykurien.com
issuer=/C=US/O=Let's Encrypt/CN=R3

[snarfed]

Re AIA fetching, yes! Thanks for the links, I ended up looking at that yesterday too. It could definitely help here, but just like maintaining my own cert bundle, I'm reluctant to take on something like that in an end user application. Those links also both describe it as a workaround for "missing" intermediate certs, and imply that the ideal answer might be for your server to include R3 (I think?) in its cert chain. From your first link:

It is a common flaw in SSL server configurations to provide an incomplete chain of certificates, often omitting intermediate certificates.

[tobykurien]

Understandable about not messing about with AIA fetching. Adding the CA to my server worked, so for others having similar issues, use the fullchain.pem. I'm using HAProxy, so I had to concat my cert + Let's encrypt CA + private key (in that order) into my PEM file.

[Quix0r]

Please remember that R3 and its siblings are now outdated certificates. You need R11 intermediate certs instead. And many servers seem to have moved to R11 already. So how can we fix this when my code tries to connect to an LE-encrypted website (R11 for example).

[snarfed]

@Quix0r Sure! The discussion here is almost three years old.

Is your question related to Bridgy? If your code is connecting to an R11 LE encrypted web site, that's unrelated to this issue or project.

[Quix0r]

We then should this to a new discussion. Sorry for hijacking this topic (my script connects to R11-LE encrypted websites).