Amazon RDS 2019 Intermediate Certificates

[jthele]

#2280 added the RDS 2019 global root certificate, sourced from:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html

That documentation provides a caveat:

This root certificate is a trusted root entity and should work in most cases but might fail if your application doesn't accept certificate chains. If your application doesn't accept certificate chains, download the AWS Region–specific certificate from the list of intermediate certificates found later in this section.

When I attempted to use just the global root certificate as provided in v2.18.0 to connect to an AWS RDS MySQL 5.6.37 server configured with the rds-ca-2019 certificate, I got HANDSHAKE_SSL_ERROR and Error: unable to verify the first certificate. Adding the intermediate certificates to ssl_profiles.js remedied that problem.

Adding the intermediate certificates in addition to the root certificate is consistent with what was done when the AWS RDS 2015 certificates were added to this module.

Editing to add: This was using Node 10.17.0, and the RDS instance was in us-east-1.

[dougwilson]

It's hard for me to believe after all the hub-bub and begging for that PR to be merged that it does not work. I need some more folks to better explain this and how after the 3 months of that PR this was never noticed and how, specifically this one is going to fix it and how we're going to verify it this time...

[jthele]

What additional info can I provide you?

[dougwilson]

What additional info can I provide you?

I need more people with access to Amazon RDS instances to explain before I just merge something blindly again. That, or access to Amazon RDS to test it myself.

[dougwilson]

So there is already one response on that PR and I see you responded. It is currently 3:30am in my time zone, so I'm going to head to bed, but once we can get some clarification on this, then we can determine to merge or not, and once we know it should be merged, I can put it in a patch.

[SimonHooker]

As noted on #2280, the global CA is sufficient for most people however some setups require the intermediates provided on this PR. #2280 really should have had them included also for consistency however I never thought to check them. Detail on global / intermediates for RDS available here : https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html

[dougwilson]

the global CA is sufficient for most people however some setups require the intermediates provided on this PR

So the question I would like to understand first is can someone explain what, specifically, the differences are between the setups that requires one vs the other? For example, just to explain the change as like "MySQL 5.6.x requires these" would be a big step towards getting it merged, but right not it sounds like it's just a mystery as to why, which makes is extremely hesitant to merge in blindly...

[SimonHooker]

@dougwilson it may be simply down to which region the RDS instance is in. @jthele which region are your RDS instances in? Additionally which OS is the application accessing RDS running, not that I think that will be important but thought it worth asking.

Unfortunately the AWS docs are vague, simply stating that "you might need to use an intermediate". I do not believe it is down to MySQL version though, as that would not alter how TDS works particularly.

[dougwilson]

So part of the reason to understand the specifics here is that maybe the answer is just that the RDS instance itself is just misconfigured? Adding all the intermediates should not be necessary, since the root is trusted. Why they were added for the original is beyond me, and being a maintainer of the module and not even being able to answer that is a problem for ongoing maintenance and support. It also means that over time intermediates should (?) probably keep getting added as well, which is just more maintenance. And of course don't forget those previous disk bytes used by all the folks who will never use Amazon RDS :)

[dougwilson]

Anyway, I'm going to sleep now, so I won't be responding for some time, but to reiterate: I'm mainly looking to better understand the change here and what it means. If I stopped to get a better understanding in the original, then I likely would have realized the intermediates not being in this recent PR were missing. Just looking for the specific reason why they are required, not just for my understanding, but at the very least to make a somewhat helpful change log entry :)

[jthele]

Node: 10.17.0
AWS Region: us-east-1
OS: Ubuntu Bionic (from the ubuntu:bionic Docker repo as of this morning)

[jthele]

Why they were added for the original is beyond me

#1001 has some discussion on why the intermediates had to be added in 2015. But that looks like it was a different issue related to verification and UNABLE_TO_VERIFY_LEAF_SIGNATURE.

[dougwilson]

Right, I already looked up that PR :) but the question I have here has never been answered to my knowledge: why does only the root work for some people and not for others? What is the difference? Just having the root is much less work of course, and so I just want to know the answer if this (adding all the intermediates) is going to happen again, lol. Node.js support certificate chains perfectly fine as far as I know, which is the only reason the linked web page gives. So that would maybe leave that the server you are connecting to is maybe just misconfigured somehow?

[SimonHooker]

Based on what I saw when I was looking into issues with mssql / tedious, the nodejs tds implementation ends up at the OS level and so I can only jump to the conclusion that this somehow ends up being a host OS issue.

I am typically using the official node image as a base instead of ubuntu, currently node:10.16.0, which ends up at debian:stretch

[dougwilson]

I can't speak to mssql, but this module uses the Node.js native SSL, which is the specific OpenSSL that Node.js statically links into the binary; it does not use the host OS for SSL.

[jthele]

I just did an experiment. I stood up 3 brand new MySQL RDS instances in us-east-1 with versions 5.6.37, 5.7.26, and 8.0.16. I then connected to them from a Node 10.18.1 REPL running on my MacOS 10.15.2 laptop. This reproduced the error I saw before for 5.6.37. Both 5.7.26 and 8.0.16 succeeded without error and were queryable.

[jthele]

Some supporting evidence that suggests RDS 5.6 is not happy with chained certificates:

MySQL 5.6 documentation at https://dev.mysql.com/doc/refman/5.6/en/ssl-libraries.html says that

MySQL distributions compiled using yaSSL ... do not follow a chained certificate tree

And this Amazon blog entry at https://aws.amazon.com/blogs/database/selecting-the-right-encryption-options-for-amazon-rds-and-amazon-aurora-database-engines/ says that:

Furthermore, the cryptographic library used by RDS changed, transitioning with newer versions of the database engines to OpenSSL from yaSSL.

[CovertLeopard]

I need some more folks to better explain this and how after the 3 months of that PR this was never noticed and how, specifically this one is going to fix it and how we're going to verify it this time...

I'm using it with AWS Lambda Functions. v2.18.0 works with nodejs8.10, nodejs10.x and nodejs12.x lambda runtimes in the US-East-1 region when connecting to an Aurora MySQL RDS instance on MySQL 5.7.12 engine. It's a standard account, not GovCloud or any other specialty.

I've put thousands of requests against the DB since putting v2.18.0 in with no DB connectivity issues.

[CovertLeopard]

Some additional info that may help. I found this VERY OLD MySQL bug:
https://bugs.mysql.com/bug.php?id=54158

Distributions built with yaSSL do not because yaSSL does not look in any directory and does not follow a chained certificate tree. yaSSL requires that all components of the CA certificate tree be contained within a single CA certificate tree and that each certificate in the file has a unique SubjectName value. To work around this yaSSL limitation, concatenate the individual certificate files comprising the certificate tree into a new file. Then specify the new file as the value of the --ssl-capath option.

While that does not mention 5.6.x explicitly, it does mention yaSSL. Per AWS Documentation:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.SSLSupport

MySQL uses yaSSL for secure connections in the following versions:

MySQL version 5.7.19 and earlier 5.7 versions

MySQL version 5.6.37 and earlier 5.6 versions

MySQL version 5.5.57 and earlier 5.5 versions

Edit - Sorry I mis-clicked before finishing my comment.

[almercier]

If I overwrite the ssl profile in the config with the new root with or without these intermediates, I still get unable to get local issuer certificate when attempting to connect.

However if I instead overwrite using the old root and the old intermediates in the config, I'm able to connect without error...

Edit: @jthele was correct in that I had not updated the RDS instance yet.

[jthele]

If I overwrite the ssl profile in the config with the new root with or without these intermediates, I still get unable to get local issuer certificate when attempting to connect.

However if I instead overwrite using the old root and the old intermediates in the config, I'm able to connect without error...

Could you verify which certificate your RDS instance is currently using? Depending on when you created it and if you chose to use the defaults or not, you will have either rds-ca-2015 or rds-ca-2019. The new root and new intermediates only work with rds-ca-2019, and some of the old ones only work with rds-ca-2015.

A common pattern if you wish to transition is to specify certificates for both old and new so your client will work with both old and new configurations as you transition your database configuration.

[jthele]

Thanks @CovertLeopard for the sleuthing to find that AWS documentation of yaSSL versions:

MySQL uses yaSSL for secure connections in the following versions:
MySQL version 5.7.19 and earlier 5.7 versions
MySQL version 5.6.37 and earlier 5.6 versions
MySQL version 5.5.57 and earlier 5.5 versions

As I refresher, I had tested the following versions:

• 5.6.37: requires BOTH root and intermediate certificates to work
• 5.7.26: requires ONLY root certificate to work
• 8.0.16: requires ONLY root certificate to work

To further narrow in on the theory that yaSSL is the issue, I just tested a couple more versions on the other sides of the yaSSL dividing line. As before these were brand new MySQL RDS instances in us-east-1. I then connected to them from a Node 10.18.1 REPL running on my MacOS 10.15.2 laptop

• 5.6.39: requires ONLY root certificate to work
• 5.7.19: requires BOTH root and intermediate certificates to work

Those results align perfectly with the yaSSL boundaries that @CovertLeopard pointed out.

[dougwilson]

So to sum up what I believe is being said here: since the MySQL server uses it's own, bundled SSL library (just like Node.js) the way it works is dependent on the version. And apparently versions 5.6 (and likely earlier) simply only send the leaf certificate to the client during the Server Hello instead of the entire chain. Thus, in order to connect to those older MySQL server versions, the client needs to hold a copy of these intermediates so it can complete the chain of trust for verficiation. Does this sound like an accurate summary @jthele ?

[jthele]

@dougwilson

Yes, that sounds accurate to me. And to further clarify which versions are affected, my testing aligns with the versions specified in the AWS documentation that @CovertLeopard found:

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_MySQL.html#MySQL.Concepts.SSLSupport

[dougwilson]

Thanks @jthele I really appreciate it. I plan to merge & push in a patch release (so 2.18.1) today (i.e. within the next 8 hours).

[jthele]

I plan to merge & push in a patch release (so 2.18.1) today (i.e. within the next 8 hours).

Thanks for the quick turnaround on this PR. I appreciate it.

[CovertLeopard]

I plan to merge & push in a patch release (so 2.18.1) today (i.e. within the next 8 hours).

Thanks for the quick turnaround on this PR. I appreciate it.

Thanks to both of you! While this change doesn't impact our QA/Dev testing, our PROD instance is still on 5.6.10 and will require this. I appreciate the quick turnaround as well.

[alexkb]

mysqljs with RDS (ssl = 'Amazon RDS') had been working fine for us for the past year until we upgraded RDS and configured mysqljs
to use the new 2019 root certificate for the new version of RDS. Note, we're still on the old version of mysqljs (2.16.0) while this is all getting figured out.

However we're using the global root certificate and every so often the connection fails. We've now set debug: true in mysqljs to see more details as to why it's failing. We'll happily use the AWS-region specific certificate if it will fix things, but thought I'd mention the results here as it seems whatever you guys had for the 2015 certificates worked well.

[alexkb]

FYI: after turning debug on, the errors we were seeing were

ErrorPacket
Host 'x.x.x.x' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'' }

When we checked the max_connect_errors value it was set to 10, which seemed low, but as mentioned, we never had these issues before. So either:

1. Using the root certificate alone (without the intermediate) causes occasional connection failures, or
2. The upgraded version of RDS has somehow changed the max_connect_errors to be too low, or
3. The upgraded version of RDS has changed another setting which is causing errors more frequently hence the block error.

For now, we've set our connection config to use both the root and intermediate certificates, as well as increase max_connect_errors to 100:

credentials.ssl = {
  ca : [
    fs.readFileSync(__dirname + '/../../certs/rds-ca-2019-root.pem'),
    fs.readFileSync(__dirname + '/../../certs/rds-ca-2019-us-west-1.pem')
  ]
};

[jthele]

@alexkb

Regarding:

The upgraded version of RDS has somehow changed the max_connect_errors to be too low, or
The upgraded version of RDS has changed another setting which is causing errors more frequently hence the block error.

Was your RDS change isolated to altering the certificate authority from rds-ca-2015 to rds-ca-2019, or did you concurrently modify the engine version and/or the parameter group too? I would not expect an isolated certificate authority change to alter other RDS configuration parameters.

[alexkb]

@jthele

Was your RDS change isolated to altering the certificate authority from rds-ca-2015 to rds-ca-2019, or did you concurrently modify the engine version and/or the parameter group too? I would not expect an isolated certificate authority change to alter other RDS configuration parameters.

Thanks for the reply. We upgraded RDS from 5.5.59 to 5.5.61 which included the certificate change. When the change occurred we maintained the same version of mysqljs (and same mysql param group) but turned SSL off while we waited for an update to mysqljs. With SSL off we had occasional errors, so we switched SSL back on but started passing the new certificates to mysqljs assuming it was certificate related. It wasn't until we turned debug on, we saw it was related to max_connect_errors - hence the possibility that it's something else in this new version of RDS causing connection errors to occur occasionally.

[jthele]

@alexkb

If you saw similar connection errors with SSL off, then I'd be inclined to believe that the errors you are seeing are tied to the engine version change rather than the certificate change.

Further, I would be very surprised if the problem that I am addressing with this PR produces a sporadic issue like you describe. If the particular client and server pair you are working with are not processing the intermediate certificates (either because the server cannot process certificate chaining or the certificates are not provided in the client configuration), then certification validation should fail 100% of the time.

You might have better luck investigating what changed from 5.5.59 to 5.5.61:
https://dev.mysql.com/doc/relnotes/mysql/5.5/en/news-5-5-60.html
https://dev.mysql.com/doc/relnotes/mysql/5.5/en/news-5-5-61.html

I see a couple interesting things in there. 5.5.61 mentions a bug fix related to SSL connections, and 5.5.60 mentions a bug fix related to failed connections. That definitely suggests there were code changes related to connections introduced by the engine version change you applied.

[ewarren]

Should the ssl_profiles.js contain the contents of the file that is linked here:
https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem

[jthele]

@ewarren:

Perhaps. I concur there are differences between those 2 collections of certificates.

I sourced the ones documented in the section titled "Intermediate Certificates" because they unambiguously fit the problem I was trying to solve:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html

The global PEM is not as clearly documented, but I wouldn't be surprised if it had certs for other situations like GovCloud. You should be able to research the provenance of each certificate in the PEM. and determine if there are additional certificates that should be added.

[skilledDeveloper]

I had the same issue after rotating the certificate on my mysql instance.
I used the latest version with the fix but still getting HANDSHAKE_SSL_ERROR.
Am I missing something?

pkg version: 2.18.1
Node v12.x
MySQL engine version: 5.6.41

[jthele]

I had the same issue after rotating the certificate on my mysql instance.
I used the latest version with the fix but still getting HANDSHAKE_SSL_ERROR.

Does it work with the contents of the global PEM? As noted earlier, there are differences, and if you can show those differences are substantive than you'll have diagnosed where your issue lies.

[CovertLeopard]

We had this issue with MySQL 5.6.10 and nodejs12.x lambda runtime on aws. It turned out to be a node12 compatibility issue with that version of MySQL server. Particularly, MySQL 5.6 only supports TLS 1.0 while nodejs12.x disables that by default. We had to revert our lambda runtime to nodejs10.x. I hope this helps. 2.18.1 connecting to RDS MySQL 5.6.10a when run in Lambda functions using runtime nodejs10.x works for us, same config on nodejs12.x fails with an equivalent error to what you described.

…

On Sat, Feb 1, 2020, 3:34 PM Alan Zavari ***@***.***> wrote: I had the same issue. I used the latest version with the fix but still getting HANDSHAKE_SSL_ERROR. Am I missing something? pkg version: 2.18.1 Node v12.x MySQL engine version: 5.6.41 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2292?email_source=notifications&email_token=ACTRJJGRL3UUJRVJZPXIHALRAXMGZA5CNFSM4KKBTFKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKRGBCQ#issuecomment-581066890>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACTRJJCR6XTFRW6HZTJU5K3RAXMGZANCNFSM4KKBTFKA> .

[skilledDeveloper]

Thank you so much @CovertLeopard
Node version was indeed the issue and switching to v10 fixed it.
This saved me a lot of time and headache. 🙌

[CovertLeopard]

Thank you so much @CovertLeopard
Node version was indeed the issue and switching to v10 fixed it.
This saved me a lot of time and headaches.

Glad it helped. We found this out when updating our PROD servers last Thursday night due to a discrepancy in MySQL versions between our dev (5.7.12) where it worked fine with nodejs12 and our prod (5.6.10) where it failed. Luckily we were able to research the error and quickly find out that nodejs12 didn't leave TLS 1.0 enabled. Since we use lambda and can't pass arguments to node when it starts, we went back to nodejs10. If you have access to the actual server running node, you might be able to pass the flag in at node launch to enable TLS 1.0.

[alexkb]

If you're running in Lambda and you want to remain on to Node v12.x and connect to an older version of MySQL, you can get around the TLS issue by setting this environment variable: NODE_OPTIONS=--tls-min-v1.0. I've tried it with RDS MySQL 5.5.x and it fixes the exact issue. Credit goes to this post by a Matt Fuller. Thanks!