enhancement: issuing certificates via a pki backend without storing them?

[slingamn]

My use case is something like this. I'm using Vault's PKI backend to issue certificates. I'm relying on Vault's ability to protect the signing certificate at rest, and also on its ability to enforce allowed_domains for the leaf certificates. However, because all the allowed_domains are non-sensitive, there is never a need to revoke an issued certificate, or to list the currently valid certificates. Meanwhile, I'm experiencing performance problems with my storage backend when many certificates are issued near-simultaneously, or a very large number of certificates are simultaneously valid.

It would help a lot if I could tell Vault "issue a certificate according to role X, but don't store it". I can see a few implementation possibilities:

1. Adding a no_store property of the role that can be set at role creation time (via an optional parameter to the /pki/roles endpoint)
2. Adding an allow_forgetting property to the role that's set at creation time, then adding an optional forget parameter to the /pki/issue endpoint

Would this contradict Vault's design goals? If not, would you be interested in a patch implementing one of these solutions? (Alternately, maybe you can see a way to avoid the problem entirely? :-)

Thanks for your time!

[jefferai]

Hi @slingamn ,

Which storage backend and what do you consider "very large"? What performance problems are you seeing? Also, what version of Vault?

You can remove expired certificates via hitting the pki/tidy path with tidy_cert_store set to true which may suffice or may not depending on the answers to those questions.

[slingamn]

This is Vault 0.6.2 with Zookeeper as the storage backend. I'm using pki/tidy and I've adjusted certificate lifetimes in order to reduce the expected number of certificates valid at any given time. Unfortunately, there are still cases where enough certificates are issued that they cause problems. I don't have hard numbers right now, but anecdotally Vault and ZK scale fine to 10^5 certificates, then start having issues (including excessive memory consumption from Vault, which I didn't mention before) around 10^6.

[jefferai]

Prior to Vault 0.7.0 the PKI backend issues leases for certs, which requires, in addition to storing the certs, keeping lease entries in storage and tracking state in memory. We made them unleased by default in 0.7.0 (although you need to rewrite the role, see upgrade notes) for exactly this kind of use-case, so upgrading will likely help.

I should also note that anecdotal evidence suggests (but I don't have direct experience with) that ZK is one of the worst storage layers to use in terms of scaling. You may be required to stick with it but other backends will likely scale better.

[slingamn]

Thanks very much! I'll give that a try.

[slingamn]

Thanks very much for the suggestion --- the issues with memory consumption are gone under 0.7.0.

However:

1. Zookeeper still starts to complain around the 10^6 mark
2. The ability to issue certificates without storing them would be useful for other reasons. For example, there would be no need to tune the certificate lifetime based on the expected rate of certificate issuance. The pki/tidy job is also an extra moving part that needs to be scheduled, monitored, etc.

Would a no-store option be problematic for security reasons, or because of the added code complexity?

[jefferai]

It's not really problematic at all although it would make it tricky or impossible to support CRLs. Mostly just a matter of time.

[slingamn]

I can see a couple options:

1. The most straightforward thing is to disallow revocation of certificates issued from no_store roles. This seems pretty easy to implement --- it's the behavior we'd get by simply omitting the store operation. Revocation would then fail this check.
2. pki/revoke could take the entire certificate to be revoked, in PEM format, as an optional argument. Then if the serial is not found in the store, we could check the certificate's signature against the backend's CA. If it's valid, we could recreate the entry in the store, then use the normal revocation logic.

1 seems viable by itself with 2 left as a follow-up. Thoughts?

[jefferai]

I think (1) is fine and not convinced (2) is ever necessary, so (1) is definitely fine by itself.

[chrishoffman]

Fixed in #2565