[Bug]: RA mode won't start up with Vault + SCEP #1560

tashian · 2023-09-26T21:18:08Z

Steps to Reproduce

Follow Vault RA mode setup docs.
Start up the RA and confirm it's working.
Add a SCEP provisioner by running step ca provisioner add scepca --type SCEP --challenge "abc123" --encryption-algorithm-identifier 2
Restart the RA.

Your Environment

OS -macOS
step-ca Version - 0.24.3-rc.5-144-gf9db22d3
step Version - CLI/0.24.2-rc.3-138-g01d4e9e8

Expected Behavior

I expected the RA to start up with a SCEP provisioner.

Actual Behavior

$ step-ca
badger 2023/09/26 14:14:37 INFO: All 1 tables opened in 0s
badger 2023/09/26 14:14:37 INFO: Replaying file id: 0 at offset: 8467
badger 2023/09/26 14:14:37 INFO: Replay took: 79.208µs
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/smallstep/certificates/authority.(*Authority).init(0x14000732280)
	/Users/carl/code/certificates/authority/authority.go:681 +0x28f0
github.com/smallstep/certificates/authority.New(0x140001d7180, {0x1400072aac0, 0x5, 0x1003869b8?})
	/Users/carl/code/certificates/authority/authority.go:140 +0xfc
github.com/smallstep/certificates/ca.(*CA).Init(0x1400072aa80, 0x140001d7180)
	/Users/carl/code/certificates/ca/ca.go:169 +0x520
github.com/smallstep/certificates/ca.New(0x140001d7180, {0x140006f3480, 0x7, 0x1029d8130?})
	/Users/carl/code/certificates/ca/ca.go:141 +0xe0
github.com/smallstep/certificates/commands.appAction(0x1400023f4a0)
	/Users/carl/code/certificates/commands/app.go:247 +0xcac
github.com/urfave/cli.HandleAction({0x1016efaa0?, 0x1019971c0?}, 0x5?)
	/Users/carl/go/pkg/mod/github.com/urfave/cli@v1.22.14/app.go:524 +0x58
github.com/urfave/cli.Command.Run({{0x101146044, 0x5}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0}, {0x1011c9454, ...}, ...}, ...)
	/Users/carl/go/pkg/mod/github.com/urfave/cli@v1.22.14/command.go:175 +0x4f0
main.main.func3(0x1400076ccd0?)
	/Users/carl/code/certificates/cmd/step-ca/main.go:201 +0x15c
github.com/urfave/cli.HandleAction({0x1016efaa0?, 0x140006f93e0?}, 0x1400012ca80?)
	/Users/carl/go/pkg/mod/github.com/urfave/cli@v1.22.14/app.go:524 +0x58
github.com/urfave/cli.(*App).Run(0x1400012ca80, {0x1400003a050, 0x1, 0x1})
	/Users/carl/go/pkg/mod/github.com/urfave/cli@v1.22.14/app.go:286 +0x548
main.main()
	/Users/carl/code/certificates/cmd/step-ca/main.go:204 +0x604

Additional Context

$ cat /Users/carl/.step/authorities/vault-test/config/ca.json
{
	"federatedRoots": null,
	"address": ":4443",
	"insecureAddress": ":3333",
	"dnsNames": [
		"localhost"
	],
	"logger": {
		"format": "text"
	},
	"db": {
		"type": "badgerv2",
		"dataSource": "/Users/carl/.step/authorities/vault-test/db",
		"badgerFileLoadingMode": ""
	},
	"authority": {
		"type": "vaultcas",
		"certificateAuthority": "http://127.0.0.1:8200",
		"certificateAuthorityFingerprint": "1edfb7fbdc0782f08893d972e359a7399a34b20d0dfcd446973ca461a265dfc7",
		"config": {
			"pkiMountPath": "pki_int1",
			"pkiRoleRSA": "rsa-role",
			"pkiRoleEC": "ec-role",
			"pkiRoleEd25519": "ed25519-role",
			"authType": "approle",
			"authMountPath": "approle",
			"namespace": "custom/vault/enterprise/namespace",
			"authOptions": {
				"roleID": "b9d4f032-4239-97b2-6198-1abe4c168bf5",
				"secretID": "82989f40-89a4-e77b-2810-a49149da33a1",
				"isWrappingToken": false
			}
		},
		"provisioners": [
			{
				"type": "JWK",
				"name": "carl@smallstep.com",
				"key": {
					"use": "sig",
					"kty": "EC",
					"kid": "UwxU8y4cw-Fg7gKbd8I76PYNiBYdGe6xlbCWcFH8atQ",
					"crv": "P-256",
					"alg": "ES256",
					"x": "u4kZJOGDIZnf2mY6963EffyHPMJAiZQnyvE1llpgZcM",
					"y": "jw4IAxplt49kz0WTcS6z7ji-YJuX-6gB6FNN6-sy8SA"
				},
				"encryptedKey": "eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjEwMDAwMCwicDJzIjoiY0otLWl1bXpCLWhCNlFOZHRSMWY1ZyJ9.GYOYWua8s5PVJXsqdwfffgI6CYpClRNlswtazvn62p29LQxRgJAItw.0Qgr4yGKCqJZemhD.nc7FzxBTsVafMlbuhdmITZZ_aJ9bOmsqrdWR6sdsqm1QhKjHIb4OdXuAb6z7JsBlsPsiv8yL4BNgsX2tH8vHE0lbU196De9rSEq8U7npr-pYO-WAy5ZG3Mdf8DPMyLVzg215fhH_kIoohDbuO1dxtqyVQuas2SEQ2RQe3maVMFbS9QfCjmuD8WiVIWfIsCyoybk4wvqFbq-7X9TRKJjIZLAqdvdAU5XkDG6VLJ0tapgub9A-HxyK_6TxtXwK8GTgox1rwTIteebReozm0gUj8dvEZRoYUZ0ADfBADww7A0EQPugG3OQ6r-MpD71B3kEkBDNQLZGR79Lu03m3x6I.VrkLEY7GFkcrpLAcuKIOPQ"
			},
			{
				"type": "ACME",
				"name": "acme"
			},
			{
				"type": "SCEP",
				"name": "scepca",
				"challenge": "abc123",
				"minimumPublicKeyLength": 2048,
				"encryptionAlgorithmIdentifier": 2,
				"options": {
					"x509": {},
					"ssh": {}
				},
				"claims": {
					"enableSSHCA": true,
					"disableRenewal": false,
					"allowRenewalAfterExpiry": false,
					"disableSmallstepExtensions": false
				}
			}
		],
		"template": {},
		"backdate": "1m0s"
	},
	"tls": {
		"cipherSuites": [
			"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
			"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
		],
		"minVersion": 1.2,
		"maxVersion": 1.3,
		"renegotiation": false
	},
	"commonName": "Step Online CA"
}

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

The text was updated successfully, but these errors were encountered:

maraino · 2023-09-27T02:33:13Z

There are a couple of problems here:

The panic is caused becausea.intermediateX509Certs is empty. This happens when VaultCAS and probably any other RA-CAS is used. The best fix for this would be to return Intermediates in the call to GetCertificateAuthority, currently, we only return the root.
The default signer and decrypter passed to the SCEP authority is based on the key of the ca.json (a.config.IntermediateKey), which, in this case, is empty. We should either not require this or wrap a signer and decrypter through the CAS. This might not be possible or quite difficult. So for RA's like VaultCAS, we might only support decrypters in the provisioners.

spieglt · 2023-09-27T15:31:33Z

This bug was uncovered by me as I was trying to use the SCEP provisioner to issue certificates through Vault. I'm exploring whether I'd be able to use ACME instead of SCEP. Considering that the problem is with GetCertificateAuthority, is this likely to affect the ACME provisioner similarly?

tashian · 2023-09-27T16:03:38Z

@spieglt in my testing, I was able to start up the RA with an ACME provisioner

spieglt · 2023-10-06T17:15:04Z

From @maraino's description, it sounds like the fix might be relatively straightforward? We're more or less blocked by this so I told the team I'd look into whether I could take a pass at fixing it. Not sure if it would interfere with @hslatman's work on SCEP he mentioned in Discord or whether the 0.25.1 milestone and his self-assignment of the issue is an indication that it's slated to be worked on in the near-term?

hslatman · 2023-10-09T09:21:06Z

Hey @spieglt, it's fairly high on my list of things I want to fix and closely related on things we're working on, so I hope I'll get to it sometime this week. But feel free to take a stab at it if you want it sooner or to be more sure about the timeline.

spieglt · 2023-10-09T20:39:23Z

Exciting news, thank you! I will not be able to work on it this week unfortunately.

spieglt · 2023-10-23T15:07:19Z

Hi @hslatman, any updates on this?

hslatman · 2023-10-24T11:42:16Z

Hey @spieglt, started looking into it, but not much actual progress yet due to other priorities. This week I should have some more time available for this and other SCEP things.

hslatman · 2023-11-14T11:33:46Z

Hey @spieglt, had to switch gears into product stuff, so haven't gotten to it yet. We're nearing the end of that, at least the most concrete and critical part, so I may get some time for this again soon.

spieglt · 2023-11-14T16:46:49Z

Thanks for the heads up. I'm finishing up another project and trying to switch focus back to this. We're eager to get this unblocked, is there anything I can do to help? I can try to fix it but am not sure of how likely my changes would be to conflict with changes you'd make in the near future.

spieglt · 2023-11-29T18:20:01Z

@hslatman I added a few lines to pass the intermediate certificates which has fixed the panic. Now I get an error thrown here saying failed to load softKMS: please define signingKeyPEM or signingKey, which seems to bring me to @maraino's point above:

The default signer and decrypter passed to the SCEP authority is based on the key of the ca.json (a.config.IntermediateKey), which, in this case, is empty. We should either not require this or wrap a signer and decrypter through the CAS. This might not be possible or quite difficult. So for RA's like VaultCAS, we might only support decrypters in the provisioners.

I feel like I can make changes to follow the "not require this" route, but I am concerned about the implication of "might only support decrypters" sentence. If the Vault RA only supports decrypters in the provisioners, does that mean I'd still be able to have step-ca issue certificates from Vault via SCEP? Can you tell me more about what signer and decrypter mean here exactly?

spieglt · 2023-11-29T18:51:25Z

After some reckless commenting-out of signer-related code and turning errors into print statements, I've gotten it to where step-ca actually starts with Vault as the RA and the SCEP provisioner in place. I get these log lines:

failed to load softKMS: please define signingKeyPEM or signingKey
no signer available for SCEP authority
2023/11/29 12:27:58 failed validating SCEP authority: SCEP provisioner "scepca" does not have a decrypter certificate

but everything else appears normal.

hslatman · 2023-11-29T19:16:40Z

@spieglt not at kb now, but the short answer is that the "decrypter approach" is probably the simplest way forward. It allows you to use an RSA certificate as the recipient (in SCEP, this would be the Registration Authority), so that messages can be encrypted to it by the client. The signer (intermediate) can then be either ECDSA or RSA, and you likely don't need more changes.

The RSA decrypter is configured as a PEM cert + key; currently no KMS URI is supported. If that's not an option, then you'll need changes to how the intermediate key is made available for both decryption and signing.

spieglt · 2023-11-29T19:36:14Z

Okay, unfortunately I don't understand much of that so I guess I have some more code to read. What is the "decrypter approach"? Having a working decrypter but not a working signer? And what is required for that? The Vault RA cert chain is RSA. How would I use the RSA decrypter configured as PEM cert + key while using Vault in RA mode? That question may not be coherent, apologies if so.

maraino · 2023-11-29T20:30:41Z

Okay, unfortunately I don't understand much of that so I guess I have some more code to read. What is the "decrypter approach"? Having a working decrypter but not a working signer? And what is required for that? The Vault RA cert chain is RSA. How would I use the RSA decrypter configured as PEM cert + key while using Vault in RA mode? That question may not be coherent, apologies if so.

What I think @hslatman suggests is configuring the SCEP provisioner with an RSA certificate and key. This option was added a few weeks ago. You can see the fields here:

certificates/authority/provisioner/scep.go

Lines 47 to 50 in 7bfe11c

    
           DecrypterCertificate []byte `json:"decrypterCertificate,omitempty"` 
        
           DecrypterKeyPEM      []byte `json:"decrypterKeyPEM,omitempty"` 
        
           DecrypterKeyURI      string `json:"decrypterKey,omitempty"` 
        
           DecrypterKeyPassword string `json:"decrypterKeyPassword,omitempty"`

If you're using step to configure the CA provisioner, step ca provisioner add has some extra flags to provide the decrypter, these are --scep-decrypter-certificate-file=<file> --scep-decrypter-key-file=<file>, --scep-decrypter-key-password-file=<file> and the still-experimental option --scep-decrypter-key-uri=<uri>.

I haven't tested this configuration with VaultCAS, but I think it will work.

hslatman · 2023-11-29T22:59:10Z

@spieglt basically what @maraino describes above.

In the SCEP protocol clients encrypt messages to the CA (usually an RA, Registration Authority; and the RA communicates with the CA using some protocol). The "decrypter approach" refers to a configuration in which each SCEP provisioner can have its own "RSA decrypter certificate". Clients can discover that decrypter certificate, and use it to encrypt messages to the public key. The SCEP provisioner decrypts the encrypted messages, incl. the CSR, and then makes the CAS sign the certificate using the intermediate key. In this setup, the SCEP provisioner with the decrypter cert operates as an RA in SCEP parlance; the CAS is the CA, and the protocol is effectively the CAS interface.

The reason that approach is likely to work and to be the simplest to support in your use case, is because the CAS interface does not need to provide a crypto.Decrypter (or a new function that provides for decryption), because the RSA decrypter certificate (+key) are used for that. So far, the CAS interface is focused on signing CSRs. But to support the SCEP protocol, it would need to support decryption operations too (or need an additional interface), if we were to use the same intermediate cert/key configured in the CAS for signing.

spieglt · 2023-11-30T16:42:36Z

Thanks for the explanations, that clears up a lot. Had not seen the new flags for creating the SCEP provisioner. Do the RSA key and cert used by the SCEP provisioner need to bear any relation to the cert chain used by Vault? Or is it just a completely separate chain?

hslatman · 2023-11-30T17:01:07Z

I don't think it's required (in all cases), but we've seen the best results with an RSA decrypter signed by the intermediate, so directly chaining to it. I remember seeing some source from Apple ecosystem requiring that; not sure about Microsoft/Linux clients, but the approach works there too.

spieglt · 2023-12-01T18:27:15Z

The steps you've described here will still require changes to the codebase though, yes?

If I follow the Vault RA and step-ca setup guides, cut an RSA cert from Vault, use it and its key with these new SCEP provisioner flags, I still get the initial panic described in this bug.

If I change a couple lines to pass around the intermediate certificates to GetCertificateAuthorityResponse, I get an error saying mismatch between signer certificate and public key.

If I run these lines from the Registration Authority Mode guide:

cat <<< $(jq 'del(.key, .crt, .root)' `step path`/config/ca.json) > $(step path)/config/ca.json
rm $(step path)/certs/root_ca.crt $(step path)/certs/intermediate_ca.crt $(step path)/secrets/intermediate_ca_key

I get errors related to the lack of a signer. If I turn a couple errors into print statements, and comment out a couple blocks of code related to signers, I can get step-ca to start.

And in that messy state, I was actually able to cut a cert from Vault via SCEP, which is what I've been trying to do for months now! So I will make a branch with my changes and link it here, but I will likely need your help making the signer optional. Does that make sense?

hslatman · 2023-12-01T22:54:15Z

@spieglt yes, just using the SCEP decrypter isn't enough. Some small changes are expected.

…s a Registration Authority with Vault, and as a SCEP provisioner, with the certificates provisioned by SCEP coming from the Vault CA. This is incomplete as I'm not sure how to properly make the signer optional.

spieglt · 2023-12-07T02:15:49Z

Here are the minimal changes required to use Vault as a Registration Authority with the SCEP provisioner: spieglt@b34dd57

To test, I:

followed the Smallstep Vault RA document with vault server -dev
added an insecure address to ca.json
made an RSA cert and key with Vault
used those with the --scep-decrypter-certificate-file and --scep-decrypter-key-file flags of step ca provisioner add scepca
deleted the step-ca-generated certs and keys, removed them from ca.json
compiled and ran this version of step-ca
acquired a cert via the certnanny/sscep client (changing KEYBITS to 4096 in mkrequest script)

When starting step-ca, the output looks like:

step-ca % go build && ./step-ca $(step path)/config/ca.json
badger 2023/12/06 19:36:49 INFO: All 1 tables opened in 1ms
badger 2023/12/06 19:36:49 INFO: Replaying file id: 0 at offset: 1467
badger 2023/12/06 19:36:49 INFO: Replay took: 59.916µs
failed to load softKMS: please define signingKeyPEM or signingKey
no signer available for SCEP authority
2023/12/06 19:36:49 Starting Smallstep CA/0000000-dev (darwin/arm64)
2023/12/06 19:36:49 Documentation: https://u.step.sm/docs/ca
2023/12/06 19:36:49 Community Discord: https://u.step.sm/discord
2023/12/06 19:36:49 Config file: /Users/username/.step/config/ca.json
2023/12/06 19:36:49 The primary server URL is https://localhost:8443/
2023/12/06 19:36:49 Root certificates are available at https://localhost:8443/roots.pem
2023/12/06 19:36:49 X.509 Root Fingerprint: [fingerprint]
2023/12/06 19:36:49 Serving HTTPS on [127.0.0.1:8443](http://127.0.0.1:8443/) ...
2023/12/06 19:36:49 Serving HTTP on :8080 ...

I'm not sure how to make the Signer optional in the proper way yet, please let me know if you have any insights there already. Thanks.

theron-idme · 2024-01-03T16:02:24Z

Hi @hslatman @tashian @maraino , happy new year. Any idea when you'll be able to work on this? Thanks.

tashian · 2024-01-03T16:49:52Z

Hi @theron-idme, happy new year! We will meet to discuss open source projects today and will get back to you shortly.

theron-idme · 2024-01-10T15:03:50Z

Hi @tashian, any word from that meeting?

hslatman · 2024-01-10T15:44:55Z

Hey @theron-idme, I'm going to look at it, but other priorities atm.

charles-low · 2024-01-16T08:18:37Z

Hi, I encountered similar issue and this thread is extremely helpful. I tried the following and is able to setup SCEP with a step-ca only setup.

Create a SCEP decrypter certificate with RSA key

step certificate create scep-decrypter scep-decrypter.crt scep-decrypter.key \
             --ca .step/certs/intermediate_ca.crt --ca-key .step/secrets/intermediate_ca_key --kty RSA --size 2048

Update the SCEP config to include the decrypter cert and key

step ca provisioner update scep --scep-decrypter-certificate-file=./scep-decrypter.crt --scep-decrypter-key-file=./scep-decrypter.key  --scep-decrypter-key-password-file=.step/secrets/root_ca_key

make sure the insecureAddress in ca.json is set as ":80"

tashian added bug needs triage Waiting for discussion / prioritization by team labels Sep 26, 2023

tashian changed the title ~~[Bug]: CA won't start up with Vault + SCEP~~ [Bug]: RA mode won't start up with Vault + SCEP Sep 26, 2023

maraino added this to the v0.25.1 milestone Sep 27, 2023

hslatman self-assigned this Oct 2, 2023

hslatman modified the milestones: v0.25.1, v0.25.2 Nov 29, 2023

birdie1 mentioned this issue Feb 19, 2024

[Bug]: Get a certificate via SCEP fails with: scep post request failed: crypto/rsa: verification error #1723

Open

hslatman modified the milestones: v0.26.0, v0.26.1 Mar 29, 2024

hslatman linked a pull request Apr 18, 2024 that will close this issue

Fix CA startup with Vault RA configuration #1803

Merged

hslatman closed this as completed in #1803 Apr 22, 2024

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: RA mode won't start up with Vault + SCEP #1560

[Bug]: RA mode won't start up with Vault + SCEP #1560

tashian commented Sep 26, 2023

maraino commented Sep 27, 2023 •

edited

Loading

spieglt commented Sep 27, 2023

tashian commented Sep 27, 2023 •

edited

Loading

spieglt commented Oct 6, 2023

hslatman commented Oct 9, 2023 •

edited

Loading

spieglt commented Oct 9, 2023

spieglt commented Oct 23, 2023

hslatman commented Oct 24, 2023

hslatman commented Nov 14, 2023

spieglt commented Nov 14, 2023

spieglt commented Nov 29, 2023 •

edited

Loading

spieglt commented Nov 29, 2023

hslatman commented Nov 29, 2023

spieglt commented Nov 29, 2023

maraino commented Nov 29, 2023

hslatman commented Nov 29, 2023

spieglt commented Nov 30, 2023

hslatman commented Nov 30, 2023

spieglt commented Dec 1, 2023

hslatman commented Dec 1, 2023

spieglt commented Dec 7, 2023

theron-idme commented Jan 3, 2024

tashian commented Jan 3, 2024

theron-idme commented Jan 10, 2024

hslatman commented Jan 10, 2024

charles-low commented Jan 16, 2024

[Bug]: RA mode won't start up with Vault + SCEP #1560

[Bug]: RA mode won't start up with Vault + SCEP #1560

Comments

tashian commented Sep 26, 2023

Steps to Reproduce

Your Environment

Expected Behavior

Actual Behavior

Additional Context

Contributing

maraino commented Sep 27, 2023 • edited Loading

spieglt commented Sep 27, 2023

tashian commented Sep 27, 2023 • edited Loading

spieglt commented Oct 6, 2023

hslatman commented Oct 9, 2023 • edited Loading

spieglt commented Oct 9, 2023

spieglt commented Oct 23, 2023

hslatman commented Oct 24, 2023

hslatman commented Nov 14, 2023

spieglt commented Nov 14, 2023

spieglt commented Nov 29, 2023 • edited Loading

spieglt commented Nov 29, 2023

hslatman commented Nov 29, 2023

spieglt commented Nov 29, 2023

maraino commented Nov 29, 2023

hslatman commented Nov 29, 2023

spieglt commented Nov 30, 2023

hslatman commented Nov 30, 2023

spieglt commented Dec 1, 2023

hslatman commented Dec 1, 2023

spieglt commented Dec 7, 2023

theron-idme commented Jan 3, 2024

tashian commented Jan 3, 2024

theron-idme commented Jan 10, 2024

hslatman commented Jan 10, 2024

charles-low commented Jan 16, 2024

maraino commented Sep 27, 2023 •

edited

Loading

tashian commented Sep 27, 2023 •

edited

Loading

hslatman commented Oct 9, 2023 •

edited

Loading

spieglt commented Nov 29, 2023 •

edited

Loading