ECDSA verify fails on windows when R,S values both have leading zeroes.

[shane-kearns]

Initially observed that some ECDSA certificates are considered to have invalid signatures by certvalidator but which are valid according to windows certutil.exe and openssl.

The root cause is the marshalling of R, S values for calling BCryptVerifySignature here:

oscrypto/oscrypto/_win/asymmetric.py

Lines 2629 to 2649 in 1547f53

	else:
	# Windows doesn't use the ASN.1 Sequence for DSA/ECDSA signatures,
	# so we have to convert it here for the verification to work
	try:
	signature = DSASignature.load(signature).to_p1363()
	except (ValueError, OverflowError, TypeError):
	raise SignatureError('Signature is invalid')
	res = bcrypt.BCryptVerifySignature(
	certificate_or_public_key.key_handle,
	padding_info,
	digest,
	len(digest),
	signature,
	len(signature),
	flags
	)
	failure = res == BcryptConst.STATUS_INVALID_SIGNATURE
	failure = failure or res == BcryptConst.STATUS_INVALID_PARAMETER
	if failure:
	raise SignatureError('Signature is invalid')

Because oscrypto passes a 62 byte signature where a 64 byte signature is expected (in the case of NIST p256 keys), windows returns STATUS_INVALID_PARAMETER, which is treated as an invalid signature at line 2647.

The DSASignature.to_p1363 function does not know the expected key size:
https://github.com/wbond/asn1crypto/blob/b763a757bb2bef2ab63620611ddd8006d5e9e4a2/asn1crypto/algos.py#L720-L736
It can't, since the (EC)DSA signature ASN1 is just a sequence of two Integers. The context of what the signature algorithm and key size are is held at a higher level.

The required length can be determined from the public key used for verification.
e.g. certificate_or_public_key.byte_size is 32 for an ECDSA p256 key (since it's encoded as a compressed point), which is also the size of the private key and thereforre the required size for each of the R and S values.

Following script usually reproduces the problem within the 100k iterations

from asn1crypto.algos import DSASignature
from oscrypto import backend, platform
from oscrypto.asymmetric import ecdsa_sign, ecdsa_verify, generate_pair
from oscrypto.errors import SignatureError
import secrets

def check():
    public, private = generate_pair("ec", curve="secp256r1")
    data = secrets.token_bytes(128)
    signature = ecdsa_sign(private, data, "sha256")
    try:
        ecdsa_verify(public, signature, data, "sha256")
    except SignatureError:
        print("\n\nBad signature!")
        print("public key:", public.asn1.dump().hex())
        print("private key:", private.asn1.dump().hex())
        print("data:", data.hex())
        print("signature:", signature.hex())
        p1363 = DSASignature.load(signature).to_p1363()
        print(f"P1363 signature: {p1363.hex()} ({len(p1363)} bytes)")
        raise

print("testing on", platform.platform(), "with backend", backend())
for i in range(100000):
    if(i % 7800 == 0):
        print()
    if(i % 100 == 0):
        print('.', end="", flush=True)
    check()