System.Security.Cryptography.Pkcs test Decrypt_512_CekDoesNotDecrypt_FixedValue fails on Fedora 38

[akoeplinger]

Outerloop runs fail on Fedora 38:

https://helix.dot.net/api/2019-06-17/jobs/037f28b7-637f-4634-8ab1-3b4527ffc077/workitems/System.Security.Cryptography.Pkcs.Tests/console

  Discovering: System.Security.Cryptography.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Tests (found 56 of 4203 test cases)
  Starting:    System.Security.Cryptography.Pkcs.Tests (parallel test collections = on, max threads = 2)
    System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingCertWithPrivateKey.Decrypt_512_CekDoesNotDecrypt_FixedValue [FAIL]
      Assert.ThrowsAny() Failure: No exception was thrown
      Expected: typeof(System.Security.Cryptography.CryptographicException)
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography.Pkcs/tests/EnvelopedCms/DecryptTests.cs(148,0): at System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTests.Decrypt_512_CekDoesNotDecrypt_FixedValue()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

[ghost]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Outerloop runs fail on Fedora 38:

https://helix.dot.net/api/2019-06-17/jobs/037f28b7-637f-4634-8ab1-3b4527ffc077/workitems/System.Security.Cryptography.Tests/console

  Discovering: System.Security.Cryptography.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.Tests (found 56 of 4203 test cases)
  Starting:    System.Security.Cryptography.Pkcs.Tests (parallel test collections = on, max threads = 2)
    System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTestsUsingCertWithPrivateKey.Decrypt_512_CekDoesNotDecrypt_FixedValue [FAIL]
      Assert.ThrowsAny() Failure: No exception was thrown
      Expected: typeof(System.Security.Cryptography.CryptographicException)
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography.Pkcs/tests/EnvelopedCms/DecryptTests.cs(148,0): at System.Security.Cryptography.Pkcs.EnvelopedCmsTests.Tests.DecryptTests.Decrypt_512_CekDoesNotDecrypt_FixedValue()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Author:	akoeplinger
Assignees:	-
Labels:	area-System.Security, os-linux
Milestone:	-

[bartonjs]

Weird. This is about as deterministic as a crypto test gets (it uses a known key to decrypt a known value), and it clearly succeeded on retry since the log linked above shows success.

This feels like one of those "the computer said 2 + 2 = 5" never-reproducible bugs... but let's see if we get some more hits to try and force a repro.

[akoeplinger]

@bartonjs where are you seeing this succeed? It fails on every outerloop run on Fedora 38 across both mono and coreclr.

[bartonjs]

where are you seeing this succeed?

The log you linked to says no failures...

[bartonjs]

Ah. The log you originally linked to, which is what I got in the email notification. I see the current link has a failure.

[akoeplinger]

Yeah sorry, that was a copy-paste mistake :D

[vcsjones]

I can reproduce it on Fedora 39 but not Ubuntu 23.04. Odd.

[vcsjones]

Okay. @bartonjs and I tracked it down to openssl/openssl#13817. This is a change in OpenSSL 3.2 that Fedora / Red Hat back ported.

We are going to disable the implicit rejection so the APIs work as they have always worked.

[vcsjones]

Re-opening because we probably need to backport this.

[tomato42]

@vcsjones We have implemented it in OpenSSL specifically to protect users of such broken API as the one provided by C# and VB.NET.

By disabling implicit rejection you are MAKING them vulnerable

Please, see https://people.redhat.com/~hkario/marvin/ and read it whole, including both of the papers.