OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported test failure

[jkotas]

    System.Security.Cryptography.Tests.OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported [FAIL]
      Assert.Throws() Failure: Exception type was not an exact match
      Expected: typeof(System.PlatformNotSupportedException)
      Actual:   typeof(Interop+Crypto+OpenSslCryptographicException)
      ---- Interop+Crypto+OpenSslCryptographicException : error:07880025:common libcrypto routines::reason(524325)
      Stack Trace:
        /_/src/libraries/System.Security.Cryptography/tests/OpenSslNamedKeysTests.manual.cs(103,0): at System.Security.Cryptography.Tests.OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(57,0): at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=784071
Build error leg or test failing: System.Security.Cryptography.Tests.OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported
Pull request: #106763

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["System.Security.Cryptography.Tests.OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported [FAIL]","common libcrypto routines::reason"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=784071
Error message validated: [System.Security.Cryptography.Tests.OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported [FAIL] common libcrypto routines::reason]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/22/2024 1:13:16 AM UTC

Report

Build	Definition	Test	Pull Request
784499	dotnet/runtime	System.Security.Cryptography.Tests.OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported	#106779
784475	dotnet/runtime	System.Security.Cryptography.Tests.OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported	#106777
784071	dotnet/runtime	System.Security.Cryptography.Tests.OpenSslNamedKeysTests.ProvidersNotSupported_ThrowsPlatformNotSupported	#106763

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
3	3	3

[vcsjones]

@krwq looks like it could be related to #106397

[krwq]

This is most likely a test issue, I'm mostly curious on why only some osx builds are affected - will be going through logs to see for any clues what's special about this particular configuration and why PlatformDetection.IsOpenSsl3 is not sufficient for detection

[krwq]

It turns out that IsOpenSsl3 always gives false on OSX but in this case the providers APIs were still available which suggests OpenSSL 3 was installed. Since I'm not sure how to write that check reliably and I don't have OSX to play around (some OSX-es seem to be missing OpenSSL versions APIs) and this issue is a test bug it doesn't seem worth to spend too much time on this and might be better to just relax the test a little bit on Apple platforms. @vcsjones @bartonjs I'm open on suggestions on improving this detection on tests side but it's hard to blindly fix this without OSX so I'll start with unblocking the CI.

[jkotas]

Do you understand why this failure does not happen on every PR? Can this issue be a sign of uninitialized variable or some other non-determinism like that?

[bartonjs]

runtime/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.Unix.cs

Lines 53 to 54 in a82cbec

	public static bool IsOpenSsl3 => !IsApplePlatform && !IsWindows && !IsAndroid && !IsBrowser ?
	GetOpenSslVersion() >= s_openssl3Version :

Should probably be

public static bool IsOpenSsl3 => OpenSslPresentOnSystem && GetOpenSslVersion() >= s_openssl3Version;

No idea if that'll break anything by making tests think they're Linux because we did a waterfall in the wrong order

[vcsjones]

No idea if that'll break anything by making tests think they're Linux because we did a waterfall in the wrong order

Probably here

runtime/src/libraries/System.Security.Cryptography/tests/AsnEncodedDataTests.cs

Lines 114 to 119 in a82cbec

	bool isOpenSsl3 = PlatformDetection.IsOpenSsl3;
	string expected = string.Join(
	", ",
	// Choice[0]: OtherName
	isOpenSsl3 ? "othername: UPN::subjectupn1@example.org" : "othername:<unsupported>",

[bartonjs]

Fair, fair. I guess really there's "is OpenSSL the main crypto platform" (Linux, FreeBSD, etc) vs "is OpenSSL available?" (all previous things, plus maybe macOS); and very few things want the latter... this just happens to be one of them (along with "should RSAOpenSsl be tested?" and friends)

[bartonjs]

Do you understand why this failure does not happen on every PR? Can this issue be a sign of uninitialized variable or some other non-determinism like that?

I haven't looked into it, but my off the cuff response is that perhaps not all of the CI machines are in the same state w/r/t OpenSSL, and some of them are throwing PNSE "I can't find OpenSSL 3", and some of them had OpenSSL 3 and thus got "I did that thing you asked, and it didn't work" (CryptographicException)