Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Security.Cryptography.X509Certificate / System.Net.Security tests intermittently crashing on Windows #30000

Closed
stephentoub opened this issue Jun 24, 2019 · 9 comments
Closed

System.Security.Cryptography.X509Certificate / System.Net.Security tests intermittently crashing on Windows #30000

stephentoub opened this issue Jun 24, 2019 · 9 comments
Labels
area-System.Security os-windows untriaged New issue has not been triaged by the area owner
Milestone
5.0.0

Comments

@stephentoub
Copy link
Member

e.g.
https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38807~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190624.5/workItem/System.Net.Security.Tests/wilogs

C:\dotnetbuild\work\9e863110-2c57-4295-9acd-965add455f98\Work\a2f6e964-b4fc-4536-ad18-e5fd7fc5df75\Exec>"C:\dotnetbuild\work\9e863110-2c57-4295-9acd-965add455f98\Payload\dotnet.exe" xunit.console.dll System.Net.Security.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonwindowstests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Net.Security.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Security.Tests (found 267 of 275 test cases)
  Starting:    System.Net.Security.Tests (parallel test collections = on, max threads = 4)
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_SelfSignedClientEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ServerEKUClientAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_ClientEKUServerAuth_Fails [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUServerAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.SslStreamEKUTest.SslStream_NoEKUClientAuth_Ok [SKIP]
      Condition(s) not met: "IsRootCertificateInstalled"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyTls13CipherSuiteAllowedButChosenProtocolsDoesNotAllowIt_Fails [SKIP]
      Condition(s) not met: "CipherSuitesPolicyAndTls13Supported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyNonTls13CipherSuiteAllowedButOtherSideDoesNotAllowIt_Fails [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowSameTwoOnBothSidesLessPreferredIsTls13_Success [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.NegotiatedCipherSuite_SslProtocolIsTls13_ShouldBeTls13 [SKIP]
      Condition(s) not met: "Tls13Supported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowSomeCipherSuitesWithNoEncryptionOption_Fails [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyNonTls13CipherSuitesAllowedButChosenProtocolDoesNotAllowIt_Fails [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_ThreeCipherSuitesWithTwoOverlapping_Success [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowedCipherSuitesIncludesSubsetOfInput_Success [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowOneOnOneSideTls13_Success [SKIP]
      Condition(s) not met: "CipherSuitesPolicyAndTls13Supported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowTwoOnBothSidesWithSingleOverlapNonTls13_Success [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_TwoCipherSuitesWithAllOverlapping_Success [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.Tls13IsSupported_GetValue_ReturnsTrue [SKIP]
      Condition(s) not met: "IsKnownPlatformSupportingTls13"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_AllowTwoOnBothSidesWithNoOverlapNonTls13_Fails [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_OnlyTls13CipherSuiteAllowedOtherSideDoesNotAllowTls13_Fails [SKIP]
      Condition(s) not met: "CipherSuitesPolicyAndTls13Supported"
    System.Net.Security.Tests.NegotiatedCipherSuiteTest.CipherSuitesPolicy_NothingAllowed_Fails [SKIP]
      Condition(s) not met: "CipherSuitesPolicySupported"
----- end 24/06/2019 12:57:08,15 ----- exit code -1073740791 ----------------------------------------------------------

That exit code maps to STATUS_STACK_BUFFER_OVERRUN.

cc: @davidsh, @bartonjs
@stephentoub
Copy link
Member Author

Another, this time in System.Security.Cryptography.X509Certificates:
https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38808~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190624.6/workItem/System.Security.Cryptography.X509Certificates.Tests/wilogs

C:\dotnetbuild\work\36c2c556-0e1e-4fb8-9361-2c78a29b348b\Work\ea3d0ecc-3ba4-41a8-9e24-05cb4a9c82bf\Exec>"C:\dotnetbuild\work\36c2c556-0e1e-4fb8-9361-2c78a29b348b\Payload\dotnet.exe" xunit.console.dll System.Security.Cryptography.X509Certificates.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonwindowstests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Security.Cryptography.X509Certificates.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.X509Certificates.Tests (found 449 of 467 test cases)
  Starting:    System.Security.Cryptography.X509Certificates.Tests (parallel test collections = on, max threads = 4)
----- end 24/06/2019 13:00:42,15 ----- exit code -1073740791 ----------------------------------------------------------

@stephentoub
Copy link
Member Author

stephentoub commented Jun 24, 2019 •
edited
Loading

This also looks to be the same:

dotnet/corefx#38779 (comment)

https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38779~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190622.10/workItem/System.Security.Cryptography.X509Certificates.Tests/wilogs

@stephentoub stephentoub changed the title System.Net.Security tests intermittently crashing on Windows System.Security.Cryptography.X509Certificate / System.Net.Security tests intermittently crashing on Windows Jun 24, 2019
@stephentoub
Copy link
Member Author

Again here:
https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F38819~2Fmerge/test~2Ffunctional~2Fcli~2Finnerloop~2F/20190624.20/workItem/System.Security.Cryptography.X509Certificates.Tests/wilogs

C:\dotnetbuild\work\4bc4aaff-7e1b-4fb8-be19-0cc5b1582588\Work\116dfdb0-94bb-46ad-a763-2966bbdcbfdd\Exec>"C:\dotnetbuild\work\4bc4aaff-7e1b-4fb8-be19-0cc5b1582588\Payload\dotnet.exe" xunit.console.dll System.Security.Cryptography.X509Certificates.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonwindowstests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Security.Cryptography.X509Certificates.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.X509Certificates.Tests (found 449 of 467 test cases)
  Starting:    System.Security.Cryptography.X509Certificates.Tests (parallel test collections = on, max threads = 4)
----- end 24/06/2019 17:10:06,29 ----- exit code -1073740791 ----------------------------------------------------------

@bartonjs
Copy link
Member

I have this running in a loop on two computers with no repro.

  • Windows Server 2012 R2 (1454 iterations as of this post)
  • Windows 10 Client 1903 (19H1) (1284 iterations as of this post)

They're (theoretically) configured to hook the debugger on a non-zero exit (via gflags). They'll stop making the room warmer at 10,000 runs.

@bartonjs
Copy link
Member

  • Windows Server 2012 R2 has hit its 10k and given up.
  • Windows 10 Client 1903 has done about 4000 iterations (it accidentally spent a day hibernated).

Without a crash dump or a stable repro there's not much to investigate or fix.

Closing as non-actionable.

@stephentoub
Copy link
Member Author

Failed again here:
https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F39262~2Fmerge/test~2Ffunctional~2Fcli~2F~2F/20190707.4/workItem/System.Security.Cryptography.X509Certificates.Tests/wilogs

C:\dotnetbuild\work\7d425377-8076-4118-8626-6a78120df760\Work\c40d8c2a-0de2-48ab-ae0d-c8e8ca69e376\Exec>"C:\dotnetbuild\work\7d425377-8076-4118-8626-6a78120df760\Payload\dotnet.exe" xunit.console.dll System.Security.Cryptography.X509Certificates.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonwindowstests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Security.Cryptography.X509Certificates.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.X509Certificates.Tests (found 451 of 469 test cases)
  Starting:    System.Security.Cryptography.X509Certificates.Tests (parallel test collections = on, max threads = 4)
----- end 07/07/2019 13:19:19,45 ----- exit code -1073740791 ----------------------------------------------------------

Maybe there's a dump available?

@stephentoub stephentoub reopened this Jul 7, 2019
@ViktorHofer
Copy link
Member

Failed again: https://dnceng.visualstudio.com/public/_build/results?buildId=333064&view=ms.vss-test-web.build-test-results-tab&runId=9774138&resultId=178845&paneView=attachments

We have a dump for it: https://helix.dot.net/api/2019-06-17/jobs/113bfa51-e910-4ebd-9deb-b93557fe5ca3/workitems/System.Net.Security.Tests/files/dotnet.exe.2836.dmp

@ViktorHofer
Copy link
Member

ViktorHofer commented Aug 31, 2019 •
edited
Loading 

Unhandled exception at 0x00007FFD6655B3EF (ntdll.dll) in dotnet.exe.2836.dmp: RangeChecks instrumentation code detected an out of range array access.

>	System.Security.Cryptography.X509Certificates.dll!Internal.Cryptography.Pal.CertificatePal.FilterPFXStore(byte[] rawData, Microsoft.Win32.SafeHandles.SafePasswordHandle password, Internal.Cryptography.Pal.Native.PfxCertStoreFlags pfxCertStoreFlags)	Unknown
 	System.Security.Cryptography.X509Certificates.dll!Internal.Cryptography.Pal.CertificatePal.FromBlobOrFile(byte[] rawData, string fileName, Microsoft.Win32.SafeHandles.SafePasswordHandle password, System.Security.Cryptography.X509Certificates.X509KeyStorageFlags keyStorageFlags)	Unknown
 	System.Security.Cryptography.X509Certificates.dll!Internal.Cryptography.Pal.CertificatePal.FromBlob(byte[] rawData, Microsoft.Win32.SafeHandles.SafePasswordHandle password, System.Security.Cryptography.X509Certificates.X509KeyStorageFlags keyStorageFlags)	Unknown
 	System.Security.Cryptography.X509Certificates.dll!System.Security.Cryptography.X509Certificates.X509Certificate.X509Certificate(byte[] data)	Unknown
 	System.Net.Security.Tests.dll!System.Net.Security.Tests.SslStreamStreamToStreamTest.SslStream_StreamToStream_Authentication_Success_MemberData()	Unknown

@msftgits msftgits transferred this issue from dotnet/corefx Feb 1, 2020
@msftgits msftgits added this to the 5.0 milestone Feb 1, 2020
@maryamariyan maryamariyan added the untriaged New issue has not been triaged by the area owner label Feb 23, 2020
@stephentoub
Copy link
Member Author

Closing as I don't believe we've seen this recently. Re-open if it happens again.

@ghost ghost locked as resolved and limited conversation to collaborators Dec 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Security os-windows untriaged New issue has not been triaged by the area owner
Projects
None yet
Milestone
5.0.0
Development

No branches or pull requests
5 participants
@stephentoub @maryamariyan @ViktorHofer @msftgits @bartonjs