[tests] Fix certificate tests after Apple started blocking the Thawte certificate. #19650

rolfbjarne · 2023-12-15T13:31:47Z

The “Thawte SGC CA” certificate we're using for testing is now blocked by
Apple, evaluating trust results in this error:

“Thawte SGC CA” certificate is blocked

So update the mail_google_com certificates to what mail.google.com serves now, and also update the corresponding parent certificates (which have changed).

Then adjust the tests accordingly.

Finally delete all unused certificates from the code.

Fixes these test failures:

MonoTouchFixtures.Security.TrustTest
    [FAIL] Trust_FullChain :   GetTrustResult-2
        Expected: RecoverableTrustFailure
        But was:  FatalTrustFailure
           at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389
           at MonoTouchFixtures.Security.TrustTest.Trust_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 321
    [FAIL] Trust2_FullChain :   GetTrustResult-2
        Expected: RecoverableTrustFailure
        But was:  FatalTrustFailure
           at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389
           at MonoTouchFixtures.Security.TrustTest.Trust2_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 449

… certificate. The “Thawte SGC CA” certificate we're using for testing is now blocked by Apple, evaluating trust results in this error: > “Thawte SGC CA” certificate is blocked So update the certificates to today: the mail_google_com certificate, and add the corresponding two parent certificates (including the root certificate). Then adjust the tests accordingly. Finally delete all unused certificates from the code. Fixes these test failures: MonoTouchFixtures.Security.TrustTest [FAIL] Trust_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 321 [FAIL] Trust2_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust2_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 449

vs-mobiletools-engineering-service2 · 2023-12-15T14:56:43Z

📚 [PR Build] Artifacts 📚

Packages generated

View packages

Pipeline on Agent
Hash: [PR build]

vs-mobiletools-engineering-service2 · 2023-12-15T15:00:35Z

💻 [CI Build] Windows Integration Tests passed 💻

✅ All Windows Integration Tests passed.

Pipeline on Agent
Hash: 02db1f9a7ab4cae074e990f070397bf9578fbcf6 [PR build]

vs-mobiletools-engineering-service2 · 2023-12-15T16:09:07Z

💻 [CI Build] Windows Integration Tests passed 💻

✅ All Windows Integration Tests passed.

Pipeline on Agent
Hash: 02db1f9a7ab4cae074e990f070397bf9578fbcf6 [PR build]

vs-mobiletools-engineering-service2 · 2023-12-15T16:11:00Z

📚 [PR Build] Artifacts 📚

Packages generated

View packages

Pipeline on Agent
Hash: [PR build]

vs-mobiletools-engineering-service2 · 2023-12-15T16:11:40Z

💻 [CI Build] Tests on macOS M1 - Mac Ventura (13.0) passed 💻

✅ All tests on macOS M1 - Mac Ventura (13.0) passed.

Pipeline on Agent
Hash: [PR build]

vs-mobiletools-engineering-service2 · 2023-12-15T16:12:08Z

💻 [CI Build] Tests on macOS M1 - Mac Big Sur (11.5) passed 💻

✅ All tests on macOS M1 - Mac Big Sur (11.5) passed.

Pipeline on Agent
Hash: [PR build]

vs-mobiletools-engineering-service2 · 2023-12-15T16:24:28Z

✅ API diff for current PR / commit

Legacy Xamarin (No breaking changes)

~~iOS~~ (no change detected)
~~tvOS~~ (no change detected)
~~watchOS~~ (no change detected)
~~macOS~~ (no change detected)

NET (empty diffs)

~~iOS~~: (empty diff detected)
~~tvOS~~: (empty diff detected)
~~MacCatalyst~~: (empty diff detected)
~~macOS~~: (empty diff detected)

✅ API diff vs stable

Legacy Xamarin (No breaking changes)

iOS: vsdrops gist (No breaking changes)
tvOS: vsdrops gist (No breaking changes)
watchOS: vsdrops gist (No breaking changes)
macOS: vsdrops gist (No breaking changes)

.NET (No breaking changes)

iOS: vsdrops gist (No breaking changes)
tvOS: vsdrops gist (No breaking changes)
MacCatalyst: vsdrops gist (No breaking changes)
macOS: vsdrops gist (No breaking changes)
Microsoft.iOS vs Microsoft.MacCatalyst: vsdrops gist

Legacy Xamarin (stable) vs .NET

iOS: vsdrops gist
tvOS: vsdrops gist
macOS: vsdrops gist

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: 02db1f9a7ab4cae074e990f070397bf9578fbcf6 [PR build]

vs-mobiletools-engineering-service2 · 2023-12-15T18:57:07Z

🔥 [CI Build] Test results 🔥

Test results

❌ Tests failed on VSTS: simulator tests

0 tests crashed, 1 tests failed, 234 tests passed.

Failures

❌ monotouch tests

1 tests failed, 40 tests passed.

monotouch-test/Mac [dotnet]/Release [dotnet]: Failed (Test run failed.
Tests run: 2777 Passed: 2685 Inconclusive: 3 Failed: 1 Ignored: 91)

Html Report (VSDrops) Download

Successes

✅ bcl: All 69 tests passed. Html Report (VSDrops) Download
✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests: All 1 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 7 tests passed. Html Report (VSDrops) Download
✅ framework: All 8 tests passed. Html Report (VSDrops) Download
✅ generator: All 2 tests passed. Html Report (VSDrops) Download
✅ interdependent_binding_projects: All 7 tests passed. Html Report (VSDrops) Download
✅ install_source: All 1 tests passed. Html Report (VSDrops) Download
✅ introspection: All 8 tests passed. Html Report (VSDrops) Download
✅ linker: All 65 tests passed. Html Report (VSDrops) Download
✅ mac_binding_project: All 1 tests passed. Html Report (VSDrops) Download
✅ mmp: All 2 tests passed. Html Report (VSDrops) Download
✅ mononative: All 6 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ mtouch: All 1 tests passed. Html Report (VSDrops) Download
✅ xammac: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 8 tests passed. Html Report (VSDrops) Download
✅ xtro: All 2 tests passed. Html Report (VSDrops) Download

Pipeline on Agent
Hash: 02db1f9a7ab4cae074e990f070397bf9578fbcf6 [PR build]

rolfbjarne · 2023-12-18T07:43:56Z

Test failure is unrelated: https://github.com/xamarin/maccore/issues/868

rolfbjarne · 2023-12-18T13:14:00Z

/sudo backport release/8.0.1xx

vs-mobiletools-engineering-service2 · 2023-12-18T13:14:30Z

Backport Job to branch release/8.0.1xx Created! The magic is happening here

vs-mobiletools-engineering-service2 · 2023-12-18T13:16:35Z

Oh no! Backport failed! Please see https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=8826430 for more details.

…locking the Thawte certificate. The “Thawte SGC CA” certificate we're using for testing is now blocked by Apple, evaluating trust results in this error: > “Thawte SGC CA” certificate is blocked So update the mail_google_com certificates to what mail.google.com serves now, and also update the corresponding parent certificates (which have changed). Then adjust the tests accordingly. Finally delete all unused certificates from the code. Fixes these test failures: MonoTouchFixtures.Security.TrustTest [FAIL] Trust_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 321 [FAIL] Trust2_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust2_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 449 Backport of xamarin#19650.

…locking the Thawte certificate. (#19665) The “Thawte SGC CA” certificate we're using for testing is now blocked by Apple, evaluating trust results in this error: > “Thawte SGC CA” certificate is blocked So update the mail_google_com certificates to what mail.google.com serves now, and also update the corresponding parent certificates (which have changed). Then adjust the tests accordingly. Finally delete all unused certificates from the code. Fixes these test failures: MonoTouchFixtures.Security.TrustTest [FAIL] Trust_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 321 [FAIL] Trust2_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust2_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 449 Backport of #19650.

rolfbjarne · 2024-01-12T07:10:57Z

/sudo backport release/7.0.3xx

vs-mobiletools-engineering-service2 · 2024-01-12T07:11:28Z

Backport Job to branch release/7.0.3xx Created! The magic is happening here

vs-mobiletools-engineering-service2 · 2024-01-12T07:14:35Z

Oh no! Backport failed! Please see https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=8913267 for more details.

… certificate. (xamarin#19650) The “Thawte SGC CA” certificate we're using for testing is now blocked by Apple, evaluating trust results in this error: > “Thawte SGC CA” certificate is blocked So update the mail_google_com certificates to what mail.google.com serves now, and also update the corresponding parent certificates (which have changed). Then adjust the tests accordingly. Finally delete all unused certificates from the code. Fixes these test failures: MonoTouchFixtures.Security.TrustTest [FAIL] Trust_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 321 [FAIL] Trust2_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust2_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 449

…locking the Thawte certificate. (#19800) The “Thawte SGC CA” certificate we're using for testing is now blocked by Apple, evaluating trust results in this error: > “Thawte SGC CA” certificate is blocked So update the mail_google_com certificates to what mail.google.com serves now, and also update the corresponding parent certificates (which have changed). Then adjust the tests accordingly. Finally delete all unused certificates from the code. Fixes these test failures: MonoTouchFixtures.Security.TrustTest [FAIL] Trust_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 321 [FAIL] Trust2_FullChain : GetTrustResult-2 Expected: RecoverableTrustFailure But was: FatalTrustFailure at MonoTouchFixtures.Security.TrustTest.Trust_FullChain(SecTrust trust, SecPolicy policy, X509CertificateCollection certs) in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 389 at MonoTouchFixtures.Security.TrustTest.Trust2_FullChain() in /Users/builder/azdo/_work/1/s/xamarin-macios/tests/monotouch-test/Security/TrustTest.cs:line 449 Backport of #19650.

This comment has been minimized.

Sign in to view

Fix build

02db1f9

This was referenced Dec 15, 2023

[msbuild] Merge the CreateInstallerPackage[TaskBase] classes. #19641

Merged

[msbuild] Merge the CreatePkgInfo[TaskBase] classes. #19642

Merged

[msbuild] Merge the DSymUtil[TaskBase] classes. #19643

Merged

dalexsoto approved these changes Dec 18, 2023

View reviewed changes

rolfbjarne merged commit eddde60 into xamarin:main Dec 18, 2023
41 of 43 checks passed

rolfbjarne deleted the fix-broken-trust branch December 18, 2023 13:13

rolfbjarne mentioned this pull request Dec 18, 2023

[release/8.0.1xx] [tests] Fix certificate tests after Apple started blocking the Thawte certificate. #19665

Merged

mandel-macaque mentioned this pull request Dec 18, 2023

[Build] Update submodules remotes. #19632

Merged

rolfbjarne mentioned this pull request Jan 12, 2024

[release/7.0.3xx] [tests] Fix certificate tests after Apple started blocking the Thawte certificate. #19800

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tests] Fix certificate tests after Apple started blocking the Thawte certificate. #19650

[tests] Fix certificate tests after Apple started blocking the Thawte certificate. #19650

rolfbjarne commented Dec 15, 2023 •

edited

Loading

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Dec 15, 2023

vs-mobiletools-engineering-service2 commented Dec 15, 2023

vs-mobiletools-engineering-service2 commented Dec 15, 2023

vs-mobiletools-engineering-service2 commented Dec 15, 2023

vs-mobiletools-engineering-service2 commented Dec 15, 2023

vs-mobiletools-engineering-service2 commented Dec 15, 2023

vs-mobiletools-engineering-service2 commented Dec 15, 2023

vs-mobiletools-engineering-service2 commented Dec 15, 2023

rolfbjarne commented Dec 18, 2023

rolfbjarne commented Dec 18, 2023

vs-mobiletools-engineering-service2 commented Dec 18, 2023

vs-mobiletools-engineering-service2 commented Dec 18, 2023

rolfbjarne commented Jan 12, 2024

vs-mobiletools-engineering-service2 commented Jan 12, 2024

vs-mobiletools-engineering-service2 commented Jan 12, 2024

[tests] Fix certificate tests after Apple started blocking the Thawte certificate. #19650

[tests] Fix certificate tests after Apple started blocking the Thawte certificate. #19650

Conversation

rolfbjarne commented Dec 15, 2023 • edited Loading

This comment has been minimized.

vs-mobiletools-engineering-service2 commented Dec 15, 2023

📚 [PR Build] Artifacts 📚

Packages generated

vs-mobiletools-engineering-service2 commented Dec 15, 2023

💻 [CI Build] Windows Integration Tests passed 💻

vs-mobiletools-engineering-service2 commented Dec 15, 2023

💻 [CI Build] Windows Integration Tests passed 💻

vs-mobiletools-engineering-service2 commented Dec 15, 2023

📚 [PR Build] Artifacts 📚

Packages generated

vs-mobiletools-engineering-service2 commented Dec 15, 2023

💻 [CI Build] Tests on macOS M1 - Mac Ventura (13.0) passed 💻

vs-mobiletools-engineering-service2 commented Dec 15, 2023

💻 [CI Build] Tests on macOS M1 - Mac Big Sur (11.5) passed 💻

vs-mobiletools-engineering-service2 commented Dec 15, 2023

✅ API diff for current PR / commit

✅ API diff vs stable

ℹ️ Generator diff

vs-mobiletools-engineering-service2 commented Dec 15, 2023

🔥 [CI Build] Test results 🔥

Test results

Failures

❌ monotouch tests

Successes

rolfbjarne commented Dec 18, 2023

rolfbjarne commented Dec 18, 2023

vs-mobiletools-engineering-service2 commented Dec 18, 2023

vs-mobiletools-engineering-service2 commented Dec 18, 2023

rolfbjarne commented Jan 12, 2024

vs-mobiletools-engineering-service2 commented Jan 12, 2024

vs-mobiletools-engineering-service2 commented Jan 12, 2024

rolfbjarne commented Dec 15, 2023 •

edited

Loading