slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes."

[jeffschwMSFT]

Build

https://dnceng.visualstudio.com/internal/_build/results?buildId=2360768&view=results

Error

##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134

Build leg reported

vsos

Pull Request

No response

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "",
    "BuildRetry": false,
    "ErrorPattern": "The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng/internal/_build/results?buildId=2360768
Error message validated: [The job running on agent Azure Pipelines .+ ran longer than the maximum time of .+ minutes.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 2/7/2024 1:03:13 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
948649	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111934
948349	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112417
948533	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
948529	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
947934	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112357
948440	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112308
948446	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112353
948462	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#110416
948414	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
948453	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112428
948404	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112178
948384	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112379
948365	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112391
947915	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112012
948302	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112413
948281	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112412
948273	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112411
948253	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112383
948248	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#111666
948247	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log	dotnet/runtime#111666
948164	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112401
948161	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112406
948145	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112345
948115	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
948114	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
948105	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
948093	dotnet/runtime	osx-arm64 Release CoreCLR_Release	Log
948018	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112308
948011	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112403
947983	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112060
947961	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112347
947966	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112400
947948	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
947912	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112350
947911	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
947838	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#98643
947835	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112391
947753	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112337
947748	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112394
947741	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112314
947739	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112393
947594	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112389
2639876	dotnet-runtime	Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0	Log
2639854	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere 9.0	Log
947524	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112382
947530	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112381
947593	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
946814	dotnet/runtime	osx-arm64 Release NativeAOT	Log	dotnet/runtime#112027
947649	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
947512	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112384
947411	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111759
947249	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112357
947434	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
947471	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112383
947554	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log
947435	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111483
947333	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112166
947423	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112295
947321	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112375
947379	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
947390	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112378
947381	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112340
947375	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#111307
947271	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#111771
947366	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112209
947316	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
947315	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
2639713	dotnet-performance	Performance scenarios Affinity_85 windows 22H2 arm64 Ampere main	Log
947346	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112221
947342	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#111434
947335	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#111410
2639685	dotnet-performance	Performance roslyn windows 22H2 arm64 Ampere 9.0	Log
947395	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#112379
947403	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log	dotnet/aspnetcore#60298
947241	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112369
947309	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#112219
947293	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#112261
947290	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#112225
947199	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112060
947332	dotnet/runtime	tvos-arm64 Release AllSubsets_NativeAOT	Log	dotnet/runtime#111410
947329	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono_AppSandbox	Log
947319	dotnet/runtime	maccatalyst-arm64 Release AllSubsets_Mono	Log
947282	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112345
947273	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112372
947182	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112256
947164	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#110819
2639576	dotnet-efcore	macOS
947173	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112353
947090	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
947044	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112364
947102	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
947116	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log
946999	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#98643
2639445	dotnet-performance	Performance scenarios ubuntu 2204 arm64 Ampere main	Log
946906	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log
946897	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112153
946959	dotnet/runtime	osx-arm64 Release NativeAOT_Libraries	Log	dotnet/runtime#112359
946926	dotnet/aspnetcore	Test: Blazor E2E tests on Linux	Log
946841	dotnet/runtime	coreclr Pri1 Runtime Tests Run osx arm64 checked	Log
946805	dotnet/runtime	coreclr Pri0 Runtime Tests Run osx arm64 checked	Log	dotnet/runtime#112308
Displaying 100 of 1480 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
72	379	1480

[lewing]

🤔

[nagilson]

@dotnet/dnceng @dougbu This has impacted a lot of PRs recently, may you PTAL into expanding this Mac resource?

[ivanpovazan]

@dotnet/dnceng we are hitting this again and it seems there is some issue with communication with Helix machines.

More context on timeouts happening in : https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=results

• On success: Run tests in Helix step reports:

Waiting for completion of job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open (Details: https://helix.dot.net/api/jobs/fbec1164-55c5-4fa8-b492-e1ba1b413119/details?api-version=2019-06-17 )
Job 7fdc2720-3d98-4bf3-8113-274dacd69c91 on osx.1200.arm64.open is completed with 6 finished work items.
  Job fbec1164-55c5-4fa8-b492-e1ba1b413119 on osx.1200.amd64.open is completed with 6 finished work items.
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.amd64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )
  Stopping Azure Pipelines Test Run Helix Tests Build_Debugosx.1200.arm64.open (Results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=ms.vss-test-web.build-test-results-tab )

Build succeeded.

SENDHELIXJOB : warning : Helix queue osx.1200.amd64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
SENDHELIXJOB : warning : Helix queue osx.1200.arm64.open was set for estimated removal date of 2025-01-01. In most cases the queue will be removed permanently due to end-of-life; please contact dnceng for any questions or concerns, and we can help you decide how to proceed and discuss other options. [/home/vsts/work/1/s/tests/integration-tests/Apple/Simulator.Tests.proj]
    2 Warning(s)
    0 Error(s)

Time Elapsed 00:03:14.97
Killing running build processes...

Finishing: Run tests in Helix

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=923987&view=logs&j=ccc97bb6-1a23-5e71-fdfa-3cdca4a74749&t=27fc7eb2-ead9-59e1-6679-a637855d40c5

• While on failure - timeout the same step gets stuck with:

Waiting for completion of job 967d92a2-ec10-4332-927f-d28a6563f367 on osx.1200.arm64.open (Details: https://helix.dot.net/api/jobs/967d92a2-ec10-4332-927f-d28a6563f367/details?api-version=2019-06-17 )
  Job 5427159b-500d-49f1-aac0-ec148a492bbe on osx.1200.amd64.open is completed with 6 finished work items.

ref: https://dev.azure.com/dnceng-public/public/_build/results?buildId=930014&view=logs&s=c58bc33c-b825-5bca-90ca-50f6e9293dd8&j=e6966639-fe40-5068-d9ae-681cccecafdf

NOTE: All the tests successfully passed on Helix, but it seems that the communication is lost.

[garath]

Looks like the timeout happened because the osx.1200.arm64.open queue was very busy while the job was running. Right now, I do not think there are any problems with the infrastructure. I will investigate a bit more to see what caused such a back-up.

[garath]

Ah, the queue was consumed with updates and patching. The patching jobs did run longer than necessary and we've communicated with our partner team about the issue. Future jobs will be much shorter and should not overly impact jobs.