Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feed unreliability affecting CI #55449

Closed
runfoapp bot opened this issue Jul 10, 2021 · 15 comments
Closed

Feed unreliability affecting CI #55449

runfoapp bot opened this issue Jul 10, 2021 · 15 comments
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' blocking-official-build tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Milestone
Future

Comments

@runfoapp
Copy link

runfoapp bot commented Jul 10, 2021 •
edited
Loading

Runfo Tracking Issue: Feed unreliability affecting CI

Definition Build Kind Job Name
runtime 1337283 PR 58544 Installer Build and Test coreclr Linux_musl_arm64 Debug
runtime 1335495 PR 58462 CoreCLR Product Build OSX arm64 release
runtime 1334501 PR 58508 Build Linux arm Debug AllSubsets_Mono
runtime 1330300 Rolling Installer Build and Test coreclr FreeBSD_x64 Release
runtime 1330012 PR 58236 CoreCLR Product Build Linux_musl arm64 release
runtime 1328346 PR 58167 Mono llvmaot Product Build Linux x64 release
runtime 1328262 Rolling Installer Build and Test coreclr OSX_x64 Release
runtime 1327734 PR 58344 Libraries Build Linux x64 Debug
runtime 1324617 PR 57988 Libraries Build OSX x64 Debug
runtime 1323766 PR 58256 CoreCLR Product Build Linux arm release
runtime 1322564 PR 58236 Build iOS arm64 Debug AllSubsets_Mono
runtime 1322277 PR 58215 Installer Build and Test coreclr Linux_x64 Release
runtime 1322032 PR 57398 Build Browser wasm Release AllSubsets_Mono
runtime 1320333 Rolling CoreCLR Product Build OSX arm64 release
runtime 1319751 Rolling Installer Build and Test coreclr Linux_x64 Release
runtime 1315328 PR 58064 Build MacCatalyst arm64 Debug AllSubsets_Mono
runtime 1313767 Rolling Build Linux_musl x64 Release AllSubsets_Mono
runtime 1313767 Rolling Mono MacCatalyst AOT offsets
runtime 1313707 PR 57996 PGO Installer Build and Test coreclr Linux_x64 Release
runtime 1311705 PR 57854 Build windows x64 Release SingleFile
runtime 1311169 PR 56281 Mono Browser AOT offsets
runtime 1310805 Rolling Libraries Build OSX x64 Release
runtime 1306720 PR 57782 Installer Build and Test coreclr Linux_musl_arm64 Debug
runtime 1306711 PR 57804 Mono Product Build OSX x64 debug
runtime 1306401 PR 57798 Build iOS arm64 Debug AllSubsets_Mono
runtime 1305366 PR 57764 CoreCLR Pri0 Runtime Tests Run OSX x64 checked
runtime 1304779 PR 57742 Libraries Test Run release coreclr Linux x64 Debug
runtime 1304660 Rolling Libraries Test Run checked coreclr windows x64 Release
runtime 1304660 Rolling Libraries Test Run checked coreclr windows x64 Release
runtime 1304660 Rolling Libraries Test Run checked coreclr windows x64 Release
runtime 1304642 PR 57700 Build OSX x64 Release AllSubsets_Mono_LLVMJIT
runtime 1304419 PR 57648 Installer Build and Test coreclr windows_x64 Release
runtime 1303596 PR 57508 Installer Build and Test coreclr windows_x64 Release
runtime 1302767 PR 57695 Libraries Test Run release mono_interpreter Linux x64 Debug
runtime 1302472 PR 57687 Build iOS arm Release AllSubsets_Mono
runtime 1301875 PR 57585 CoreCLR Product Build FreeBSD x64 release
runtime 1301761 PR 57424 CoreCLR Product Build Linux_musl x64 checked
runtime 1301761 PR 57424 CoreCLR Product Build Linux_musl x64 checked
runtime 1301761 PR 57424 CoreCLR Product Build windows x86 release PGO
runtime 1301761 PR 57424 Build Browser wasm Release AllSubsets_Mono_WasmBuildTests
runtime 1301761 PR 57424 CoreCLR Product Build windows x86 release
runtime 1301761 PR 57424 CoreCLR Product Build Linux arm release
runtime 1301761 PR 57424 CoreCLR Product Build Linux_musl x64 release
runtime 1301761 PR 57424 CoreCLR Product Build Linux x64 checked
runtime 1301761 PR 57424 CoreCLR Product Build Linux arm64 release
runtime 1301761 PR 57424 CoreCLR GCC Product Build Linux x64 checked
runtime 1301761 PR 57424 CoreCLR Common Pri0 Test Build AnyOS AnyCPU checked
runtime 1301761 PR 57424 CoreCLR Product Build Linux_musl x64 checked
runtime 1301701 PR 57292 CoreCLR Product Build Linux x64 release
runtime 1301701 PR 57292 CoreCLR Product Build Linux x64 checked
runtime 1301701 PR 57292 CoreCLR Product Build OSX x64 checked
runtime 1301687 PR 55186 CoreCLR Tools Unit Tests Linux x64 checked
runtime 1301683 PR 57655 Installer Build and Test coreclr windows_arm Debug
runtime 1301683 PR 57655 Installer Build and Test coreclr Linux_arm64 Release
runtime 1301683 PR 57655 Installer Build and Test coreclr windows_arm64 Debug
runtime 1301643 PR 57653 Libraries Test Run release mono_interpreter Linux x64 Debug
runtime 1301540 PR 57648 Libraries Build OSX x64 Debug
runtime 1301338 Rolling Libraries Test Run checked coreclr Linux x64 Release
runtime 1301321 PR 57637 CoreCLR Product Build Linux x64 release
runtime 1301311 PR 57156 Build tvOS arm64 Release AllSubsets_Mono
runtime 1301254 PR 57635 Libraries Test Run release mono OSX x64 Debug
runtime 1300342 Rolling Libraries Build OSX arm64 Release
runtime 1300088 PR 57614 CoreCLR Product Build Linux x64 release
runtime 1299989 PR 52402 CoreCLR Product Build Linux x64 release PGO
runtime 1299890 PR 57396 Mono tvOS AOT offsets
runtime 1299890 PR 57396 Build Linux x64 Release AllSubsets_Mono_LLVMAOT
runtime 1299890 PR 57396 CoreCLR Product Build windows arm checked
runtime 1299793 PR 57607 Installer Build and Test coreclr windows_x86 Debug
runtime 1299793 PR 57607 CoreCLR Product Build OSX arm64 release
runtime 1299767 Rolling CoreCLR Product Build windows x64 checked
runtime 1299767 Rolling CoreCLR Product Build Linux_musl arm release
runtime 1299767 Rolling Libraries Test Run release coreclr Linux_musl x64 Release
runtime 1299767 Rolling CoreCLR Product Build Linux arm checked
runtime 1299767 Rolling Libraries Test Run release mono Linux arm64 Release
runtime 1299767 Rolling Mono MacCatalyst AOT offsets
runtime 1299702 PR 56501 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1299702 PR 56501 Build iOSSimulator x86 Debug AllSubsets_Mono
runtime 1299656 PR 57603 Libraries Build OSX x64 Debug
runtime 1299656 PR 57603 Libraries Build OSX x64 Debug
runtime 1299641 PR 56659 CoreCLR Product Build Linux arm64 checked
runtime 1299641 PR 56659 Libraries Build OSX x64 Debug
runtime 1299641 PR 56659 Libraries Build OSX x64 Debug
runtime 1299641 PR 56659 Mono Product Build OSX x64 release
runtime 1299641 PR 56659 Installer Build and Test coreclr OSX_arm64 Release
runtime 1299582 PR 57023 Libraries Build OSX arm64 Debug
runtime 1299520 PR 57599 Libraries Test Run release coreclr Linux x64 Debug
runtime 1299488 PR 57595 Installer Build and Test coreclr OSX_x64 Release
runtime 1299263 Rolling Mono Browser AOT offsets
runtime 1299263 Rolling Build Browser wasm Release AllSubsets_Mono_WasmBuildTests
runtime 1299263 Rolling Build Linux x64 Release SingleFile
runtime 1299263 Rolling CoreCLR Product Build windows arm checked
runtime 1299263 Rolling Build Browser wasm Release AllSubsets_Mono_AOT
runtime 1299263 Rolling Build Linux arm Release AllSubsets_Mono
runtime 1299263 Rolling Build Browser wasm Release AllSubsets_Mono_RuntimeTests
runtime 1299263 Rolling Build Linux x64 Release AllSubsets_Mono_LLVMJIT
runtime 1299263 Rolling Mono Android AOT offsets
runtime 1299263 Rolling Build Linux arm64 Release AllSubsets_Mono_LLVMAOT
runtime 1299263 Rolling Build Browser wasm Release AllSubsets_Mono_EAT
runtime 1299263 Rolling Build Android x64 Release AllSubsets_Mono_RuntimeTests
runtime 1299155 PR 57457 Build MacCatalyst arm64 Debug AllSubsets_Mono

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
0 4 57
@dotnet-issue-labeler dotnet-issue-labeler bot added the untriaged New issue has not been triaged by the area owner label Jul 10, 2021
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost
Copy link

ghost commented Jul 10, 2021

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Runfo Tracking Issue: Feed unreliability affecting CI

Definition Build Kind Job Name
runtime 1229684 PR 55426 Build windows x64 Release SingleFile
runtime 1229619 PR 55423 CoreCLR Product Build windows x64 release PGO
runtime 1229561 PR 55295 Installer Build and Test coreclr windows_x86 Debug
runtime 1229543 PR 55384 Installer Build and Test coreclr windows_x86 Debug
runtime 1229543 PR 55384 Installer Build and Test coreclr windows_x64 Release
runtime 1229142 PR 54640 Installer Build and Test coreclr windows_x86 Debug
runtime 1229104 PR 55409 Installer Build and Test coreclr windows_x86 Debug
runtime 1229095 PR 55408 Installer Build and Test coreclr windows_x64 Release
runtime 1229090 PR 46499 Installer Build and Test coreclr windows_x86 Debug
runtime 1229000 PR 47935 Installer Build and Test coreclr windows_x64 Release
runtime 1229000 PR 47935 Installer Build and Test coreclr windows_x64 Release
runtime 1229000 PR 47935 Libraries Test Run release mono Linux x64 Debug
runtime 1228930 PR 54954 CoreCLR Pri0 Runtime Tests Run windows arm64 checked
runtime 1228930 PR 54954 CoreCLR Pri0 Runtime Tests Run windows arm64 checked
runtime 1228855 PR 55395 Installer Build and Test coreclr windows_x86 Debug
runtime 1228789 PR 55373 CoreCLR Pri0 Runtime Tests Run windows x86 checked
runtime 1228789 PR 55373 CoreCLR Pri0 Runtime Tests Run windows x86 checked
runtime 1228776 PR 55384 Installer Build and Test coreclr windows_x64 Release
runtime 1228090 PR 55037 Installer Build and Test coreclr windows_x86 Debug
runtime 1227275 Rolling CoreCLR Product Build Windows_NT x64 release
runtime 1227275 Rolling Mono Product Build Linux x64 debug
runtime 1227275 Rolling Build Linux arm Release AllSubsets_Mono
runtime 1227275 Rolling Libraries Build Linux_musl x64 Release
runtime 1227264 PR 55216 Libraries Build Linux arm Release
runtime 1227264 PR 55216 Libraries Build Linux x64 Debug
runtime 1227264 PR 55216 Mono llvmaot Product Build Linux arm64 release
runtime 1227264 PR 55216 Libraries Build windows net48 x86 Release
runtime 1227264 PR 55216 Libraries Build Linux_musl x64 Debug
runtime 1227264 PR 55216 Build Linux_musl x64 Release AllSubsets_Mono
runtime 1227264 PR 55216 Build Browser wasm Release AllSubsets_Mono_AOT
runtime 1227264 PR 55216 CoreCLR Product Build windows x86 release PGO
runtime 1227264 PR 55216 Build Browser wasm Release AllSubsets_Mono_RuntimeTests
runtime 1227264 PR 55216 Mono Android AOT offsets
runtime 1227264 PR 55216 Build windows x64 Release SingleFile
runtime 1227264 PR 55216 CoreCLR Common Pri0 Test Build AnyOS AnyCPU checked
runtime 1227264 PR 55216 Libraries Build windows x64 Debug
runtime 1227264 PR 55216 Build Linux arm Debug AllSubsets_Mono
runtime 1227264 PR 55216 Build Browser wasm Release AllSubsets_Mono_EAT
runtime 1227264 PR 55216 Libraries Build windows arm64 Release
runtime 1227264 PR 55216 Libraries Build Linux_musl arm Release
runtime 1227264 PR 55216 CoreCLR Product Build windows x64 release
runtime 1227264 PR 55216 Mono Product Build Linux arm64 debug
runtime 1227248 PR 55299 Libraries Build windows x86 Release
runtime 1227248 PR 55299 CoreCLR Product Build windows arm release
runtime 1227248 PR 55299 Libraries Build Linux_musl arm64 Release
runtime 1227248 PR 55299 CoreCLR Product Build windows x86 release
runtime 1227248 PR 55299 Libraries Build windows arm64 Release
runtime 1227201 PR 55354 Build OSX x64 Debug AllSubsets_Mono_LLVMAOT
runtime 1227201 PR 55354 CoreCLR Product Build Linux x64 release PGO
runtime 1227201 PR 55354 Mono Product Build OSX x64 release
runtime 1227201 PR 55354 Installer Build and Test coreclr Linux_arm64 Release
runtime 1227201 PR 55354 CoreCLR Product Build Linux x64 release
runtime 1227192 PR 55353 Installer Build and Test coreclr Linux_x64 Release
runtime 1227192 PR 55353 PGO Installer Build and Test coreclr Linux_x64 Release
runtime 1227192 PR 55353 Installer Build and Test coreclr Linux_arm64 Release
runtime 1227082 PR 55345 Installer Build and Test coreclr Linux_musl_x64 Release
runtime 1226905 PR 55279 Libraries Build OSX x64 Debug
runtime 1226905 PR 55279 Build tvOS arm64 Release AllSubsets_Mono
runtime 1226847 PR 55282 Installer Build and Test coreclr OSX_x64 Release
runtime 1226843 PR 55327 Libraries Test Run release mono Linux arm64 Debug
runtime 1226638 PR 55095 CoreCLR Product Build windows x64 release
runtime 1226212 PR 55032 Installer Build and Test coreclr windows_x86 Debug
runtime 1225919 PR 55308 Installer Build and Test coreclr windows_x64 Release
runtime 1225919 PR 55308 Installer Build and Test coreclr windows_x64 Release
runtime 1225906 PR 55299 Installer Build and Test coreclr windows_x64 Release
runtime 1225680 PR 55303 Installer Build and Test coreclr windows_x86 Debug
runtime 1224489 PR 55283 Installer Build and Test coreclr Linux_arm64 Release
runtime 1224489 PR 55283 Libraries Test Run release coreclr Linux_musl x64 Debug
runtime 1224332 PR 55280 Installer Build and Test coreclr windows_x86 Debug
runtime 1223962 PR 55260 Installer Build and Test coreclr windows_x64 Release
runtime 1223962 PR 55260 Installer Build and Test coreclr windows_x64 Release
runtime 1223735 PR 55257 Installer Build and Test coreclr windows_x86 Debug

Build Result Summary

Day Hit Count Week Hit Count Month Hit Count
13 32 32
Author: runfoapp[bot]
Assignees: -
Labels:

area-Infrastructure, untriaged
Milestone: -

@hoyosjs hoyosjs added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' blocking-official-build tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly and removed untriaged New issue has not been triaged by the area owner labels Jul 11, 2021
@hoyosjs hoyosjs added this to the 7.0.0 milestone Jul 11, 2021
@hoyosjs hoyosjs modified the milestones: 7.0.0, Future Jul 11, 2021
@runfoapp runfoapp bot mentioned this issue Jul 12, 2021
Closed
@SingleAccretion SingleAccretion mentioned this issue Jul 12, 2021
Merged
@Daniel-Genkin Daniel-Genkin mentioned this issue Jul 12, 2021
Closed
7 tasks
@steveisok
Copy link
Member

Seems to be acting up today on https://dev.azure.com/dnceng/public/_build/results?buildId=1233089&view=logs&jobId=4ec48297-f0d4-5d6c-7d7c-174fafbbae1c&j=f7707b4d-03d4-5981-287a-9de6d0a30080

@shirhatti shirhatti mentioned this issue Jul 12, 2021
Merged
@lewing lewing mentioned this issue Jul 12, 2021
Merged
@sandreenko
Copy link
Contributor

Are these hangs https://dev.azure.com/dnceng/public/_build/results?buildId=1237740&view=ms.vss-test-web.build-test-results-tab&runId=36827102&resultId=110545&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab instances of this issue?
Looks like arm64 OSX is the most affected, for example:https://dev.azure.com/dnceng/public/_build/results?buildId=1237122&view=ms.vss-test-web.build-test-results-tab&runId=36812660&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=110689 all OSX arm64 test runs stopped without any reason in the log:

+ dotnet /tmp/helix/working/9EF108CC/p/xunit/xunit.console.dll baseservices/threading/baseservices.threading.XUnitWrapper.dll -parallel collections -nocolor -noshadow -xml testResults.xml -trait TestGroup=baseservices.threading
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 6.0.0-preview.4.21253.7)
  Discovering: baseservices.threading.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  baseservices.threading.XUnitWrapper (found 200 of 339 test cases)
  Starting:    baseservices.threading.XUnitWrapper (parallel test collections = on, max threads = 8)

and nothing here.

@jaredpar
Copy link
Member

Are these hangs https://dev.azure.com/dnceng/public/_build/results?buildId=1237740&view=ms.vss-test-web.build-test-results-tab&runId=36827102&resultId=110545&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab instances of this issue?

At a glance no. If they were then you would see this GitHub issue associated with the build in runfo. At the moment there are no associated issues.

https://runfo.azurewebsites.net/view/build/?Number=1237740

Dug in a bit and it's hitting the Python PIP error I saw in some other logs. Filed the following to track in core-eng

https://github.com/dotnet/core-eng/issues/13736

@sandreenko
Copy link
Contributor

Thanks @jaredpar, I see that dotnet/core-eng#13736 covers failures like "The tests all completed successfully but ended up failing with the following root error:" but it does not cover OSX arm64 hangs where we have the test starting and then there is nothing in the log https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-050e73acf1cf4c5b91/JIT.CodeGenBringUpTests/console.ae514c4e.log?sv=2019-07-07&se=2021-08-03T16%3A52%3A11Z&sr=c&sp=rl&sig=ZfN3aMhfYLiFMMdWWmscAGvcn3aPJXnHHtkVWyCmHqk%3D.

Do we need a separate issue for it?

@jaredpar
Copy link
Member

@sandreenko yes seems like we should have a separate issue for that. Can you point me to some builds which hit that failure? I want to look at some of the other log files.

@hoyosjs
Copy link
Member

hoyosjs commented Jul 15, 2021

@sandreenko this is not a regular hang, those get killed by helix and it adds a timeout disclaimer. This looks like an agent disconnect. I had reported somethin like this to @Chrisboh earlier this week. See for example:

It also seems suspicious because it creates an error log for every test but they are not downloadable on the helix API last I checked.

@jaredpar
Copy link
Member

@hoyosjs do you have links to the AzDO builds that produced those logs?

@sandreenko
Copy link
Contributor

@hoyosjs do you have links to the AzDO builds that produced those logs?

This build has many cases. It is a scheduled main testing.

This is a PR triggered job but also with many such failures https://dev.azure.com/dnceng/public/_build/results?buildId=1237123&view=ms.vss-test-web.build-test-results-tab

@sandreenko sandreenko mentioned this issue Jul 15, 2021
Merged
@jaredpar
Copy link
Member

@sandreenko

Looking at the run client.py:

2021-07-14T05:53:26.853Z	INFO   	executor(112)	run	Successfully downloaded work item payload
2021-07-14T05:53:26.853Z	INFO   	interval(60)	set_abort_after_timer	Set the timer's abort_after to 7260 seconds from now: abort_after = 2021-07-14 00:54:26.853923
2021-07-14T05:53:26.854Z	INFO   	executor(771)	_execute_command	running $HELIX_CORRELATION_PAYLOAD/scripts/52ff2aa3710f4b2b9efdda8fac721ae2/execute.sh in /tmp/helix/working/ACD1099D/w/B9320A2B/e max 7200 seconds
2021-07-14T05:53:46.726Z	INFO   	job(38)	wait	Work item's process exited normally.
2021-07-14T05:53:46.777Z	INFO   	job(100)	kill_any_leaked_workitem_processes	Killing leaked process : 3437 bash
2021-07-14T05:53:46.777Z	INFO   	job(100)	kill_any_leaked_workitem_processes	Killing leaked process : 3440 corerun
2021-07-14T05:53:46.777Z	INFO   	job(100)	kill_any_leaked_workitem_processes	Killing leaked process : 3441 bash
2021-07-14T05:53:46.778Z	INFO   	job(100)	kill_any_leaked_workitem_processes	Killing leaked process : 3444 corerun
2021-07-14T05:53:46.778Z	INFO   	job(100)	kill_any_leaked_workitem_processes	Killing leaked process : 3445 bash
2021-07-14T05:53:46.779Z	INFO   	job(100)	kill_any_leaked_workitem_processes	Killing leaked process : 3448 corerun
2021-07-14T05:53:46.779Z	INFO   	job(100)	kill_any_leaked_workitem_processes	Killing leaked process : 3451 corerun
2021-07-14T05:53:46.780Z	INFO   	interval(63)	cancel_abort_after	Cancel abort_after for timer
2021-07-14T05:53:46.780Z	INFO   	executor(802)	_execute_command	The return code was not 0. Sleeping for 30 seconds in case it's still writing to disk.
2021-07-14T05:54:16.785Z	INFO   	executor(719)	_dump_file_upload	Dump files created in /cores are ready to be uploaded
2021-07-14T05:54:16.786Z	INFO   	executor(727)	_dump_file_upload	Dump file :core.53368 , size : 2772762624
2021-07-14T05:54:16.787Z	ERROR  	executor(739)	_dump_file_upload	Dump file could not be copied to the upload folder and deletedcore.53368

It looks like the job exited but left a lot of child processes around that were then killed but later there were issues uploading the core files (believe that is what the log says). @MattGal or @ChadNedzlek would need to confirm.

@MattGal
Copy link
Member

MattGal commented Jul 15, 2021

taking a look.

@MattGal
Copy link
Member

MattGal commented Jul 15, 2021

I looked at this. There are two annoying things going on,

  • The work item didn't actually leak processes, the machine started two Helix clients and they killed each other. We only recently added killing of them (basically now, before and after work's done, we walk the machine's processes and kill any that contain the work items' specific environment variables, because they can't always be trusted to die on their own. This is tracked by https://github.com/dotnet/core-eng/issues/13596 but thus far noone has figured out how the second client gets started. There are many theories related to manual machine maintenance but no smoking gun yet. I took all the machines hitting this offline, it should stop now, but noone has root-caused it so we don't know when it will impact other macs.

To see this more clearly, try this Kusto query:

WorkItems
| where Started >= datetime(2021-07-14 05:50:11.5190000) and Started <= datetime(2021-07-14 05:59:31.3480000)
| where QueueName == "osx.1100.arm64.open"
| where MachineName	== "dci-macm1-build-003.local"

... and look closely at the overlap in start times for work items

  • The problem with the dumps is not that your tests crashed (indeed, by virtue of being kill()'ed they didn't likely crash). This particular machine (dci-macm1-build-003) wasn't doing this before and, interestingly, was interacted-with by the DDFUN team yesterday and started doing the double-client thing then. This problem is simply that the helix runner user lacks permissions to the /cores folder, which means crash dumps go in but never get cleaned up. I'll check that @parose1 fixed this last night like he mentioned, though it matters little as this machine is no offline as an investigation machine for the double-client issue above.

@hoyosjs hoyosjs mentioned this issue Jul 21, 2021
Closed
This was referenced Aug 10, 2021
Merged
Merged
Merged
Merged
@hoyosjs hoyosjs mentioned this issue Aug 18, 2021
Merged
This was referenced Aug 20, 2021
Merged
Merged
@ViktorHofer ViktorHofer mentioned this issue Aug 30, 2021
Merged
@andrew-boyarshin andrew-boyarshin mentioned this issue Aug 30, 2021
Closed
@ViktorHofer
Copy link
Member

Mitigated with 3827b7f (based on the data that runfo shows). Keeping open until the change gets into the 6.0 branches (via an Arcade update).

@ViktorHofer
Copy link
Member

Closing as the fix is now part of every branch that showed the symptom (main, release/6.0*).

@ghost ghost locked as resolved and limited conversation to collaborators Oct 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' blocking-official-build tracking-external-issue The issue is caused by external problem (e.g. OS) - nothing we can do to fix it directly
Projects
None yet
Milestone
Future
Development

No branches or pull requests
7 participants
@jaredpar @steveisok @ViktorHofer @MattGal @jeffschwMSFT @hoyosjs @sandreenko