Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.Threading.Tasks.Dataflow tests timing out on coreclr Linux_musl arm #71475

Closed
radical opened this issue Jun 30, 2022 · 8 comments
Closed

System.Threading.Tasks.Dataflow tests timing out on coreclr Linux_musl arm #71475

radical opened this issue Jun 30, 2022 · 8 comments
Labels
area-System.Threading.Tasks
Milestone
Future

Comments

@radical
Copy link
Member

radical commented Jun 30, 2022 •
edited by karelz
Loading

Frequency:

Day Run OS Notes
8/12-8/26 (logs available and verified)
8/5-8/11 PR #73472 6x Alpine.314.Arm64.Open + 1x Ubuntu.1804.Arm64.Open Specific to the PR
7/30 PR #49576 Alpine.314.Arm32.Open (logs not available for verfication)
7/1-7/22 (logs available and verified)
7/16 PR #72296 Debian.10.Arm32.Open System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestProducerConsumerAsyncEnumerable
7/13 PR #72108 Debian.10.Arm32.Open System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestProducerConsumerAsyncEnumerable
7/12 PR #71175 Debian.10.Arm32.Open System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestProducerConsumerAsyncEnumerable
7/7 Official run Work item killed without any tests reported as hanged - log
7/6 Official run Debian.11.Arm32.Open System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestProducerConsumerAsyncEnumerable
4/1-6/30 (logs not available for verification)
6/30 PR #71474 Alpine.314.Arm32.Open
6/30 PR #71164 Alpine.314.Arm32.Open
6/30 PR #71460 Alpine.314.Arm32.Open
6/29 PR #71440 Debian.10.Arm32.Open
6/15 PR #70756 2 failures - perhaps different root cause or caused by the PR?
6/4-6/6 PR #70240 13 failures - likely caused by the PR itself
5/13 PR #69291 2 failures - perhaps different root cause or caused by the PR?
5/6 PR #67894 Alpine.314.Arm64.Open
4/11 PR #67666 27 failures -- most likely caused by the PR itself
4/7 PR #67694 8 failures -- likely caused by the PR itself 
let timeouts = (friendlyNamePrefix : string, includePR : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').WorkItems
| where Status == "Timeout"
| where FriendlyName startswith friendlyNamePrefix
//| where Finished > datetime('2021-09-07')
| distinct JobId, WorkItemId, Name, FriendlyName, ConsoleUri
| join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
    | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")) or (Branch startswith 'refs/heads/release/6.0'))
    | where Type startswith "test/functional/cli/"
        and not(Properties contains "runtime-staging")
    | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
| project-rename JobType = Type) on JobId
| order by Finished desc
};
timeouts('System.Threading.Tasks.Dataflow', true);

Build, and Log:

  Starting:    System.Threading.Tasks.Dataflow.Tests (parallel test collections = on, max threads = 4)
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:02:11
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:04:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:06:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:08:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:10:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:12:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:14:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:16:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:18:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:20:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:22:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:24:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:26:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:28:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:30:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:32:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:34:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:36:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:38:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:40:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:42:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:44:12

...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 2700 seconds and was killed
['System.Threading.Tasks.Dataflow.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Jun 30, 2022
@ghost
Copy link

ghost commented Jun 30, 2022

Tagging subscribers to this area: @dotnet/area-system-threading-tasks
See info in area-owners.md if you want to be subscribed.

Issue Details

Build, and Log:

  Starting:    System.Threading.Tasks.Dataflow.Tests (parallel test collections = on, max threads = 4)
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:02:11
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:04:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:06:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:08:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:10:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:12:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:14:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:16:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:18:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:20:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:22:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:24:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:26:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:28:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:30:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:32:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:34:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:36:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:38:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:40:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:42:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:44:12

...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 2700 seconds and was killed
['System.Threading.Tasks.Dataflow.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
Author: radical
Assignees: -
Labels:

area-System.Threading.Tasks, untriaged
Milestone: -

@radical radical added blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' and removed area-System.Threading.Tasks untriaged New issue has not been triaged by the area owner labels Jun 30, 2022
@radical radical mentioned this issue Jun 30, 2022
Merged
@runfoapp runfoapp bot mentioned this issue Jun 30, 2022
Closed
@ghost
Copy link

ghost commented Jun 30, 2022

Tagging subscribers to this area: @dotnet/area-system-threading-tasks
See info in area-owners.md if you want to be subscribed.

Issue Details

Build, and Log:

  Starting:    System.Threading.Tasks.Dataflow.Tests (parallel test collections = on, max threads = 4)
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:02:11
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:04:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:06:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:08:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:10:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:12:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:14:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:16:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:18:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:20:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:22:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:24:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:26:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:28:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:30:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:32:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:34:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:36:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:38:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:40:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:42:12
   System.Threading.Tasks.Dataflow.Tests: [Long Running Test] 'System.Threading.Tasks.Dataflow.Tests.TransformManyBlockTests.TestOrdering_Sync_OrderedEnabledAsyncEnumerable', Elapsed: 00:44:12

...
[EXECUTION TIMED OUT]
Exit Code:-3Executor timed out after 2700 seconds and was killed
['System.Threading.Tasks.Dataflow.Tests' END OF WORK ITEM LOG: Command timed out, and was killed]
Author: radical
Assignees: -
Labels:

area-System.Threading.Tasks, blocking-clean-ci
Milestone: -

@stephentoub
Copy link
Member

This is only happening on "coreclr Linux_musl"? There shouldn't be anything OS-specific about this, so that's surprising.

@danmoseley
Copy link
Member

It is not just musl. It is Arm and Arm64, Mac and Linux. Sporadic.
I clicked on a bunch of the console outputs below, and they're all hanging in TestProducerConsumerAsyncEnumerable , not the TestOrdering_Sync_OrderedEnabledAsyncEnumerable shown above.

Some weakness in locking/memory model ? We need a dump, and we do not have the ability to get dumps from Helix, so the next step may be for someone with access to a mac M1 to loop this test. @radical do you have that ability?

WorkItems
| where FriendlyName == "System.Threading.Tasks.Dataflow.Tests"
| where  Queued > ago(30d)
| where Status == "Timeout"
//| where QueueName endswith "armarch.open"
| join Jobs on JobId 
//| extend Elapsed = (Finished-  Started)
//| extend Slow = Elapsed > make_timespan(0, 0, 10, 0)
//| where QueueName startswith "osx"
| project //Elapsed//, Slow,
  Queued,
  //FriendlyName, ExitCode, Status,
 // ConsoleUri,
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  RuntimeFlavor = tostring(parse_json(Properties).runtimeFlavor),
  //BuildId = tostring(parse_json(Properties).BuildId),
  QueueName, Source, //Started, MachineName
 ConsoleUri
 | where Pipeline != "runtime-community"
Queued PhaseName Pipeline RuntimeFlavor QueueName Source ConsoleUri
2022-06-26 20:03:39.2060000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-94aee793398742ad8c/System.Threading.Tasks.Dataflow.Tests/1/console.5637e6b1.log?helixlogtype=result
2022-06-29 08:25:07.8250000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-0d11741ccac64121b3/System.Threading.Tasks.Dataflow.Tests/1/console.58c62be0.log?helixlogtype=result
2022-06-29 16:53:28.9320000 libraries_test_run_release_coreclr_Linux_arm_Release runtime-extra-platforms coreclr ubuntu.1804.armarch.open pr/public/dotnet/runtime/refs/pull/71440/merge https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-71440-merge-f9f14ec2da8c4aadb4/System.Threading.Tasks.Dataflow.Tests/1/console.fdba12ed.log?helixlogtype=result
2022-06-30 00:23:14.3550000 libraries_test_run_checked_coreclr_Linux_musl_arm_Release runtime coreclr ubuntu.1804.armarch.open pr/public/dotnet/runtime/refs/pull/71460/merge https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-71460-merge-28fc8baa404841df9b/System.Threading.Tasks.Dataflow.Tests/1/console.46545206.log?helixlogtype=result
2022-06-30 04:30:36.6420000 libraries_test_run_checked_coreclr_Linux_musl_arm_Release runtime coreclr ubuntu.1804.armarch.open pr/public/dotnet/runtime/refs/pull/71164/merge https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-71164-merge-db9bb13888954684aa/System.Threading.Tasks.Dataflow.Tests/1/console.019996ad.log?helixlogtype=result
2022-06-30 22:29:43.5790000 libraries_test_run_checked_coreclr_Linux_musl_arm_Release runtime coreclr ubuntu.1804.armarch.open pr/public/dotnet/runtime/refs/pull/71474/merge https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-71474-merge-5f26058fe49d495bb2/System.Threading.Tasks.Dataflow.Tests/1/console.9138deea.log?helixlogtype=result
2022-07-02 20:06:06.6750000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-dea5375d89e14bb6bd/System.Threading.Tasks.Dataflow.Tests/1/console.5833cbbc.log?helixlogtype=result
2022-07-06 21:52:19.2920000 libraries_test_run_release_coreclr_Linux_arm_Release runtime-extra-platforms coreclr ubuntu.1804.armarch.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-958286d3176241138d/System.Threading.Tasks.Dataflow.Tests/1/console.ea9656b8.log?helixlogtype=result
2022-07-07 20:19:54.2230000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-e52cbd4dbf2c469ea3/System.Threading.Tasks.Dataflow.Tests/1/console.fa61c88c.log?helixlogtype=result
2022-07-08 20:49:26.1380000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-5565af61a27947fca9/System.Threading.Tasks.Dataflow.Tests/1/console.2a4f4885.log?helixlogtype=result
2022-07-12 03:15:32.2470000 libraries_test_run_release_coreclr_Linux_arm_Release runtime-extra-platforms coreclr ubuntu.1804.armarch.open pr/public/dotnet/runtime/refs/pull/71175/merge https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-71175-merge-b4995a2cf97e469c9f/System.Threading.Tasks.Dataflow.Tests/1/console.af613320.log?helixlogtype=result
2022-07-13 08:08:22.4340000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-b2e98b9d4d8241f094/System.Threading.Tasks.Dataflow.Tests/1/console.4c2c36ac.log?helixlogtype=result
2022-07-13 17:42:36.5330000 libraries_test_run_release_coreclr_Linux_arm_Release runtime-extra-platforms coreclr ubuntu.1804.armarch.open pr/public/dotnet/runtime/refs/pull/72108/merge https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-72108-merge-649ba84d4feb4cbda0/System.Threading.Tasks.Dataflow.Tests/1/console.0e27c2dc.log?helixlogtype=result
2022-07-14 08:10:51.3010000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-e517faca94714b1bb6/System.Threading.Tasks.Dataflow.Tests/1/console.84885ba3.log?helixlogtype=result
2022-07-16 00:15:17.6620000 libraries_test_run_release_coreclr_Linux_arm_Release runtime-extra-platforms coreclr ubuntu.1804.armarch.open pr/public/dotnet/runtime/refs/pull/72296/merge https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-72296-merge-4be89c1e9f8844d8ac/System.Threading.Tasks.Dataflow.Tests/1/console.bbe15dce.log?helixlogtype=result
2022-07-18 06:55:57.7160000 libraries_test_run_release_coreclr_Linux_arm_Release runtime-extra-platforms coreclr ubuntu.1804.armarch.open ci/public/dotnet/runtime/refs/heads/release/7.0-preview7 https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-70-preview7ce47b1ce4cdf4b21b6/System.Threading.Tasks.Dataflow.Tests/1/console.f573177e.log?helixlogtype=result
2022-07-19 18:49:41.5130000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/release/7.0-preview7 https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-70-preview71b16a9905f434841b1/System.Threading.Tasks.Dataflow.Tests/1/console.bb8e9563.log?helixlogtype=result
2022-07-23 08:08:12.4140000 libraries_build_OSX_arm64_Release runtime-staging osx.1200.arm64.open ci/public/dotnet/runtime/refs/heads/main https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-ca5c5f6db1ae431b91/System.Threading.Tasks.Dataflow.Tests/1/console.f7bc7485.log?helixlogtype=result 
WorkItems
| where FriendlyName == "System.Threading.Tasks.Dataflow.Tests"
| where  Queued > ago(30d)
//| where Status == "Timeout"
//| where QueueName endswith "armarch.open"
| join Jobs on JobId 
//| extend Elapsed = (Finished-  Started)
//| extend Slow = Elapsed > make_timespan(0, 0, 10, 0)
//| where QueueName startswith "osx"
| project //Elapsed//, Slow,
  Queued,
  //FriendlyName, ExitCode, Status,
 // ConsoleUri,
  PhaseName = tostring(parse_json(Properties)["System.PhaseName"]),
  Pipeline = tostring(parse_json(Properties).DefinitionName),
  RuntimeFlavor = tostring(parse_json(Properties).runtimeFlavor),
  //BuildId = tostring(parse_json(Properties).BuildId),
  QueueName, Source, Status//Started, MachineName
 | where Pipeline != "runtime-community"
 | where Status == "Pass" or Status == "Timeout"
    | summarize count() by QueueName, Status // bin(Started, 1d), bin(Elapsed, 1m)
QueueName Status count_
centos.7.amd64.open.svc Pass 70
centos.7.amd64.svc Pass 5
debian.9.amd64.open.svc Pass 27
osx.1015.amd64 Pass 8
osx.1015.amd64.appletv.open Pass 355
osx.1015.amd64.open Pass 1209
osx.1100.amd64 Pass 5
osx.1100.amd64.appletv.open Pass 40
osx.1100.amd64.open Pass 8
osx.1100.arm64.open Pass 363
osx.1200.amd64.open Pass 5724
osx.1200.arm64.open Pass 218
osx.1200.arm64.open Timeout 9
redhat.7.amd64.open.rt Pass 3069
redhat.7.amd64.open.svc Pass 421
redhat.7.amd64.rt Pass 11
redhat.7.amd64.svc Pass 7
sles.12.amd64.open.svc Pass 122
sles.15.amd64.open.rt Pass 224
sles.15.amd64.open.svc Pass 513
ubuntu.1604.amd64.open.rt Pass 4010
ubuntu.1604.amd64.open.svc Pass 3055
ubuntu.1604.amd64.rt Pass 11
ubuntu.1604.amd64.svc Pass 15
ubuntu.1804.amd64 Pass 22
ubuntu.1804.amd64.android.29.open.rt Pass 1935
ubuntu.1804.amd64.android.open.svc Pass 480
ubuntu.1804.amd64.open.rt Pass 12406
ubuntu.1804.amd64.open.svc Pass 2094
ubuntu.1804.amd64.svc Pass 9
ubuntu.1804.armarch Pass 21
ubuntu.1804.armarch.open Pass 8959
ubuntu.1804.armarch.open Timeout 9
ubuntu.2204.amd64.open.rt Pass 104
windows.10.amd64.android.open Pass 2468
windows.10.amd64.client21h1.open.rt Pass 1585
windows.10.amd64.client21h1.open.svc Pass 512
windows.10.amd64.client21h1.svc Pass 11
windows.10.amd64.clientrs5.open.svc Pass 32
windows.10.amd64.clientrs5.svc Pass 24
windows.10.amd64.open.rt Pass 3261
windows.10.amd64.open.svc Pass 72
windows.10.amd64.server2022.es.open.rt Pass 3119
windows.10.amd64.server2022.es.open.svc Pass 584
windows.10.amd64.server2022.es.rt Pass 11
windows.10.amd64.server2022.es.svc Pass 2
windows.10.amd64.serverrs5.open.rt Pass 1740
windows.10.amd64.serverrs5.open.svc Pass 748
windows.10.amd64.serverrs5.rt Pass 11
windows.10.amd64.serverrs5.svc Pass 16
windows.10.arm64 Pass 6
windows.10.arm64.open Pass 346
windows.10.arm64v8.open Pass 485
windows.11.amd64.client.open.rt Pass 2124
windows.11.amd64.client.open.svc Pass 571
windows.11.amd64.client.rt Pass 11
windows.11.amd64.client.svc Pass 1
windows.7.amd64.open.rt Pass 1430
windows.7.amd64.open.svc Pass 259
windows.7.amd64.svc Pass 11
windows.81.amd64.open.rt Pass 1673
windows.81.amd64.open.svc Pass 611
windows.81.amd64.rt Pass 11
windows.81.amd64.svc Pass 11
windows.amd64.server2022.open.rt Pass 442
windows.amd64.server2022.open.svc Pass 822
windows.amd64.server2022.svc Pass 1

@danmoseley
Copy link
Member

Reopened dotnet/dnceng#1216 as we still need dumps from hangs.

@radical
Copy link
Member Author

radical commented Aug 5, 2022

so the next step may be for someone with access to a mac M1 to loop this test. @radical do you have that ability

Sorry, I missed this. No, I don't have access to a M1 Mac.

@buyaa-n buyaa-n added this to the Future milestone Aug 15, 2022
@karelz karelz removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label Aug 26, 2022
@karelz
Copy link
Member

karelz commented Aug 26, 2022

Does not seem to be happening lately, removing blocking-clean-ci label

@stephentoub
Copy link
Member

If it's not been happening since July or August, I'll close this for now and we can reopen if it reoccurs.

@stephentoub stephentoub closed this as not planned Won't fix, can't repro, duplicate, stale Jan 20, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Feb 20, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
area-System.Threading.Tasks
Projects
None yet
Milestone
Future
Development

No branches or pull requests
6 participants
@radical @stephentoub @karelz @danmoseley @jeffschwMSFT @buyaa-n