[browser] Run tests in parallel #98492

maraf · 2024-02-15T13:33:14Z

By default run tests in parallel with 8 threads
Control number of threads with WasmXHarnessMaxParallelThreads
Disabled parallel execution for
- WebSockets.Client - OperationCancelledExceptions
- Json - OOM killed
- Buffers - OOM killed - [browser][mt] Running Buffers tests in parallel results in OOM #100628

ghost · 2024-02-15T13:33:43Z

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

null

Author:	maraf
Assignees:	maraf
Labels:	`arch-wasm`, `area-Build-mono`, `os-browser`
Milestone:	-

maraf · 2024-02-22T12:25:20Z

/azp run runtime

azure-pipelines · 2024-02-22T12:25:49Z

Azure Pipelines successfully started running 1 pipeline(s).

maraf · 2024-02-23T09:51:28Z

https://helix.dot.net/api/2019-06-17/jobs/23d8d42d-43fc-42b1-a3a9-233d7b3a8a01/workitems/WasmTestOnBrowser-System.Runtime.InteropServices.JavaScript.BackgroundExec.Tests/console

[13:25:08] info: [STRT][9] System.Runtime.InteropServices.JavaScript.Tests.JSExportTest.JsExportNullableDouble(value: 3.1415926535897931)
[13:25:08] info: [FAIL][11] System.Runtime.InteropServices.JavaScript.Tests.JSExportAsyncTest.JsExportInt32DiscardNoWait(value: 42)
[13:25:08] info: System.PlatformNotSupportedException : Cannot call synchronous JS function from inside a synchronous call to a C# method.
[13:25:08] info:    at System.Runtime.InteropServices.JavaScript.Tests.JavaScriptTestHelper.invoke1O(Int32 a1)
[13:25:08] info:    at System.Runtime.InteropServices.JavaScript.Tests.JSExportAsyncTest.JsExportInt32DiscardNoWait(Int32 value)
[13:25:08] info: --- End of stack trace from previous location ---

https://helix.dot.net/api/2019-06-17/jobs/23d8d42d-43fc-42b1-a3a9-233d7b3a8a01/workitems/WasmTestOnBrowser-System.Runtime.InteropServices.JavaScript.Tests/console

[13:25:12] info: [FAIL][9] System.Runtime.InteropServices.JavaScript.Tests.JSImportTest.JsImportObjectArrayTask
[13:25:12] info: Assert.True() Failure
[13:25:12] info: Expected: True
[13:25:12] info: Actual:   False
[13:25:12] info:    at System.Runtime.InteropServices.JavaScript.Tests.JSImportTest.JsImportObjectArrayTask()
[13:25:12] info: --- End of stack trace from previous location ---
...
[13:25:12] info: [STRT][14] System.Runtime.InteropServices.JavaScript.Tests.WebWorkerTest.JSDelay_Cancellation(executor: NewThread)
[13:25:12] info: HttpClient_CancelInDifferentThread: TaskCanceledException is thrown with message: System.Threading.Tasks.TaskCanceledException: The operation was canceled.
[13:25:12] info:  ---> Error: OperationCanceledException
[13:25:12] info:    --- End of inner exception stack trace ---
[13:25:12] info:    at System.Net.Http.CancellationHelper.ThrowOperationCanceledException(Exception , CancellationToken )
[13:25:12] info:    at System.Net.Http.CancellationHelper.ThrowIfCancellationRequested(Exception , CancellationToken )
[13:25:12] info:    at System.Net.Http.BrowserHttpInterop.CancellationHelper(Task , CancellationToken , JSObject )
[13:25:12] info:    at System.Net.Http.BrowserHttpInterop.<CancellationHelper>d__22`1[[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
[13:25:12] info:    at System.Net.Http.BrowserHttpReadStream.ReadAsync(Memory`1 , CancellationToken )
[13:25:12] info:    at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream , Stream , Int32 , CancellationToken )
[13:25:12] info:    at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task , Stream )
[13:25:12] info:    at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task , MemoryStream )
[13:25:12] info:    at System.Net.Http.HttpContent.<WaitAndReturnAsync>d__82`2[[System.Net.Http.HttpContent, System.Net.Http, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.String, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
[13:25:12] info:    at System.Runtime.InteropServices.JavaScript.Tests.WebWorkerHttpTest.<>c__DisplayClass7_0.<<HttpClient_CancelInDifferentThread>b__1>d.MoveNext()
...
[13:25:12] info: [STRT][9] System.Runtime.InteropServices.JavaScript.Tests.WebWorkerTest.JSDelay_Cancellation(executor: JSWebWorker)
[13:25:12] info: HttpClient_CancelInDifferentThread: TaskCanceledException is thrown with message: System.Threading.Tasks.TaskCanceledException: The operation was canceled.
[13:25:12] info:  ---> Error: OperationCanceledException
[13:25:12] info:    --- End of inner exception stack trace ---
[13:25:12] info:    at System.Net.Http.CancellationHelper.ThrowOperationCanceledException(Exception , CancellationToken )
[13:25:12] info:    at System.Net.Http.CancellationHelper.ThrowIfCancellationRequested(Exception , CancellationToken )
[13:25:12] info:    at System.Net.Http.BrowserHttpInterop.CancellationHelper(Task , CancellationToken , JSObject )
[13:25:12] info:    at System.Net.Http.BrowserHttpInterop.<CancellationHelper>d__22`1[[System.Int32, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
[13:25:12] info:    at System.Net.Http.BrowserHttpReadStream.ReadAsync(Memory`1 , CancellationToken )
[13:25:12] info:    at System.IO.Stream.<CopyToAsync>g__Core|27_0(Stream , Stream , Int32 , CancellationToken )
[13:25:12] info:    at System.Net.Http.StreamToStreamCopy.<CopyAsync>g__DisposeSourceAsync|1_0(Task , Stream )
[13:25:12] info:    at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task , MemoryStream )
[13:25:12] info:    at System.Net.Http.HttpContent.<WaitAndReturnAsync>d__82`2[[System.Net.Http.HttpContent, System.Net.Http, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[System.String, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].MoveNext()
[13:25:12] info:    at System.Runtime.InteropServices.JavaScript.Tests.WebWorkerHttpTest.<>c__DisplayClass7_0.<<HttpClient_CancelInDifferentThread>b__1>d.MoveNext()
[13:25:12] info: [0x05e30040-jsww 13:25:12.620] MONO_WASM: forceDisposeProxies done: 2 imports, 0 exports, 1 GCHandles, 0 JSHandles.
[13:55:05] fail: Tests timed out. Killing driver service pid 79
[13:55:05] fail: Application has finished with exit code TIMED_OUT but 0 was expected

maraf · 2024-02-27T14:13:55Z

https://helix.dot.net/api/2019-06-17/jobs/2d35e4c0-0c20-47e6-988b-528540385c43/workitems/WasmTestOnBrowser-System.Collections.Concurrent.Tests/console

[10:56:10] info: [PASS][16] System.Collections.Tests.DebugView_Tests.TestDebuggerAttributes_Null
[10:56:15] info: STATE CUE CARD: (? means a positive number, usually 1 or 2, * means any number)
[10:56:15] info: 	0x0	- starting (GOOD, unless the thread is running managed code)
[10:56:15] info: 	0x1	- detached (GOOD, unless the thread is running managed code)
[10:56:15] info: 	0x2	- running (BAD, unless it's the gc thread)
[10:56:15] info: 	0x?03	- async suspended (GOOD)
[10:56:15] info: 	0x?04	- self suspended (GOOD)
[10:56:15] info: 	0x?05	- async suspend requested (BAD)
[10:56:15] info: 	0x6	- blocking (BAD, unless there's no suspend initiator)
[10:56:15] info: 	0x?07	- blocking async suspended (GOOD)
[10:56:15] info: 	0x?08	- blocking self suspended (GOOD)
[10:56:15] info: 	0x?09	- blocking suspend requested (GOOD in coop; BAD in hybrid)
[10:56:15] info: --thread 0x34ab0e0 id 0xdc96c [0] state 109
[10:56:15] info: --thread 0x200a658 id 0x1b014d8 [0] state 109
[10:56:15] info: --thread 0x3619de0 id 0x361df78 [0] state 109
[10:56:15] info: --thread 0x3617878 id 0x392d638 [0] state 108
[10:56:15] info: --thread 0x3c76b00 id 0x3c93d18 [0] state 108
[10:56:15] info: --thread 0x41fddd8 id 0x47102f0 [0] state 109
[10:56:15] info: --thread 0x429e940 id 0x5030030 [0] state 109
[10:56:15] info: --thread 0x428a9e0 id 0x52302e0 [0] state 108
[10:56:15] info: --thread 0x42af450 id 0x5540040 [0] state 109
[10:56:15] info: --thread 0x43b35a8 id 0x57402f0 [0] state 105
[10:56:15] info: --thread 0x543d1c0 id 0x5b60040 [0] state 109
[10:56:15] info: --thread 0x594e5c0 id 0x5d602f0 [0] state 104
[10:56:15] info: --thread 0x594f420 id 0x6070040 [0] state 109
[10:56:15] info: --thread 0x594fb70 id 0x6380040 [0] state 2  GC INITIATOR
[10:56:15] info: --thread 0x7d59458 id 0x690fbe8 [0] state 109
[10:56:15] info: --thread 0x72e6680 id 0x7520040 [0] state 109
[10:56:15] info: --thread 0x6ef9080 id 0x7a40030 [0] state 109
[10:56:15] info: --thread 0x747ff98 id 0x8eb0040 [0] state 109
[10:56:15] info: WAITING for 2 threads, got 1 suspended
[10:56:15] fail: [0x06380040-long 10:56:15.895] [MONO] /__w/1/s/src/mono/mono/utils/mono-threads.c:330 <disabled>

https://helix.dot.net/api/2019-06-17/jobs/2d35e4c0-0c20-47e6-988b-528540385c43/workitems/WasmTestOnBrowser-System.Runtime.InteropServices.JavaScript.BackgroundExec.Tests/console

[10:56:35] info: [STRT][13] System.Runtime.InteropServices.JavaScript.Tests.WebWorkerTest.WaitAssertsOnJSInteropThreads(executor: Main, method: ManualResetEventSlim.Wait)
[11:26:29] fail: Tests timed out. Killing driver service pid 79

https://helix.dot.net/api/2019-06-17/jobs/2d35e4c0-0c20-47e6-988b-528540385c43/workitems/WasmTestOnBrowser-System.Runtime.InteropServices.JavaScript.Tests/console

[10:56:37] info: [PASS][12] System.Runtime.InteropServices.JavaScript.Tests.WebWorkerTest.JSDelay_ContinueWith
[11:26:32] fail: Tests timed out. Killing driver service pid 81
[11:26:32] fail: Application has finished with exit code TIMED_OUT but 0 was expected

This reverts commit 8bd31e8.

pavelsavara · 2024-04-03T14:06:14Z

src/libraries/System.Runtime/tests/System.Buffers.Tests/System.Buffers.Tests.csproj

@@ -5,6 +5,9 @@
    <IncludeRemoteExecutor>true</IncludeRemoteExecutor>
    <TargetFramework>$(NetCoreAppCurrent)</TargetFramework>
  </PropertyGroup>
+  <PropertyGroup Condition="'$(TargetOS)' == 'browser'">
+    <WasmXHarnessMaxParallelThreads>1</WasmXHarnessMaxParallelThreads>


What was the problem with buffers ?

OOM killed (from PR description)

[FAIL][8] System.Buffers.ArrayPool.Tests.ArrayPoolUnitTests.RentingSpecificLengthsYieldsExpectedLengths(requestedMinimum: 2097152, expectedLength: 2097152) info: System.OutOfMemoryException : Out of memory info: at System.Buffers.ConfigurableArrayPool`1[[System.String, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Rent(Int32 minimumLength) info: at System.Buffers.ArrayPool.Tests.ArrayPoolUnitTests.RentingSpecificLengthsYieldsExpectedLengths(Int32 requestedMinimum, Int32 expectedLength) info: at System.Object.InvokeStub_ArrayPoolUnitTests.RentingSpecificLengthsYieldsExpectedLengths(Object , Span`1 ) info: at System.Reflection.MethodBaseInvoker.InvokeWithFewArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

We can re-parallelize them if you want to see the whole log

It seems that GC is unable to keep up with the traffic. Let's create issue for that.

I wonder if Mono on x64 has the same problem but survives this because of more available memory ?

Opened #100628

pavelsavara · 2024-04-04T11:26:49Z

/ba-g the failure is #95805

pavelsavara · 2024-04-04T11:35:36Z

The other failure is #98964

Co-authored-by: Larry Ewing <lewing@microsoft.com> Co-authored-by: Pavel Savara <pavel.savara@gmail.com>

maraf added 2 commits February 15, 2024 14:30

Run tests in parallel. Pass max number of parallel threads as param

723a3b8

[revert] Log all test starts

60e26d9

maraf self-assigned this Feb 15, 2024

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Feb 15, 2024

maraf added arch-wasm WebAssembly architecture area-Build-mono os-browser Browser variant of arch-wasm and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels Feb 15, 2024

maraf added this to the 9.0.0 milestone Feb 15, 2024

This was referenced Feb 15, 2024

System.Text.Json failing some large file tests #59678

Closed

[wasm] Assertion: should not be reached at D:/a/_work/1/s/src/mono/mono/sgen/sgen-scan-object.h:93 in System.Reflection.Tests #95805

Closed

Disable parallel execution for WS and JSON tests

3611072

build-analysis bot mentioned this pull request Feb 16, 2024

[browser][MT] GC deadlock #96190

Closed

Turn off parallel processing when max threads is equal to 1

475818e

This was referenced Feb 17, 2024

[browser][mt] RangeError: Invalid atomic access index #96545

Closed

[browser][MT] Unhandled JS error #97546

Closed

This comment was marked as resolved.

Sign in to view

maraf and others added 2 commits February 18, 2024 12:00

Disable parallel execution for Buffers tests

ec8185a

Merge branch 'main' into BrowserTestsParallelRun

21a8516

build-analysis bot mentioned this pull request Feb 21, 2024

System.Net.Http.Functional.Tests.HttpMetricsTest_Http11_Async_HttpMessageInvoker Test failure #96407

Closed

Merge branch 'main' into BrowserTestsParallelRun

a6f4769

This comment was marked as outdated.

Sign in to view

This was referenced Feb 24, 2024

Wasm.Build.Tests.WasmTemplateTests.RunWithDifferentAppBundleLocations times out #94821

Closed

WasmTestOnBrowser-System.Runtime.InteropServices.JavaScript.Tests time out #97112

Closed

build-analysis bot mentioned this pull request Mar 18, 2024

[browser][MT] TIMED_OUT but 0 was expected #99888

Closed

Revert "[torevert] Remove all 1 threads"

3ba705a

This reverts commit 8bd31e8.

build-analysis bot mentioned this pull request Mar 19, 2024

Build linux-x64 Debug Mono_MiniJIT_LibrariesTests failure #99942

Closed

Merge branch 'main' into BrowserTestsParallelRun

e769332

maraf requested review from pavelsavara and ilonatommy March 25, 2024 10:55

maraf marked this pull request as draft March 25, 2024 10:55

Merge branch 'main' into BrowserTestsParallelRun

024c131

ilonatommy approved these changes Mar 25, 2024

View reviewed changes

Merge branch 'main' into BrowserTestsParallelRun

932ebe2

This was referenced Apr 3, 2024

[browser] Process terminated assert handling at icall.c:6162 #100249

Closed

[wasi] Cannot find host wasmtime #100394

Closed

1 thread for JS interop tests

687fefe

pavelsavara self-assigned this Apr 3, 2024

pavelsavara reviewed Apr 3, 2024

View reviewed changes

build-analysis bot mentioned this pull request Apr 3, 2024

System.IO.Net5Compat.Tests and System.IO.Tests suddenly exiting with error 137 #100558

Open

pavelsavara marked this pull request as ready for review April 4, 2024 09:04

pavelsavara requested a review from lewing as a code owner April 4, 2024 09:04

Merge branch 'main' into BrowserTestsParallelRun

a2b0bfa

maraf mentioned this pull request Apr 4, 2024

[browser][mt] Running Buffers tests in parallel results in OOM #100628

Open

pavelsavara merged commit d96f224 into dotnet:main Apr 4, 2024
94 of 97 checks passed

maraf deleted the BrowserTestsParallelRun branch April 4, 2024 12:52

matouskozak pushed a commit to matouskozak/runtime that referenced this pull request Apr 30, 2024

[browser] Run tests in parallel (dotnet#98492)

d1fd532

Co-authored-by: Larry Ewing <lewing@microsoft.com> Co-authored-by: Pavel Savara <pavel.savara@gmail.com>

github-actions bot locked and limited conversation to collaborators May 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[browser] Run tests in parallel #98492

[browser] Run tests in parallel #98492

maraf commented Feb 15, 2024 •

edited

Loading

ghost commented Feb 15, 2024

This comment was marked as resolved.

This comment was marked as resolved.

maraf commented Feb 22, 2024

azure-pipelines bot commented Feb 22, 2024

maraf commented Feb 23, 2024

This comment was marked as outdated.

maraf commented Feb 27, 2024

pavelsavara Apr 3, 2024

maraf Apr 4, 2024

maraf Apr 4, 2024

pavelsavara Apr 4, 2024

maraf Apr 4, 2024

pavelsavara commented Apr 4, 2024

pavelsavara commented Apr 4, 2024

[browser] Run tests in parallel #98492

[browser] Run tests in parallel #98492

Conversation

maraf commented Feb 15, 2024 • edited Loading

ghost commented Feb 15, 2024

This comment was marked as resolved.

This comment was marked as resolved.

maraf commented Feb 22, 2024

azure-pipelines bot commented Feb 22, 2024

maraf commented Feb 23, 2024

This comment was marked as outdated.

maraf commented Feb 27, 2024

pavelsavara Apr 3, 2024

Choose a reason for hiding this comment

maraf Apr 4, 2024

Choose a reason for hiding this comment

maraf Apr 4, 2024

Choose a reason for hiding this comment

pavelsavara Apr 4, 2024

Choose a reason for hiding this comment

maraf Apr 4, 2024

Choose a reason for hiding this comment

pavelsavara commented Apr 4, 2024

pavelsavara commented Apr 4, 2024

maraf commented Feb 15, 2024 •

edited

Loading