Experiment, parallelize some tests

[majocha]

Just to see if it works and if it's worth it.

[github-actions]

❗ Release notes required

---

✅ Found changes and release notes in following paths:

Change path	Release notes path	Description
src/Compiler	docs/release-notes/.FSharp.Compiler.Service/9.0.100.md

[majocha]

Yeah this will not do much.

A lot of test cases write to stdout. We do captureConsoleOutputs in CompilerAssert but in a way that does not allow parallel execution. Not to mention in a lot of places there are printfn calls that will mangle outputs even more. All of those tests must run sequentially.

This needs a systemic approach to deal with stdout capture, maybe xUnit have some mechanism to do this in parallel.

[psfinaki]

@majocha guess what, I was going to try the same today :)

Thanks for looking into that. Indeed, it seems like we should just damn stop printing everything to the console, it's likely an artifact of older testing approaches. I cannot see any reason to do this instead of just in memory processing.

[majocha]

xUnit does not run tests from the same module in parallel. It also does not parallelize Theories.
This can slow things down even with parallel execution enabled. In FSharpSuite we have modules with lots of slow tests.

This can be mitigated by customizing xUnit in code, I think?

[psfinaki]

By customizing xUnit you mean setting up some special runner settings and assembly attributes?

We can do that. However - I am not a fan of this idea. xUnit's philosophy is really to apply good coding practices to tests. As in, write tests as you write code. Hence, e.g. compared to Nunit, it offers a very limited test platform voodoo (think fixtures, setup/teardown and so on), instead making as much as possible of the builtin language capabilities.

And so I would instead prefer keeping up with this philosophy. If, by default, xUnit parallelizes execution on the module level, then we should actually split modules into smaller ones - thereby it will improve code clarity and will generally add to better code organization :)

What do you think?

[majocha]

By customizing I meant something like https://www.meziantou.net/parallelize-test-cases-execution-in-xunit.htm

But this is not the most pressing thing and probably not needed if splitting modules would do.

The biggest hurdle for now is correctly isolating the console when running tests in parallel. Redirecting with Console.SetOut for each test won't work anymore when another thread can also redirect it unpredictably.

Console writes come from multiple sources:

1. In process executing fsi, probably also fsc
2. Various printfn and log calls sprinkled through the test cases and helper code.
3. Source code compiled and executed in AssemblyLoadContext (or AppDomain in case of net472).

While we can manage 1. and 2., 3. is a bit harder.

[majocha]

I've been chatting with Bing / Copilot about it, and it actually proposed a not bad idea:

Don't redirect the Console at all for individual tests. Instead install a custom thread splitting TextWriter upfront.
That splitting writer will keep a ThreadLocal inner TextWriter and direct all writes to it.

[psfinaki]

Thanks for the further investigations here.

In the spirit of my comment above - I just vote for reinventing as few wheels as possible and removing those we've already reinvented here :) Unit tests rarely need any output at all, but if they do - it's good to use those few means that xUnit provides for this, which are basically "plug in the writer if and when you need to".

I think it aligns with your thoughts above? It's important to make gradual changes here, probably actually in the way you outline it above. The current direction you're taking (removing stuff) looks promising!

---

Note, I am off until Monday with limited internet connection so cannot play with the code myself. Also, we've discussed this PR internally yesterday and were all very happy that things are moving in this space!

[majocha]

This is at a state that can be run locally in VS test explorer or from the console with build -c Release -testCoreClr
I think it's shaving a few minutes from build -testCoreClr locally, this could be further improved by breaking up large modules in ComponentTests and FSharpSuite. Definitely I see much better CPU utilization.

In the CI there's that weird TaskCancelledException in random tests, no idea where it comes from.

Still, there are some minor fixes here that I'll try to extract to another PR.

[majocha]

Hmm ok. A lucky run, nothing timed out.
This was with maxParallelThreads: 2 so there is some room to improvement. Also ComponentTests can still be split into smaller modules to improve CPU utilization more.

[psfinaki]

Wow! That was just about an hour!

[psfinaki]

That's super promising :)

[majocha]

Yes, I'm not giving up on that.

There's still a lot of room for improvement:
The thing I was not aware is that to run multiple assemblies in parallel you need to launch the whole solution e.g. like this:
dotnet test .\FSharp.sln -c Release -f net472 -- xUnit.DiagnosticMessages=true

In build script it seems we launch test projects in sequence, apart from FSharpSuite that runs in a background task? So the CI run does not get all the benefits yet, but the local runs are much more intense now on the CPU.

[psfinaki]

That's correct - one of relatively low hanging fruits is to parallelize stuff on the CI level. The CI matrix is already a bit complex and confusing tho, so this should not be making things even more complex. And yeah, it's PowerShell programming 👀

[majocha]

fsharp/eng/Build.ps1

Lines 603 to 614 in f4860a4

	if ($testDesktop) {
	$bgJob = TestUsingMSBuild -testProject "$RepoRoot\tests\fsharp\FSharpSuite.Tests.fsproj" -targetFramework $script:desktopTargetFramework -testadapterpath "$ArtifactsDir\bin\FSharpSuite.Tests\" -asBackgroundJob $true
	TestUsingMSBuild -testProject "$RepoRoot\tests\FSharp.Compiler.ComponentTests\FSharp.Compiler.ComponentTests.fsproj" -targetFramework $script:desktopTargetFramework -testadapterpath "$ArtifactsDir\bin\FSharp.Compiler.ComponentTests\"
	TestUsingMSBuild -testProject "$RepoRoot\tests\FSharp.Compiler.Service.Tests\FSharp.Compiler.Service.Tests.fsproj" -targetFramework $script:desktopTargetFramework -testadapterpath "$ArtifactsDir\bin\FSharp.Compiler.Service.Tests\"
	TestUsingMSBuild -testProject "$RepoRoot\tests\FSharp.Compiler.Private.Scripting.UnitTests\FSharp.Compiler.Private.Scripting.UnitTests.fsproj" -targetFramework $script:desktopTargetFramework -testadapterpath "$ArtifactsDir\bin\FSharp.Compiler.Private.Scripting.UnitTests\"
	TestUsingMSBuild -testProject "$RepoRoot\tests\FSharp.Build.UnitTests\FSharp.Build.UnitTests.fsproj" -targetFramework $script:desktopTargetFramework -testadapterpath "$ArtifactsDir\bin\FSharp.Build.UnitTests\"
	TestUsingMSBuild -testProject "$RepoRoot\tests\FSharp.Core.UnitTests\FSharp.Core.UnitTests.fsproj" -targetFramework $script:desktopTargetFramework -testadapterpath "$ArtifactsDir\bin\FSharp.Core.UnitTests\"
	# Collect output from background jobs
	Wait-job $bgJob | out-null
	Receive-Job $bgJob -ErrorAction Stop

Now I think this might have something to do with those OOM exceptions in CI that were happening in this PR.
I'd try to not run that as a background task, instead make things straightforward for now.
Run the test projects one by one, but don't limit maxthreads.

[psfinaki]

That's definitely worth giving a shot :)

I don't think this is related to the OOMs. In fact, Tomas made this change to speed up the CI back in a day, it's the only paralellelization we have there really therefore.

[majocha]

There are two classes of errors that still remain to be dealt with:

1.Timing: timeouts tripping in tests that rely on hardcoded delays like Async.Sleep 1000, .Wait(timeout=..., tests expecting the threadpool jobs to be scheduled in one particular order etc.
For example:

fsharp/tests/FSharp.Core.UnitTests/FSharp.Core/Microsoft.FSharp.Control/AsyncModule.fs

Lines 471 to 481 in 52133d0

	async {
	let mutable counter = 0
	let job i = async {
	if i = 55 then failwith "boom"
	else
	do! Async.Sleep 1000
	counter <- counter + 1
	}
	let! _ = Async.Parallel [ for i in 1 .. 100 -> job i ] |> Async.Catch
	do! Async.Sleep 5000

2. Variations on value cannot be null that happen during fsi session or checker initialization. This one shows up under heavy load, when xUnit maxtreads are not limited, usually at the beginning of the project test run:
   https://dev.azure.com/dnceng-public/public/_build/results?buildId=809681&view=logs&j=170942dc-5349-5022-2275-77744f335216&t=20776a93-33c2-52f0-ee8f-b8c9809b2e23&l=4635

Now what the plan is?

1. can be mitigated by marking the offending tests with DisableParallelization. In the long term we can rewrite the offending tests. I was trying to use "orchestration" with synchronization primitives instead of timeouts in some of these tests and it seems to work. Additionally we could use xUnit setting to detect stuck / hanged tests instead of putting timeouts in tests.

2. We do reuse some global checkers in many tests, for example:
   

   fsharp/tests/FSharp.Test.Utilities/CompilerAssert.fs

   Line 305 in 52133d0

   let checker = FSharpChecker.Create(suggestNamesForErrors=true, useTransparentCompiler=UseTransparentCompiler)

   
   Now what happens when the test methods are started by reflection, in a worker thread, in parallel? Effectively our context is initialized in some static constructors. Or is it? How theradsafe is it all?
   I think I'll just put some initialization guard in a xUnit BeforeAfter attribute, this cannot be helped.

[psfinaki]

I think it's a reasonable direction.

1. can be mitigated by marking the offending tests with DisableParallelization

Yes, definitely no need to rewrite all the tests properly. Since those tests don't fail all the time but tend to fail, I'd probably just proactively mark all of them to not run in parallel. Bonus points for creating an item and referencing it in the code - because one of things we are doing here is explaining why things are the way the are :)

2. We do reuse some global checkers in many tests

Explicit is indeed better than implicit here, reflection is already obscure enough.

[majocha]

I went for it and modified the -testDesktop and -testCoreClr in the build script.
Now these options run the whole FSharp.sln in parallel (respecting the exclusions in xunit.runner.json, I hope).

Locally, on a i5 13gen I see runtimes around 4 minutes for -testCoreClr, this is a few minutes faster than main. -testDesktop runs in 18 minutes, I have no idea how it compares to main, I never had the patience :)

[psfinaki]

Note that since the matters are very subtle here, I'd probably separate CI parallelization also into a separate PR :)