Akka.TestKit: remove ConfigureAwait(false) from internal synchronous TestKit methods and fix Watch / Unwatch bugs

[Aaronontheweb]

Changes

close #7038
close #7064

Removed all internal usages of ConfigureAwait(false) from synchronous Akka.TestKit method implementations - this should help reduce the incidences of mysterious racy unit tests.

[Aaronontheweb]

This innocuous test is still failing, even after these changes, so it makes me think there's more async funk going on inside the TestKit that isn't completely resolved yet:

akka.net/src/core/Akka.Persistence.TestKit.Tests/Bug4762FixSpec.cs

Lines 67 to 85 in 0486f97

	[Fact]
	public async Task TestJournal_PersistAll_should_only_count_each_event_exceptions_once()
	{
	var probe = CreateTestProbe();
	await WithJournalWrite(write => write.Pass(), async () =>
	{
	var actor = ActorOf(() => new TestActor2(probe));
	Watch(actor);
	var command = new WriteMessage();
	actor.Tell(command, actor);
	await probe.ExpectMsgAsync<RecoveryCompleted>();
	await probe.ExpectMsgAsync<Done>();
	await probe.ExpectMsgAsync<Done>();
	await probe.ExpectNoMsgAsync(3000);
	});
	}
	}

[Aaronontheweb]

@akkadotnet/core not going to include this in Akka.NET v1.5.15 - too much of a hot potato and it needs time for lots of subsequent CI/CD runs to validate it before we roll out the "mission accomplished!" banner to our users

[Aaronontheweb]

Nasty, decade old bug in the TestKit - Watch and Unwatch have always been secretly asynchronous, which is why tests like this one failed:

[Fact]
        public async Task Routers_in_general_must_evict_terminated_routees()
        {
            var router = Sys.ActorOf(new RoundRobinPool(2).Props(Props.Create<Echo>()));
            router.Tell("");
            router.Tell("");

            var c1 = await ExpectMsgAsync<IActorRef>();
            var c2 = await ExpectMsgAsync<IActorRef>();

            Watch(router);
            Watch(c2);
            Sys.Stop(c2);
            (await ExpectTerminatedAsync(c2)).ExistenceConfirmed.Should().BeTrue();

            // it might take a while until the Router has actually processed the Terminated message
            await AwaitConditionAsync(async () =>
            {
                router.Tell("");
                router.Tell("");

                var res = await ReceiveWhileAsync(100.Milliseconds(), x =>
                {
                    if (x is IActorRef actorRef)
                        return actorRef;

                    return null;
                }, msgs: 2).ToListAsync();

                return res.Count == 2 && res.All(c => c.Equals(c1));
            });

            Sys.Stop(c1);
            (await ExpectTerminatedAsync(router)).ExistenceConfirmed.Should().BeTrue();
        }

Failed on (await ExpectTerminatedAsync(c2)).ExistenceConfirmed.Should().BeTrue();, which can only mean the actor was dead before the TestActor had a chance to go through.

[Aaronontheweb]

Messaging fixes to support the Watch and Unwatch raciness.

[Aaronontheweb]

Left some comments about my Watch and Unwatch fixes.

[Arkatufus]

I have some concern on the Watch and Unwatch implementation and also how sync over async is implemented

[Arkatufus]

LGTM