Akka.TestKit: Watch and Unwatch are secretly asynchronous

[Aaronontheweb]

Nasty, decade old bug in the TestKit - Watch and Unwatch have always been secretly asynchronous, which is why tests like this one failed:

[Fact]
        public async Task Routers_in_general_must_evict_terminated_routees()
        {
            var router = Sys.ActorOf(new RoundRobinPool(2).Props(Props.Create<Echo>()));
            router.Tell("");
            router.Tell("");

            var c1 = await ExpectMsgAsync<IActorRef>();
            var c2 = await ExpectMsgAsync<IActorRef>();

            Watch(router);
            Watch(c2);
            Sys.Stop(c2);
            (await ExpectTerminatedAsync(c2)).ExistenceConfirmed.Should().BeTrue();

            // it might take a while until the Router has actually processed the Terminated message
            await AwaitConditionAsync(async () =>
            {
                router.Tell("");
                router.Tell("");

                var res = await ReceiveWhileAsync(100.Milliseconds(), x =>
                {
                    if (x is IActorRef actorRef)
                        return actorRef;

                    return null;
                }, msgs: 2).ToListAsync();

                return res.Count == 2 && res.All(c => c.Equals(c1));
            });

            Sys.Stop(c1);
            (await ExpectTerminatedAsync(router)).ExistenceConfirmed.Should().BeTrue();
        }

Failed on (await ExpectTerminatedAsync(c2)).ExistenceConfirmed.Should().BeTrue();, which can only mean the actor was dead before the TestActor had a chance to go through.

Originally posted by @Aaronontheweb in #7037 (comment)