Akka.Router: sending a message to a remote actor via IScheduledTellMsg results in serialization error

[Aaronontheweb]

Version Information
Version of Akka.NET? v1.5.13 and higher
Which Akka.NET Modules? Akka.Remote

Describe the bug

This bug was introduced in Akka.NET v1.5.13 via #6461

Repro from our Discord - imagine you have the following code:

 public class SchedulerActor : ReceiveActor
    {
        private readonly IActorRef _receiverActor;
        private readonly ILoggingAdapter _log = Context.GetLogger();

        public SchedulerActor(IActorRef receiverActor)
        {
            _receiverActor = receiverActor;

            // Schedule the first message
            ScheduleMessage();
        }

        private void ScheduleMessage()
        {
            Context.System.Scheduler.ScheduleTellRepeatedly(
                initialDelay: TimeSpan.FromSeconds(1),
                interval: TimeSpan.FromSeconds(5),
                receiver:  _receiverActor,
                message: new Tick(),
                sender: Self);
        }

        // Tick message class
        public class Tick
        {
        }
    }

Imagine if the _receiverActor is a RemoteActorRef - this will cause the Tick message to be wrapped inside a ScheduledTellMsg

akka.net/src/core/Akka/Actor/Scheduler/IScheduledTellMsg.cs

Lines 27 to 34 in a14bb84

	internal sealed class ScheduledTellMsg : IScheduledTellMsg
	{
	public ScheduledTellMsg(object message)
	{
	Message = message;
	}
	public object Message { get; }
	}

Now this might not normally be a problem for many serializers - Newtonsoft.Json can probably handle this just fine. But it completely disrupts custom serialization and prevents it from working correctly.

We added ScheduledTellMsg in order to make it easier for libraries like Phobos in order to stop us from tracing scheduled messages, since that's almost always useless noise.

Expected behavior

ScheduledTellMsg is supposed to be automatically unpacked by the ActorCell - therefore it should never be something that pops into user-space.

Actual behavior

However, RemoteActorRefs do not have an ActorCell! Therefore the message never gets unpacked and will, instead, be serialized and transmitted over the network. This is a bug - we probably need to just unpack the underlying message and send that sans the ScheduledTellMsg envelope.

[Aaronontheweb]

As it turns out, this was not an issue with Akka.Remote or with the scheduler at all - but rather it was really an issue with routers in Akka.NET. The key actor in the reproduction was a router.

We've had measures in-place to prevent IScheduledTellM,sg from being serialized from the beginning:

akka.net/src/core/Akka/Actor/Scheduler/HashedWheelTimerScheduler.cs

Lines 557 to 566 in 260aee9

	public ScheduledTell(ICanTell receiver, object message, IActorRef sender)
	{
	_receiver = receiver;
	_message = receiver is not ActorRefWithCell
	? message
	: message is INotInfluenceReceiveTimeout
	? new ScheduledTellMsgNoInfluenceReceiveTimeout(message)
	: new ScheduledTellMsg(message);
	_sender = sender;
	}

That code would detect a RemoteActorRef and not include the ScheduledTellMsg envelope in order to prevent serialization issues.

However, as it turns out - we do send ScheduledTellMsgs to routers (because they have a RoutedActorCell) and those messages can still be forwarded to RemoteActorRefs with the IScheduledTellMsg wrapper intact:

akka.net/src/core/Akka/Routing/RoutedActorCell.cs

Lines 191 to 197 in 260aee9

	public override void SendMessage(Envelope envelope)
	{
	if (RouterConfig.IsManagementMessage(envelope.Message))
	base.SendMessage(envelope);
	else
	Router.Route(envelope.Message, envelope.Sender);
	}

That's what we need to fix