Null reference exception due to RemoteActorRefProvider.RemoteInternals being null

[crasshacker]

I have a .NET 5 WPF application using Akka and Akka.Remote v1.4.10. The application crashes in a manner that looks similar to the situation described in issue #4530, when I add a remote provider to the app's HOCON config file. (This is the first time I've done this; until now all actors have been in-process.) In particular, the exception is thrown on line 156 of RemoteActorRefProvider:

    public RemoteTransport Transport { get { return RemoteInternals.Transport; } }

A null reference exception is thrown here because RemoteInternals is null. Stack trace:

Akka.Remote.dll!Akka.Remote.RemoteActorRefProvider.Transport.get() Line 156
Akka.Remote.dll!Akka.Remote.RemoteActorRefProvider.DefaultAddress.get() Line 190
Akka.dll!Akka.Event.LogSource.FromActorRef(Akka.Actor.IActorRef a, Akka.Actor.ActorSystem system) Line 110
Akka.dll!Akka.Event.LogSource.FromActor(Akka.Actor.IActorContext actor, Akka.Actor.ActorSystem system) Line 103
Akka.dll!Akka.Event.LogSource.Create(object o, Akka.Actor.ActorSystem system) Line 62
Akka.dll!Akka.Event.Logging.GetLogger(Akka.Actor.IActorContext context, Akka.Event.ILogMessageFormatter logMessageFormatter) Line 212
Akka.Logger.NLog.dll!Akka.Logger.NLog.NLogLogger.NLogLogger() Unknown
[Native to Managed Transition]
[Managed to Native Transition]
System.Private.CoreLib.dll!System.RuntimeType.CreateInstanceDefaultCtorSlow(bool publicOnly, bool wrapExceptions, bool fillCache) Line 4005
System.Private.CoreLib.dll!System.Activator.CreateInstance(System.Type type, bool nonPublic, bool wrapExceptions) Line 97
System.Private.CoreLib.dll!System.RuntimeType.CreateInstanceImpl(System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, object[] args, System.Globalization.CultureInfo culture) Line 3935
System.Private.CoreLib.dll!System.Activator.CreateInstance(System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, object[] args, System.Globalization.CultureInfo culture, object[] activationAttributes) Line 38
Akka.dll!Akka.Actor.Props.ActivatorProducer.Produce() Line 639
Akka.dll!Akka.Actor.Props.NewActor() Line 575
Akka.dll!Akka.Actor.ActorCell.CreateNewActorInstance() Line 350
Akka.dll!Akka.Actor.ActorCell.NewActor.AnonymousMethod__0() Line 335
Akka.dll!Akka.Actor.ActorCell.UseThreadContext(System.Action action) Line 374
Akka.dll!Akka.Actor.ActorCell.NewActor() Line 341
Akka.dll!Akka.Actor.ActorCell.Create(System.Exception failure) Line 446
Akka.dll!Akka.Actor.ActorCell.SysMsgInvokeAll(Akka.Dispatch.SysMsg.EarliestFirstSystemMessageList messages, int currentState) Line 267
Akka.dll!Akka.Actor.ActorCell.SystemInvoke(Akka.Dispatch.SysMsg.ISystemMessage envelope) Line 335
Akka.dll!Akka.Dispatch.Mailbox.ProcessAllSystemMessages() Line 409
Akka.dll!Akka.Dispatch.Mailbox.Run.AnonymousMethod__36_0() Line 358
Akka.dll!Akka.Actor.ActorCell.UseThreadContext(System.Action action) Line 374
Akka.dll!Akka.Dispatch.Mailbox.Run() Line 355
Akka.dll!Akka.Dispatch.ThreadPoolExecutorService..cctor.AnonymousMethod__3_0(object t) Line 28
System.Private.CoreLib.dll!System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute() Line 856
System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() Line 655
System.Private.CoreLib.dll!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() Line 29

[crasshacker]

Problem only occurs when I use a logger. If I comment out the following line in my configuration file the error doesn't occur:

loggers = ["Akka.Logger.NLog.NLogLogger, Akka.Logger.NLog"]

I'm going to assume I'm doing something wrong here and the problem is on my end, so I'll be closing the issue.

[Aaronontheweb]

@crasshacker we had another user report this issue when running with the Serilog logger enabled. My guess is that it has to do with the logger plugins not being up to date. We're going to try pushing updates to each of them and see if that resolves the issue.

[Aaronontheweb]

Serilog upgrade to Akka.NET v1.4.10 did not fix this. Going to create a reproduction spec and try again.

[Aaronontheweb]

So this NRE that is thrown - it's still caught internally here right?

akka.net/src/core/Akka/Event/Logging.cs

Lines 106 to 116 in 3705c8c

	public static string FromActorRef(IActorRef a, ActorSystem system)
	{
	try
	{
	return a.Path.ToStringWithAddress(system.AsInstanceOf<ExtendedActorSystem>().Provider.DefaultAddress);
	}
	catch // can fail if the ActorSystem (remoting) is not completely started yet
	{
	return a.Path.ToString();
	}
	}

In which case, this exception shouldn't be getting propagated to any of your own code even though it might get picked up when the debugger is attached. Is that the case or am I missing something?

[Aaronontheweb]

I figured out why this exception is thrown inside Akka.Logger.Serilog:

https://github.com/akkadotnet/Akka.Logger.Serilog/blob/9a513ac94c6e807246bd607a46fbb51173735076/src/Akka.Logger.Serilog/SerilogLogger.cs#L24-L27

That Context.GetLogger gets a reference to the logging actor for Serilog and only ever writes this one line during logger initialization:

https://github.com/akkadotnet/Akka.Logger.Serilog/blob/9a513ac94c6e807246bd607a46fbb51173735076/src/Akka.Logger.Serilog/SerilogLogger.cs#L85-L89

I am 99% sure that the NLog logger does the same thing.

But the important thing is that both of these exceptions are handled. There's no crashing / issues to worry about it. The loggers tried to grab ahold of the Akka.Remote bound address before it was populated, which can happen when lots of things are starting up asynchronously during ActorSystem initialization is there. This is a first-chance exception which will only appear when the VS debugger is attached. Enable "Just My Code" if this bothers you.

That being said, we can rewrite the way these loggers are initialized so this error doesn't come up - but it's something you can typically only see when SourceLink is enabled and Just My Code is disabled since the exception does get handled by Akka.NET's own code by the looks of it.

[Aaronontheweb]

Based on the feedback we've received from our Serliog user - looks like this is indeed a first chance exception. I'll apply the same patch I just applied to Akka.Logger.Serilog to the NLog logger and that should resolve this issue.

akkadotnet/Akka.Logger.Serilog@040c1cc

[Aaronontheweb]

I'll close this out with a patch to https://github.com/AkkaNetContrib/Akka.Logger.NLog

[Aaronontheweb]

Going to do a push of Akka.Logger.NLog v1.4.10 in a few moments which should resolve this - thanks for your report @crasshacker