enable ChannelTaskScheduler to work inside Akka.Cluster without causing errors inside /system actors

[Aaronontheweb]

enable ChannelTaskScheduler to work inside Akka.Cluster without causing errors inside /system actors

Fixes #5498

Changes

Modifying the constructors of some /system actors to avoid errors when running with the ChannelTaskScheduler.

Checklist

For significant changes, please ensure that the following have been completed (delete if not relevant):

• This change follows the Akka.NET API Compatibility Guidelines.
• This change follows the Akka.NET Wire Compatibility Guidelines.
• I have reviewed my own pull request.

[Aaronontheweb]

Commented on HeartbeatReceiver changes.

[Aaronontheweb]

No real reason for a Lazy<Cluster> here since we're already receiving an instance in this actor's constructor at the time it's created.

[Aaronontheweb]

Avoid creating a delegate for the Lazy<Cluster> that we're removing from the HeartbeatReceiver

[Aaronontheweb]

Validated that this has fixed the issue with the ClusterHeartbeatSender so far. SBR actors are still throwing when using the channel-dispatcher though.

[Aaronontheweb]

Described SBR changes

[Aaronontheweb]

This is ready for review - solved #5498 completely in my test sample.

[Aaronontheweb]

For a reproduction spec, I wouldn't worry about it for now - we'll likely change the defaults to use the channel-executor in Akka.NET v1.5.

[Aaronontheweb]

Have some MNTR failures - my guess is that the SBR is missing some of the events it needs at startup in order to participate correctly. I'll shorten the timeframe and see if that helps.

[Aaronontheweb]

Looks like the SBR changes need to be reconsidered. I think we'll probably need to consider a signature change there to work around this issue.

[Zetanova]

The realy resolve the issue we would need to implement
#5447

' ... One idea would be to create an ExtensionManager actor,
that would lazy create/supervise all extensions and publish an ExtensionLoaded event on the EventStream.'

[Arkatufus]

Code looks good, but I think we need to look into those MNTR failures, it might be caused by some hidden gotchas caused by code interactions.

[Aaronontheweb]

The realy resolve the issue we would need to implement #5447

' ... One idea would be to create an ExtensionManager actor, that would lazy create/supervise all extensions and publish an ExtensionLoaded event on the EventStream.'

I don't think the "preload all plugins" idea is necessarily a good thing as it comes with a ton of unintended consequences. The lazy loading implementation not needing to be topologically sorted and gives users the options of not loading it if it's not actually needed by the application. Those are valuable trade-offs.

[Aaronontheweb]

This fix was a lot simpler than I expected - just acquire the Cluster after this actor has started, rather than doing it in the constructor.

[Zetanova]

It will produce the same problem in the startup. Maybe the delay will only suppress the issue.
The general problem is that currently in akka.net is no event system or check to detect if an extension is fully loaded or not.

Extensions are created with the Lazy<> LazyThreadSafetyMode.ExecutionAndPublication mode this will lock threads
this locks all threads until the extensions is loaded but throws if the init-thread access the Lazy instance itself.

The only method to overcome this currently would be:
A) to force the actor that require to use Cluster.Get(Context.System) on a dedicated thread, to be sure not to use the cluster-init thread
B) To make a workaround with a try-catch loop. If Cluster.Get(Context.System) throws then retry in view mills later

[Aaronontheweb]

What it does is it avoids crashing the actor who spawned it, the ClusterDaemon, which is what was happening when this call was made in the constructor. The PreStart does happen on a different run on the scheduler because it's part of the Create messaging-loop powered by the actors parent. It just all happens through /system messages that way.

[Zetanova]

The realy resolve the issue we would need to implement #5447
' ... One idea would be to create an ExtensionManager actor, that would lazy create/supervise all extensions and publish an ExtensionLoaded event on the EventStream.'

I don't think the "preload all plugins" idea is necessarily a good thing as it comes with a ton of unintended consequences. The lazy loading implementation not needing to be topologically sorted and gives users the options of not loading it if it's not actually needed by the application. Those are valuable trade-offs.

The ExtensionManager actor would still lazy load the extensions.
But then there would be a possibility for components to request, check or get notified, when extensions got loaded and are ready to be used.
It can be simply with the same pattern as the cluster state.

[Aaronontheweb]

The ExtensionManager actor would still lazy load the extensions.
But then there would be a possibility for components to request, check or get notified, when extensions got loaded and are ready to be used.

I see what you mean - ok, yep, I think that's doable. Would that mean that fetching a plugin is always an async Task operation now though?

[Zetanova]

It should be lock free of course or async requested or notified.

some Pseudo code in an actor could be:

if(!Context.System.Extensions.TryGet<Cluster>(out var cluster)) {
    Context.System.Extensions.GetAsync<Cluster>().PipeTo(Self, c => new Status.Success(c));
}

or other feature would be simlar to Cluster State

Context.System.Extensions.Subscribe(Self, ExtensionEvent.SubscriptionInitialStateMode.InitialStateAsSnapshot, new[] { typeof(Akka.IExtensionEvent) });
Context.System.Extensions.Unsubscribe(Self, typeof(IExtensionEvent)); 

maybe even better to make a view general EventStream system events
and one of them would be the type IExtensionEvent :

Context.System.Subscribe(Self, new[] { typeof(Akka.IExtensionEvent) });
Context.System.Unsubscribe(Self, typeof(IExtensionEvent)); 

or to query the current extension state:

Context.System.Extensions.SendCurrentExtensionState(Self);

The IDowningProvider would still use the locking Cluster.Get(_system)
but the Lock (if any) would then be over a TCS and not over the Lazy<> init-lock.