Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stashing fix #1480

Merged
merged 1 commit into from
Dec 6, 2015
Merged

Stashing fix #1480

merged 1 commit into from
Dec 6, 2015

Conversation

rogeralsing
Copy link
Contributor

This is a suggestion on how to solve the stashing #1400 issue
The mechanics in this PR works by incrementing an envelopeId in the actorcell for each user message received.
Once you call Stash the stash plugin will check if it has already stashed this envelope Id.

This has minimal impact on performance and we can still keep our envelopes as structs.
The solution is also unaffected by async await features as the envelope ID is only touched when processing user messages.

Thoughts?

@@ -43,7 +43,7 @@ public void ConsistentHashingGroup_must_use_same_hash_ring_indepenent_of_self_ad
var keys = new List<string>(new [] { "A", "B", "C", "D", "E", "F", "G"});
var result1 = keys.Select(k => consistentHash1.NodeFor(k).Routee);
var result2 = keys.Select(k => consistentHash2.NodeFor(k).Routee);
result1.ShouldBeEquivalentTo(result2);
Assert.Equal(result2,result1);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignore this line, this is a fix for the broken dev branch.

@rogeralsing
Copy link
Contributor Author

We could ofcourse just replace the counter with a bool, but that would make a weird coupling between the actorcell and the stashing plugin.

@Aaronontheweb
Copy link
Member

Looks like this build crashed on running the normal Xunit assemblies. Going to re-run it and see if we can reproduce.

[13:20:22][Step 1/1] [WARNING][11/29/2015 1:20:21 PM][Thread 0022][LocalActorRefProvider(akka://RemoteDaemonSpec-52)] akka://RemoteDaemonSpec-52/remote trying to remove non-child $b
[13:20:22][Step 1/1] [WARNING][11/29/2015 1:20:21 PM][Thread 0683][LocalActorRefProvider(akka://RemoteDaemonSpec-52)] akka://RemoteDaemonSpec-52/temp trying to remove non-child d
[13:20:22][Test collection for Akka.Remote.Tests.DLL (13)] Akka.Remote.Tests.RemoteDeployerSpec.RemoteDeployer_must_be_able_to_parse_akka_actor_deployment_with_specified_remote_nodes
[13:20:22][Step 1/1] [WARNING][11/29/2015 1:20:22 PM][Thread 0689][LocalActorRefProvider(akka://RemoteDeployerSpec-53)] akka://RemoteDeployerSpec-53/temp trying to remove non-child d
[13:20:22][Test collection for Akka.Remote.Tests.DLL (13)] Akka.Remote.Tests.RemoteDeployerSpec.RemoteDeployer_must_reject_remote_deployment_when_the_source_requires_LocalScope
[13:20:22][Step 1/1] [WARNING][11/29/2015 1:20:22 PM][Thread 0675][LocalActorRefProvider(akka://RemoteDeployerSpec-54)] akka://RemoteDeployerSpec-54/temp trying to remove non-child d
[13:20:22][Test collection for Akka.Remote.Tests.DLL (13)] Akka.Remote.Tests.RemoteConsistentHashingRouterSpec.ConsistentHashingGroup_must_use_same_hash_ring_indepenent_of_self_address
[13:20:22][Step 1/1] [WARNING][11/29/2015 1:20:22 PM][Thread 0700][LocalActorRefProvider(akka://RemoteConsistentHashingRouterSpec-55)] akka://RemoteConsistentHashingRouterSpec-55/temp trying to remove non-child d
[13:20:22][Test collection for Akka.Remote.Tests.DLL (13)] Akka.Remote.Tests.RemoteRouterSpec.RemoteRouter_must_deploy_remote_routers_based_on_configuration (1s)
[13:20:23][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0704][LocalActorRefProvider(akka://RemoteRouterSpec-56)] akka://RemoteRouterSpec-56/temp trying to remove non-child e
[13:20:23][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0706][LocalActorRefProvider(akka://RemoteRouterSpec-56)] akka://RemoteRouterSpec-56/temp trying to remove non-child f
[13:20:23][Step 1/1] [ERROR][11/29/2015 1:20:23 PM][Thread 0710][[akka://MasterRemoteRouterSpec-56/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-56%40localhost%3a49274-1/endpointWriter]] AssociationError [akka.tcp://MasterRemoteRouterSpec-56@localhost:49275] -> akka.tcp://RemoteRouterSpec-56@localhost:49274: Error [Shut down address: akka.tcp://RemoteRouterSpec-56@localhost:49274] [   at Akka.Remote.EndpointReader.HandleDisassociated(DisassociateInfo info) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka.Remote\Endpoint.cs:line 1694
[13:20:23][Step 1/1]    at Akka.Actor.UntypedActor.Receive(Object message) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\UntypedActor.cs:line 22
[13:20:23][Step 1/1]    at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorBase.cs:line 155
[13:20:23][Step 1/1]    at Akka.Actor.ActorCell.ReceiveMessage(Object message) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 144
[13:20:23][Step 1/1]    at Akka.Actor.ActorCell.Invoke(Envelope envelope) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 62]
[13:20:23][Step 1/1] Cause: Unknown
[13:20:23][Step 1/1] [ERROR][11/29/2015 1:20:23 PM][Thread 0713][akka://MasterRemoteRouterSpec-56/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-56%40localhost%3a49274-1] Shut down address: akka.tcp://RemoteRouterSpec-56@localhost:49274
[13:20:23][Step 1/1] Cause: Akka.Remote.ShutDownAssociationException: Shut down address: akka.tcp://RemoteRouterSpec-56@localhost:49274 ---> Akka.Remote.Transport.InvalidAssociationException: The remote system terminated the association because it is shutting down.
[13:20:23][Step 1/1]    --- End of inner exception stack trace ---
[13:20:23][Step 1/1]    at Akka.Actor.ActorCell.HandleFailed(Failed f) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.FaultHandling.cs:line 406
[13:20:23][Step 1/1]    at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 176
[13:20:23][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0712][LocalActorRefProvider(akka://MasterRemoteRouterSpec-56)] akka://MasterRemoteRouterSpec-56/temp trying to remove non-child d
[13:20:23][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0712][LocalActorRefProvider(akka://MasterRemoteRouterSpec-56)] akka://MasterRemoteRouterSpec-56/temp trying to remove non-child e
[13:20:23][Test collection for Akka.Remote.Tests.DLL (13)] Akka.Remote.Tests.RemoteRouterSpec.RemoteRouter_must_deploy_its_children_on_remote_host_driven_by_programmatic_definition
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0720][LocalActorRefProvider(akka://RemoteRouterSpec-57)] akka://RemoteRouterSpec-57/temp trying to remove non-child e
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0719][LocalActorRefProvider(akka://RemoteRouterSpec-57)] akka://RemoteRouterSpec-57/temp trying to remove non-child f
[13:20:24][Step 1/1] [ERROR][11/29/2015 1:20:23 PM][Thread 0726][[akka://MasterRemoteRouterSpec-57/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-57%40localhost%3a49277-1/endpointWriter]] AssociationError [akka.tcp://MasterRemoteRouterSpec-57@localhost:49278] -> akka.tcp://RemoteRouterSpec-57@localhost:49277: Error [Shut down address: akka.tcp://RemoteRouterSpec-57@localhost:49277] [   at Akka.Remote.EndpointReader.HandleDisassociated(DisassociateInfo info) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka.Remote\Endpoint.cs:line 1694
[13:20:24][Step 1/1]    at Akka.Actor.UntypedActor.Receive(Object message) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\UntypedActor.cs:line 22
[13:20:24][Step 1/1]    at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorBase.cs:line 155
[13:20:24][Step 1/1]    at Akka.Actor.ActorCell.ReceiveMessage(Object message) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 144
[13:20:24][Step 1/1]    at Akka.Actor.ActorCell.Invoke(Envelope envelope) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 62]
[13:20:24][Step 1/1] Cause: Unknown
[13:20:24][Step 1/1] [ERROR][11/29/2015 1:20:23 PM][Thread 0728][akka://MasterRemoteRouterSpec-57/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-57%40localhost%3a49277-1] Shut down address: akka.tcp://RemoteRouterSpec-57@localhost:49277
[13:20:24][Step 1/1] Cause: Akka.Remote.ShutDownAssociationException: Shut down address: akka.tcp://RemoteRouterSpec-57@localhost:49277 ---> Akka.Remote.Transport.InvalidAssociationException: The remote system terminated the association because it is shutting down.
[13:20:24][Step 1/1]    --- End of inner exception stack trace ---
[13:20:24][Step 1/1]    at Akka.Actor.ActorCell.HandleFailed(Failed f) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.FaultHandling.cs:line 406
[13:20:24][Step 1/1]    at Akka.Actor.ActorCell.SystemInvoke(Envelope envelope) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 176
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0727][LocalActorRefProvider(akka://MasterRemoteRouterSpec-57)] akka://MasterRemoteRouterSpec-57/temp trying to remove non-child d
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0012][akka://MasterRemoteRouterSpec-57/deadLetters] DeadLetter from [akka://MasterRemoteRouterSpec-57/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-57%40localhost%3a49277-1/endpointWriter] to [akka://MasterRemoteRouterSpec-57/deadLetters]: <Received dead letter from [akka://MasterRemoteRouterSpec-57/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-57%40localhost%3a49277-1/endpointWriter]: Akka.Remote.EndpointWriter+FlushAndStop>
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0726][LocalActorRefProvider(akka://MasterRemoteRouterSpec-57)] akka://MasterRemoteRouterSpec-57/temp trying to remove non-child e
[13:20:24][Test collection for Akka.Remote.Tests.DLL (13)] Akka.Remote.Tests.RemoteRouterSpec.RemoteRouter_must_set_supplied_SupervisorStrategy
[13:20:24][Test collection for Akka.Remote.Tests.DLL (13)] Akka.Remote.Tests.RemoteRouterSpec.RemoteRouter_must_let_remote_deployment_be_overridden_by_remote_configuration (running for 1s)
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0734][LocalActorRefProvider(akka://RemoteRouterSpec-58)] akka://RemoteRouterSpec-58/temp trying to remove non-child e
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0736][LocalActorRefProvider(akka://RemoteRouterSpec-58)] akka://RemoteRouterSpec-58/temp trying to remove non-child f
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0651][[akka://MasterRemoteRouterSpec-58/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-58%40localhost%3a49280-1]] Association with remote system akka.tcp://RemoteRouterSpec-58@localhost:49280 has failed; address is now gated for 1000 ms. Reason is: [Disassociated]
[13:20:24][Step 1/1] [ERROR][11/29/2015 1:20:23 PM][Thread 0651][akka://MasterRemoteRouterSpec-58/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-58%40localhost%3a49280-1/endpointWriter] Disassociated
[13:20:24][Step 1/1] Cause: Akka.Remote.EndpointDisassociatedException: Disassociated
[13:20:24][Step 1/1]    at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka.Remote\Endpoint.cs:line 1085
[13:20:24][Step 1/1]    at Akka.Actor.ActorCell.<>c__DisplayClass106_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.cs:line 180
[13:20:24][Step 1/1]    at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorBase.cs:line 155
[13:20:24][Step 1/1]    at Akka.Actor.ActorCell.ReceiveMessage(Object message) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 144
[13:20:24][Step 1/1]    at Akka.Actor.ActorCell.AutoReceiveMessage(Envelope envelope) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 114
[13:20:24][Step 1/1]    at Akka.Actor.ActorCell.Invoke(Envelope envelope) in D:\BuildAgent\work\49b164d63843fb4\src\core\Akka\Actor\ActorCell.DefaultMessages.cs:line 60
[13:20:24][Step 1/1] [WARNING][11/29/2015 1:20:23 PM][Thread 0021][akka://MasterRemoteRouterSpec-58/deadLetters] DeadLetter from [akka://MasterRemoteRouterSpec-58/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-58%40localhost%3a49280-1/endpointWriter] to [akka://MasterRemoteRouterSpec-58/deadLetters]: <Received dead letter from [akka://MasterRemoteRouterSpec-58/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fRemoteRouterSpec-58%40localhost%3a49280-1/endpointWriter]: Akka.Remote.EndpointWriter+FlushAndStop>
[13:20:24][Step 1/1] System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
[13:20:24][Step 1/1]    at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
[13:20:24][Step 1/1]    at Helios.Reactor.Tcp.TcpProxyReactor.ReceiveCallback(IAsyncResult ar)
[13:20:24][Step 1/1]    at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
[13:20:24][Step 1/1]    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[13:20:24][Step 1/1]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
[13:20:24][Step 1/1]    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
[13:20:24][Step 1/1]    at System.Net.ContextAwareResult.Complete(IntPtr userToken)
[13:20:24][Step 1/1]    at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object result, IntPtr userToken)
[13:20:24][Step 1/1]    at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
[13:20:24][Step 1/1]    at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
[13:20:24][Step 1/1] Running build failed.
[13:20:24][Step 1/1] Error:
[13:20:24][Step 1/1] System.Exception: xUnit2 reported an error (Error Code 1)
[13:20:24][Step 1/1]    at Fake.Testing.XUnit2.ResultHandling.failBuildWithMessage@182-3.Invoke(String message) in C:\code\fake\src\app\FakeLib\UnitTest\XUnit\XUnit2.fs:line 182
[13:20:24][Step 1/1]    at Microsoft.FSharp.Core.OptionModule.Iterate[T](FSharpFunc`2 action, FSharpOption`1 option)
[13:20:24][Step 1/1] ##teamcity[buildStatus status='FAILURE' text='System.Exception: xUnit2 reported an error (Error Code 1)   at Fake.Testing.XUnit2.ResultHandling.failBuildWithMessage@182-3.Invoke(String message) in C:\code\fake\src\app\FakeLib\UnitTest\XUnit\XUnit2.fs:line 182   at Microsoft.FSharp.Core.OptionModule.Iterate|[T|](FSharpFunc`2 action, FSharpOption`1 option)   at Fake.Testing.XUnit2.ResultHandling.failBuildIfXUnitReportedError@186-2.Invoke(FSharpOption`1 option) in C:\code\fake\src\app\FakeLib\UnitTest\XUnit\XUnit2.fs:line 186   at Fake.Testing.XUnit2.xUnit2(FSharpFunc`2 setParams, IEnumerable`1 assemblies) in C:\code\fake\src\app\FakeLib\UnitTest\XUnit\XUnit2.fs:line 210   at FSI_0001.Build.clo@217-16.Invoke(Unit _arg10) in D:\BuildAgent\work\49b164d63843fb4\build.fsx:line 235   at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\fake\src\app\FakeLib\TargetHelper.fs:line 411']
[13:20:24][Step 1/1]    at Fake.Testing.XUnit2.ResultHandling.failBuildIfXUnitReportedError@186-2.Invoke(FSharpOption`1 option) in C:\code\fake\src\app\FakeLib\UnitTest\XUnit\XUnit2.fs:line 186
[13:20:24][Step 1/1]    at Fake.Testing.XUnit2.xUnit2(FSharpFunc`2 setParams, IEnumerable`1 assemblies) in C:\code\fake\src\app\FakeLib\UnitTest\XUnit\XUnit2.fs:line 210
[13:20:24][Step 1/1]    at FSI_0001.Build.clo@217-16.Invoke(Unit _arg10) in D:\BuildAgent\work\49b164d63843fb4\build.fsx:line 235
[13:20:24][Step 1/1]    at Fake.TargetHelper.runSingleTarget(TargetTemplate`1 target) in C:\code\fake\src\app\FakeLib\TargetHelper.fs:line 411

LOOKS like it choked out on an Akka.Remote test, but can't say for certain. Might have moved on and started running another assembly after that last error message.

@Aaronontheweb
Copy link
Member

Need some code review on this cc @akkadotnet/contributors

@Aaronontheweb
Copy link
Member

We could ofcourse just replace the counter with a bool, but that would make a weird coupling between the actorcell and the stashing plugin.

@rogeralsing I think the decoupled design you have now has better conceptual integrity, so I like your current approach

@Horusiath
Copy link
Contributor

Could you provide some before/after benchmarks on that?

@rogeralsing
Copy link
Contributor Author

Benchmarks on the single _currentEnvelopeId++; line?

@Horusiath
Copy link
Contributor

@rogeralsing consequences may not end up on a single line ;) After all, change from struct to class is not even a single line, and that had hit us hard. I just want to make sure, that we don't miss anything.

@rogeralsing
Copy link
Contributor Author

Original code, no fix applied

withoutstashfix

Code from this PR
withstashfix

(and no they are not mixed up and the new code is not actually faster than the old, its just error margins :P )

Providing a benchmark in code would be problematic as that would require that we add conditional compilation directives to enable/disable the stash fix

@rogeralsing
Copy link
Contributor Author

Related #1489
We have seen a perf drop over time, not related to stashing, but the topic was brought up here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants