Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Akka Clustering failing to Serialize "Akka.Cluster.ClusterActorRefProvider" out of the box -- Causes entire cluster to break down on some machines #1692

Closed
BrainSlugs83 opened this issue Feb 5, 2016 · 15 comments

Comments

@BrainSlugs83
Copy link

I don't understand why, but it seems this issue happens on everyone on my team's machine but mine -- I checked and we're all using the same versions of Akka, etc (1.6), and JSON.NET (8.0) in the solution.

But the issue (from our log file) basically goes something like this:

SEED: I'm ready
NODE: I want to Join
SEED: Welcome / Up
NODE: I tried to send you some cluster life cycle message, but I failed to serialize it -- I guess I should just quit life.
SEED: Peace, yo!

The "SEED" is at address "akka.tcp://System@127.0.0.1:13337".

Relevant Log File From the "NODE" actor system:

2016/02/05 14:20:26.686|CARLOSM-PC|WARN|Akka.Remote.ReliableDeliverySupervisor
Association with remote system akka.tcp://System@127.0.0.1:13337 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointException: Failed to write message to the transport ---> Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'Provider' with type 'Akka.Cluster.ClusterActorRefProvider'. Path 'System.Provider.Transport'.
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
at Akka.Serialization.NewtonSoftJsonSerializer.ToBinary(Object obj)
at Akka.Remote.Serialization.DaemonMsgCreateSerializer.GetPropsData(Props props)
at Akka.Remote.Serialization.DaemonMsgCreateSerializer.ToBinary(Object obj)
at Akka.Serialization.Serialization.SerializeWithTransport[T](ActorSystem system, Address address, Func1 action) at Akka.Remote.MessageSerializer.Serialize(ActorSystem system, Address address, Object message) at Akka.Remote.EndpointWriter.WriteSend(Send send) --- End of inner exception stack trace --- at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level) at Akka.Remote.EndpointWriter.WriteSend(Send send) at Akka.Remote.EndpointWriter.<Writing>b__25_0(Send s) at lambda_method(Closure , Object , Action1 , Action1 , Action1 )
at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture4.Handle(T value) at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction1 partialAction)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)]

2016/02/05 14:20:26.711|CARLOSM-PC|ERROR|Akka.Actor.OneForOneStrategy
Failed to write message to the transport

2016/02/05 14:20:26.728|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message DaemonMsgCreate from akka://System/user/Document-Store to akka://System/deadLetters was not delivered. 1 dead letters encountered.

2016/02/05 14:20:26.728|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message DaemonMsgCreate from akka://System/user/Document-Notifier to akka://System/deadLetters was not delivered. 2 dead letters encountered.

2016/02/05 14:20:26.749|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message DaemonMsgCreate from akka://System/user/Logging-Repeater to akka://System/deadLetters was not delivered. 3 dead letters encountered.

2016/02/05 14:20:26.759|CARLOSM-PC|INFO|Akka.Actor.LocalActorRef
Message AckIdleCheckTimer from akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter to akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter was not delivered. 4 dead letters encountered.

2016/02/05 14:20:26.759|CARLOSM-PC|INFO|Akka.Actor.LocalActorRef
Message DeathWatchNotification from akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter to akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter was not delivered. 5 dead letters encountered.

2016/02/05 14:20:26.787|CARLOSM-PC|INFO|Akka.Actor.LocalActorRef
Message Disassociated from akka://System/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1 to akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter/endpointReader-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1 was not delivered. 6 dead letters encountered.

2016/02/05 14:20:26.793|CARLOSM-PC|INFO|Akka.Actor.LocalActorRef
Message Disassociated from NoSender to akka://System/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1 was not delivered. 7 dead letters encountered.

2016/02/05 14:20:27.401|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message GossipStatus from akka://System/system/cluster/core/daemon to akka://System/deadLetters was not delivered. 8 dead letters encountered.

2016/02/05 14:20:27.464|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message Heartbeat from akka://System/system/cluster/core/daemon/heartbeatSender to akka://System/deadLetters was not delivered. 9 dead letters encountered.

2016/02/05 14:20:32.484|CARLOSM-PC|WARN|Akka.Cluster.ClusterCoreDaemon
Cluster Node [akka.tcp://System@0.0.0.0:31010] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://System@127.0.0.1:13337, status = Up]

2016/02/05 14:20:37.520|CARLOSM-PC|INFO|Akka.Event.DummyClassForStringSources
Cluster Node [akka.tcp://System@0.0.0.0:31010] - Leader is auto-downing unreachable node [akka.tcp://System@0.0.0.0:31010]

2016/02/05 14:20:37.547|CARLOSM-PC|INFO|Akka.Cluster.ClusterCoreDaemon
Marking unreachable node [akka.tcp://System@127.0.0.1:13337] as Down

2016/02/05 14:20:38.471|CARLOSM-PC|INFO|Akka.Cluster.ClusterCoreDaemon
Leader is removing unreachable node [akka.tcp://System@127.0.0.1:13337]

@rogeralsing
Copy link
Contributor

It looks like you are doing a remote deployment as the daemonmsgcreate is involved.
What are you passing over to it?

The provider should never be serialized, it is a big internal part of akka and is not intended to be passed across the wire.

Are you using some funky constructor args to your remote actor?

/Roger

Skickat från min iPhone

6 feb. 2016 kl. 00:00 skrev Mikey notifications@github.com:

I don't understand why, but it seems this issue happens on everyone on my team's machine but mine -- I checked and we're all using the same versions of Akka, etc, and JSON.NET in the solution.

But the issue (from our log file) basically goes something like this:

SEED: I'm ready
NODE: I want to Join
SEED: Welcome / Up
NODE: I tried to send you some cluster life cycle message, but I failed to serialize it -- I guess I should just quit life.
SEED: Peace, yo!

The "SEED" is at address "akka.tcp://System@127.0.0.1:13337".

Relevant Log File From the "NODE" actor system:

2016/02/05 14:20:26.686|CARLOSM-PC|WARN|Akka.Remote.ReliableDeliverySupervisor
Association with remote system akka.tcp://System@127.0.0.1:13337 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointException: Failed to write message to the transport ---> Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'Provider' with type 'Akka.Cluster.ClusterActorRefProvider'. Path 'System.Provider.Transport'.
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
at Akka.Serialization.NewtonSoftJsonSerializer.ToBinary(Object obj)
at Akka.Remote.Serialization.DaemonMsgCreateSerializer.GetPropsData(Props props)
at Akka.Remote.Serialization.DaemonMsgCreateSerializer.ToBinary(Object obj)
at Akka.Serialization.Serialization.SerializeWithTransportT
at Akka.Remote.MessageSerializer.Serialize(ActorSystem system, Address address, Object message)
at Akka.Remote.EndpointWriter.WriteSend(Send send)
--- End of inner exception stack trace ---
at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level)
at Akka.Remote.EndpointWriter.WriteSend(Send send)
at Akka.Remote.EndpointWriter.b__25_0(Send s)
at lambda_method(Closure , Object , Action1 , Action1 , Action1 )
at Akka.Tools.MatchHandler.PartialHandlerArgumentsCapture4.Handle(T value)
at Akka.Actor.ReceiveActor.ExecutePartialMessageHandler(Object message, PartialAction`1 partialAction)
at Akka.Actor.ActorCell.<>c__DisplayClass109_0.b__0(Object m)
at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
at Akka.Actor.ActorCell.ReceiveMessage(Object message)
at Akka.Actor.ActorCell.Invoke(Envelope envelope)]

2016/02/05 14:20:26.711|CARLOSM-PC|ERROR|Akka.Actor.OneForOneStrategy
Failed to write message to the transport

2016/02/05 14:20:26.728|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message DaemonMsgCreate from akka://System/user/Document-Store to akka://System/deadLetters was not delivered. 1 dead letters encountered.

2016/02/05 14:20:26.728|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message DaemonMsgCreate from akka://System/user/Document-Notifier to akka://System/deadLetters was not delivered. 2 dead letters encountered.

2016/02/05 14:20:26.749|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message DaemonMsgCreate from akka://System/user/Logging-Repeater to akka://System/deadLetters was not delivered. 3 dead letters encountered.

2016/02/05 14:20:26.759|CARLOSM-PC|INFO|Akka.Actor.LocalActorRef
Message AckIdleCheckTimer from akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter to akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter was not delivered. 4 dead letters encountered.

2016/02/05 14:20:26.759|CARLOSM-PC|INFO|Akka.Actor.LocalActorRef
Message DeathWatchNotification from akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter to akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter was not delivered. 5 dead letters encountered.

2016/02/05 14:20:26.787|CARLOSM-PC|INFO|Akka.Actor.LocalActorRef
Message Disassociated from akka://System/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1 to akka://System/system/endpointManager/reliableEndpointWriter-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1/endpointWriter/endpointReader-akka.tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1 was not delivered. 6 dead letters encountered.

2016/02/05 14:20:26.793|CARLOSM-PC|INFO|Akka.Actor.LocalActorRef
Message Disassociated from NoSender to akka://System/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3a%2f%2fSystem%40127.0.0.1%3a13337-1 was not delivered. 7 dead letters encountered.

2016/02/05 14:20:27.401|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message GossipStatus from akka://System/system/cluster/core/daemon to akka://System/deadLetters was not delivered. 8 dead letters encountered.

2016/02/05 14:20:27.464|CARLOSM-PC|INFO|Akka.Remote.RemoteActorRefProvider+RemoteDeadLetterActorRef
Message Heartbeat from akka://System/system/cluster/core/daemon/heartbeatSender to akka://System/deadLetters was not delivered. 9 dead letters encountered.

2016/02/05 14:20:32.484|CARLOSM-PC|WARN|Akka.Cluster.ClusterCoreDaemon
Cluster Node [akka.tcp://System@0.0.0.0:31010] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://System@127.0.0.1:13337, status = Up]

2016/02/05 14:20:37.520|CARLOSM-PC|INFO|Akka.Event.DummyClassForStringSources
Cluster Node [akka.tcp://System@0.0.0.0:31010] - Leader is auto-downing unreachable node [akka.tcp://System@0.0.0.0:31010]

2016/02/05 14:20:37.547|CARLOSM-PC|INFO|Akka.Cluster.ClusterCoreDaemon
Marking unreachable node [akka.tcp://System@127.0.0.1:13337] as Down

2016/02/05 14:20:38.471|CARLOSM-PC|INFO|Akka.Cluster.ClusterCoreDaemon
Leader is removing unreachable node [akka.tcp://System@127.0.0.1:13337]


Reply to this email directly or view it on GitHub.

@BrainSlugs83
Copy link
Author

Thanks, Roger, I'll take a look and see.

@BrainSlugs83
Copy link
Author

Ahh yes, apparently I am passing some crazy stuff around in this case -- totally my fault -- these things can be looked up in the Actor instead of passed into the constructor.

This is total PEBKAC on my end, apologies for the bug -- but thanks very much for pointing me in the right direction! 👍

@rohitnandi12
Copy link

rohitnandi12 commented May 21, 2020

Is it an imposed rule that Actors which are remotely deployed should not have any constructor arguments ?
I encountered similar exception like the Author mentioned and after removing the constructor parameters from Actor class definition, It worked.
I am not clear on "why it worked?" can anyone throw some light on it ?

It looks like you are doing a remote deployment as the daemonmsgcreate is involved.
What are you passing over to it?

The provider should never be serialized, it is a big internal part of akka and is not intended to be passed across the wire.

Are you using some funky constructor args to your remote actor?

/Roger

@rogeralsing
Copy link
Contributor

Is it a imposed rule that Actors which are remotely deployed should not have any constructor arguments ?

Ctor args should work, there are however cases where it is hard to identify the correct constructor given the passed args due polymorphism and implicit conversions.

Especially with the old json serializer there were cases where int/long/byte/double/float etc was mixed up as json is untyped, thus failing to find the correct ctor overload.

@CumpsD
Copy link
Member

CumpsD commented Oct 25, 2020

I am having the same issue, but with Akka.Persistence.TCK.Journal and postgres.

It does:

var probe = CreateTestProbe();
var @event = new TestPayload(probe.Ref);

var aw = new AtomicWrite(
    new Persistent(@event, 6L, Pid, sender: ActorRefs.NoSender, writerGuid: WriterGuid));

Which ends up in the postgres provider:

case StoredAsType.Json:
    _serialize = e => new SerializationResult(NpgsqlDbType.Json, JsonConvert.SerializeObject(e.Payload, configuration.JsonSerializerSettings), null);
    _deserialize = (type, serialized, manifest, serializerId) => JsonConvert.DeserializeObject((string)serialized, type, configuration.JsonSerializerSettings);
    break;

causing it to blow up on a referencing loop on Provider:

Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'Provider' with type 'Akka.Actor.LocalActorRefProvider'. Path 'Ref.Underlying.Mailbox.Dispatcher.Configurator.Prerequisites.Settings.System.Provider.DeadLetters'.

Full exception:

Akka.Persistence.PostgreSql.Tests.Json.PostgreSqlJournalJsonSpec.Journal_should_serialize_events

Failed: Expected a message of type Akka.Persistence.WriteMessageSuccess, but received {WriteMessageRejected<actorInstanceId: 1, message: Persistent<pi...

Xunit.Sdk.TrueException
Failed: Expected a message of type Akka.Persistence.WriteMessageSuccess, but received {WriteMessageRejected<actorInstanceId: 1, message: Persistent<pid: p-1, seqNr: 6, deleted: False, manifest: , sender: , payload: Akka.Persistence.TCK.Serialization.TestPayload, writerGuid: b8e9f45b-0cb7-48d3-9f34-9643f2d45bc4>, cause: Newtonsoft.Json.JsonSerializationException: Self referencing loop detected for property 'Provider' with type 'Akka.Actor.LocalActorRefProvider'. Path 'Ref.Underlying.Mailbox.Dispatcher.Configurator.Prerequisites.Settings.System.Provider.DeadLetters'.
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CheckForCircularReference(JsonWriter writer, Object value, JsonProperty property, JsonContract contract, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, JsonSerializerSettings settings)
   at Akka.Persistence.PostgreSql.Journal.PostgreSqlQueryExecutor.<>c__DisplayClass42_0.<.ctor>b__4(IPersistentRepresentation e) in /home/cumpsd/dotfiles/repos/gh-exira/exira-backend/src/Akka.Persistence.PostgreSql/Journal/PostgreSqlQueryExecutor.cs:line 196
   at Akka.Persistence.PostgreSql.Journal.PostgreSqlQueryExecutor.WriteEvent(DbCommand command, IPersistentRepresentation e, IImmutableSet`1 tags) in /home/cumpsd/dotfiles/repos/gh-exira/exira-backend/src/Akka.Persistence.PostgreSql/Journal/PostgreSqlQueryExecutor.cs:line 489
   at Akka.Persistence.PostgreSql.Journal.PostgreSqlQueryExecutor.InsertBatchAsync(DbConnection connection, CancellationToken cancellationToken, WriteJournalBatch write) in /home/cumpsd/dotfiles/repos/gh-exira/exira-backend/src/Akka.Persistence.PostgreSql/Journal/PostgreSqlQueryExecutor.cs:line 418
   at Akka.Persistence.Sql.Common.Journal.SqlJournal.<>c__DisplayClass26_0.<<WriteMessagesAsync>b__0>d.MoveNext()>} (type Akka.Persistence.WriteMessageRejected) instead  from [akka://PostgreSqlJournalJsonSpec/deadLetters]
Expected: True
Actual:   False
   at Akka.TestKit.Xunit2.XunitAssertions.Fail(String format, Object[] args)
   at Akka.TestKit.TestKitBase.InternalExpectMsgEnvelope[T](Nullable`1 timeout, Action`2 assert, String hint, Boolean shouldLog)
   at Akka.TestKit.TestKitBase.InternalExpectMsgEnvelope[T](Nullable`1 timeout, Action`1 msgAssert, Action`1 senderAssert, String hint)
   at Akka.TestKit.TestKitBase.InternalExpectMsg[T](Nullable`1 timeout, Action`1 msgAssert, String hint)
   at Akka.TestKit.TestKitBase.ExpectMsg[T](Action`1 assert, Nullable`1 timeout, String hint)
   at Akka.Persistence.TCK.Journal.JournalSpec.Journal_should_serialize_events()

@Aaronontheweb
Copy link
Member

@CumpsD so the TestPayload - is that trying to serialize the TestProbe itself?

@CumpsD
Copy link
Member

CumpsD commented Oct 25, 2020

@Aaronontheweb it's from here: https://github.com/akkadotnet/akka.net/blob/dev/src/core/Akka.Persistence.TCK/Journal/JournalSpec.cs/#L282

TestPayload:

It has an IActorRef property, Ref:

So I'm guessing the json serializer wants to serialize that.

Judging from the error it is Ref.Underlying.Mailbox.Dispatcher.Configurator.Prerequisites.Settings.System.Provider.DeadLetters

@CumpsD
Copy link
Member

CumpsD commented Oct 25, 2020

I've tried bypassing it with ReferenceLoopHandling = ReferenceLoopHandling.Ignore to see if it would help, but then the serialisation blows out with:

Akka.Persistence.PostgreSql.Tests.Json.PostgreSqlJournalJsonSpec.Journal_should_serialize_events

Failed: Expected a message of type Akka.Persistence.WriteMessageSuccess, but received {WriteMessageRejected<actorInstanceId: 1, message: Persistent<pi...

Xunit.Sdk.TrueException
Failed: Expected a message of type Akka.Persistence.WriteMessageSuccess, but received {WriteMessageRejected<actorInstanceId: 1, message: Persistent<pid: p-1, seqNr: 6, deleted: False, manifest: , sender: , payload: Akka.Persistence.TCK.Serialization.TestPayload, writerGuid: 91c1759d-d6a6-4373-8d28-42b3c95c41a3>, cause: Newtonsoft.Json.JsonSerializationException: Error getting value from 'Provider' on 'Akka.Actor.Nobody'.
 ---> System.NotSupportedException: Nobody does not provide
   at Akka.Actor.Nobody.get_Provider()
   at lambda_method(Closure , Object )
   at Newtonsoft.Json.Serialization.ExpressionValueProvider.GetValue(Object target)
   --- End of inner exception stack trace ---
   at Newtonsoft.Json.Serialization.ExpressionValueProvider.GetValue(Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
   at Newtonsoft.Json.JsonConvert.SerializeObjectInternal(Object value, Type type, JsonSerializer jsonSerializer)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.SerializeObject(Object value, JsonSerializerSettings settings)
   at Akka.Persistence.PostgreSql.Journal.PostgreSqlQueryExecutor.<>c__DisplayClass42_0.<.ctor>b__4(IPersistentRepresentation e) in /home/cumpsd/dotfiles/repos/gh-exira/exira-backend/src/Akka.Persistence.PostgreSql/Journal/PostgreSqlQueryExecutor.cs:line 196
   at Akka.Persistence.PostgreSql.Journal.PostgreSqlQueryExecutor.WriteEvent(DbCommand command, IPersistentRepresentation e, IImmutableSet`1 tags) in /home/cumpsd/dotfiles/repos/gh-exira/exira-backend/src/Akka.Persistence.PostgreSql/Journal/PostgreSqlQueryExecutor.cs:line 489
   at Akka.Persistence.PostgreSql.Journal.PostgreSqlQueryExecutor.InsertBatchAsync(DbConnection connection, CancellationToken cancellationToken, WriteJournalBatch write) in /home/cumpsd/dotfiles/repos/gh-exira/exira-backend/src/Akka.Persistence.PostgreSql/Journal/PostgreSqlQueryExecutor.cs:line 418
   at Akka.Persistence.Sql.Common.Journal.SqlJournal.<>c__DisplayClass26_0.<<WriteMessagesAsync>b__0>d.MoveNext()>} (type Akka.Persistence.WriteMessageRejected) instead  from [akka://PostgreSqlJournalJsonSpec/deadLetters]
Expected: True
Actual:   False
   at Akka.TestKit.Xunit2.XunitAssertions.Fail(String format, Object[] args)
   at Akka.TestKit.TestKitBase.InternalExpectMsgEnvelope[T](Nullable`1 timeout, Action`2 assert, String hint, Boolean shouldLog)
   at Akka.TestKit.TestKitBase.InternalExpectMsgEnvelope[T](Nullable`1 timeout, Action`1 msgAssert, Action`1 senderAssert, String hint)
   at Akka.TestKit.TestKitBase.InternalExpectMsg[T](Nullable`1 timeout, Action`1 msgAssert, String hint)
   at Akka.TestKit.TestKitBase.ExpectMsg[T](Action`1 assert, Nullable`1 timeout, String hint)
   at Akka.Persistence.TCK.Journal.JournalSpec.Journal_should_serialize_events()

@CumpsD
Copy link
Member

CumpsD commented Oct 25, 2020

Looks a bit like this guy's error: #2486 (comment)

But I can't figure out what it's about :)

@CumpsD
Copy link
Member

CumpsD commented Oct 25, 2020

I'm starting to worry this is related? Aaronontheweb/Akka.Persistence.SqlServer@b841ef4

From #3811

So, for now in Postgres tests:

// TODO: hack. Replace when https://github.com/akkadotnet/akka.net/issues/3811
protected override bool SupportsSerialization => false;

@Aaronontheweb
Copy link
Member

I think this is the TCK working as intended then - you're not serializing IActorRef correctly inside the Postgres journal.

In order for this test to pass the following must be true:

  1. Journal must be using WithTransport serialization that encodes IActorRefs correctly - which is what resolves this issue for Cluster.Sharding and others;
  2. Journal must be using the built-in Protobuf envelope to store all messages, not JSON-ifying objects; and
  3. Deserialization must support both 1 and 2 as well.

Added this to see if everything is up to speed and voltage with Sql.Common: akkadotnet/Akka.Persistence.SqlServer#174 - looks like the test isn't passing here yet so I may not have finished patching the issue inside the Sql.Common yet. It's part of the 1.5 milestone but we've already implemented this change inside journals like Akka.Persistence.MongoDb, Azure, and some others.

You can probably disable that test for now using the same method override and just listen for updates on #3811

@CumpsD
Copy link
Member

CumpsD commented Oct 26, 2020

Looks like it yeah. For the record, it's not mine, I'm simply running the tests on my .net core clone, since the AkkaContrib is still Framework.

@Aaronontheweb
Copy link
Member

No worries - need us to review a PR there?

@CumpsD
Copy link
Member

CumpsD commented Oct 28, 2020

Not really, fixed it locally by disabling those serialisation tests, like with SQL Server and I'll just keep an eye on the progress of #3811

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants