Synchronization exceptions

[nwaldispuehl]

On the master:

2015-06-29 09:12:14,393 [hz.master.migration] WARN  c.h.p.InternalPartitionService - [10.100.99.14]:5701 [myUsername] [3.4.2] Failed migration from Member [10.100.99.30]:5701
java.util.concurrent.ExecutionException: com.hazelcast.spi.exception.TargetNotMemberException: Destination of migration could not be found! => com.hazelcast.partition.impl.MigrationRequestOperation{partitionId=11, migration=MigrationInfo{ partitionId=11, source=Address[10.100.99.30]:5701, destination=Address[10.100.99.34]:5701, master=Address[10.100.99.14]:5701, valid=true, processing=false}}
    at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:343) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:194) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:173) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.executeMigrateOperation(InternalPartitionServiceImpl.java:1698) [hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrateTask.run(InternalPartitionServiceImpl.java:1667) [hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.processTask(InternalPartitionServiceImpl.java:1805) [hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.doRun(InternalPartitionServiceImpl.java:1781) [hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.partition.impl.InternalPartitionServiceImpl$MigrationThread.run(InternalPartitionServiceImpl.java:1758) [hazelcast-3.4.2.jar:3.4.2]
Caused by: com.hazelcast.spi.exception.TargetNotMemberException: Destination of migration could not be found! => com.hazelcast.partition.impl.MigrationRequestOperation{partitionId=11, migration=MigrationInfo{ partitionId=11, source=Address[10.100.99.30]:5701, destination=Address[10.100.99.34]:5701, master=Address[10.100.99.14]:5701, valid=true, processing=false}}
    at com.hazelcast.partition.impl.MigrationRequestOperation.verifyExistingTarget(MigrationRequestOperation.java:170) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.partition.impl.MigrationRequestOperation.run(MigrationRequestOperation.java:65) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:749) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$500(BasicOperationService.java:725) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:699) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:643) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.access$1500(BasicOperationService.java:630) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:582) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:466) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.processPriorityMessages(BasicOperationScheduler.java:480) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:457) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:432) ~[hazelcast-3.4.2.jar:3.4.2]
    at ------ End remote and begin local stack-trace ------.(Unknown Source) ~[na:na]
    at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponse(BasicInvocationFuture.java:387) ~[hazelcast-3.4.2.jar:3.4.2]
    at com.hazelcast.spi.impl.BasicInvocationFuture.resolveApplicationResponseOrThrowException(BasicInvocationFuture.java:320) ~[hazelcast-3.4.2.jar:3.4.2]
    ... 7 common frames omitted

On another node:

Exception in thread "Thread-5" com.hazelcast.spi.exception.TargetNotMemberException: Not Member! target:Address[10.100.99.30]:5701, partitionId: 0, operation: com.hazelcast.map.impl.operation.QueryPartitionOperation, service: hz:impl:mapService
    at com.hazelcast.spi.impl.BasicInvocation.initInvocationTarget(BasicInvocation.java:327)
    at com.hazelcast.spi.impl.BasicInvocation.doInvoke(BasicInvocation.java:253)
    at com.hazelcast.spi.impl.BasicInvocation.run(BasicInvocation.java:614)
    at com.hazelcast.spi.impl.BasicPartitionInvocation.run(BasicPartitionInvocation.java:28)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:744)
    at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
    at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:92)

[nwaldispuehl]

Ah, for three nodes A, B and M where M has the role of the implicit master, this occurrs when both leave nodes A and B can communicate with the master M, but fail to do so with each other. This happens when the nodes are in the same WIFI network with client isolation activated.