Reconnect application when a node goes down

[emiteze]

I'm currently testing some behaviors on a local dynomite cluster using Docker and Docker Compose and I'm having this problem where when I kill a dynomite node from compose, my application doesn't fallback to any other hosts and just crash. (exactly like #256)

My topology here is dynomite01 and dynomite02 on rack1 and dynomite03 on rack2. After inserting 3 registers, I do a Thread.sleep(30000) on the test thread and restart dynomite01. After restarting, my thread wakes up and I try to insert a new record.

When using setRetryPolicyFactory(new RetryNTimes.RetryFactory(10, true)) I get these errors on log output:

2019-07-04 15:45:30.707  WARN 10305 --- [           main] c.n.dyno.jedis.JedisConnectionFactory    : Caught JedisConnectionException: Unexpected end of stream.
2019-07-04 15:45:30.710  WARN 10305 --- [           main] c.n.d.c.impl.ConnectionPoolImpl          : Received FatalConnectionException; closing connection {port=8102, host=localhost} to host Host [hostname=localhost, ipAddress=null, port=8102, rack: dc1-rack1, datacenter: dc1, status: Up, hashtag=null]
2019-07-04 15:45:30.711 ERROR 10305 --- [           main] c.n.d.c.impl.HostConnectionPoolImpl      : Failed to close connection for host: Host [hostname=localhost, ipAddress=null, port=8102, rack: dc1-rack1, datacenter: dc1, status: Up, hashtag=null] java.net.SocketException: Broken pipe (Write failed)

com.netflix.dyno.connectionpool.exception.PoolOfflineException: PoolOfflineException: [host=Host [hostname=UNKNOWN, ipAddress=UNKNOWN, port=0, rack: UNKNOWN, datacenter: UNKNOW, status: Down, hashtag=null], latency=0(0), attempts=0]host pool is offline and no Racks available for fallback

	at com.netflix.dyno.connectionpool.impl.lb.HostSelectionWithFallback.getConnection(HostSelectionWithFallback.java:152)

When not using setRetryPolicyFactory(new RetryNTimes.RetryFactory(10, true)) I get these errors on log output:

2019-07-04 15:41:42.924  WARN 9064 --- [           main] c.n.dyno.jedis.JedisConnectionFactory    : Caught JedisConnectionException: Unexpected end of stream.
2019-07-04 15:41:42.925  WARN 9064 --- [           main] c.n.d.c.impl.ConnectionPoolImpl          : Received FatalConnectionException; closing connection {port=8102, host=localhost} to host Host [hostname=localhost, ipAddress=null, port=8102, rack: dc1-rack1, datacenter: dc1, status: Up, hashtag=null]
2019-07-04 15:41:42.926 ERROR 9064 --- [           main] c.n.d.c.impl.HostConnectionPoolImpl      : Failed to close connection for host: Host [hostname=localhost, ipAddress=null, port=8102, rack: dc1-rack1, datacenter: dc1, status: Up, hashtag=null] java.net.SocketException: Broken pipe (Write failed)

com.netflix.dyno.connectionpool.exception.FatalConnectionException: FatalConnectionException: [host=Host [hostname=localhost, ipAddress=null, port=8102, rack: dc1-rack1, datacenter: dc1, status: Up, hashtag=null], latency=0(0), attempts=1]redis.clients.jedis.exceptions.JedisConnectionException: Unexpected end of stream.

	at com.netflix.dyno.jedis.JedisConnectionFactory$JedisConnection.execute(JedisConnectionFactory.java:111)

Anyone else still facing this issue?