Dynamic sharding implementation #174
Conversation
…t a JedisDynamicShardsProvider instead of a static shards list
- Use ReentrantReadWriteLock in AbstractDynamicShardsProvider - Fix read/write locks in Sharded - Add integration tests with concurrency
|
nice! |
|
Hi yaourt. Did you test this already? |
|
Mmmmh, to be honest, I added several tests for this but I don't remember if I ran all the Jedis test suite ... So please, gimme some time to confirm, so you will not loose your time ... |
|
yes that would be a big help thanks. I'd like your dynamic sharding! |
|
I've your answer : Results : Tests run: 215, Failures: 0, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ |
|
nice! I assume it also passed your own custom tests for add/removing shards. forwarding to jonathan |
|
Real life sample usage code .... (DAO impl NOT provided) |
|
ah ok thanks, I just wrote you a personal mail asking just those two things. Have a read anyway for another thing |
|
Up ;-) Getting started sample available here : https://gist.github.com/1211016 |
|
Will this be merged into Jedis? :) |
| resources.put(shardInfo, shardInfo.createResource()); | ||
| } | ||
| nodes.clear(); | ||
| resources.clear(); |
There was a problem hiding this comment.
Isn't this leaking resources? Don't you have to close them all rather than just clearing references to them here?
|
We agree that we don't treat ShardedJedis as core of Jedis, and for now we don't improve it. |
|
Is this feature still under consideration? As a workaround I thought about disconnecting reconnecting the connection to get the new nodes. What do you think about that? |
Please could you review the following proposal and tell me what do you think about it ?
Currently, Jedis doesn't support dynamic sharding.
If we want to add / remove a Redis server from the available shards, then the only way to achieve this is to stop the application, change the sharding configuration and then restart it.
This implementation allows various things like :
This proposal delegates the sharding configuration to a component outside of Jedis and allow to dynamically update the sharding configuration without down time.
JedisDynamicShardsProvider class is doing the bridge between this component and Jedis internals.
The external component manages the sharding configuration the way it wants and when an update is required, it only has to call the relevant method add/removeShard( ) or setShards( ) on JedisDynamicShardsProvider.
Upon such a call, JedisDynamicShardsProvider then notify (when absolutely required) Sharded instances about such a change (inspired from the Observable / Observer pattern)
When Sharded handles such a notification, then all the read access on the shard configuration are blocked until it has finished to change the sharding configuration (ReentrantReadWriteLock usage).