Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After SUBSCRIBE in CLUSTER mode, an error is thrown when slots are refreshed #768

Closed
natesilva opened this issue Dec 26, 2018 · 7 comments
Closed
Labels

Comments

@natesilva
Copy link

natesilva commented Dec 26, 2018

When you have a CLUSTER connection and you issue the SUBSCRIBER command, ioredis continues to periodically refresh its slots cache by calling CLUSTER slots.

When this happens, an error is thrown (see lib/redis.js line 571): https://github.com/luin/ioredis/blob/ec1e85267f79c31825a0340f981be94f8edbed2d/lib/redis.js#L570-L573

I don’t know Redis well enough to know if CLUSTER slots should be allowed here, but this seems like a bug: either CLUSTER slots should be added to the VALID_IN_SUBSCRIBER_MODE list, or it should stop refreshing slots once it’s in SUBSCRIBER mode.

@stale
Copy link

stale bot commented Jan 28, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 7 days if no further activity occurs, but feel free to re-open a closed issue if needed.

@stale stale bot added the wontfix label Jan 28, 2019
@natesilva
Copy link
Author

This issue hasn’t been fixed, stalebot. Please keep it open.

@stale stale bot removed the wontfix label Jan 28, 2019
@luin
Copy link
Collaborator

luin commented Jan 28, 2019

@natesilva This issue should have been fixed in v4.0.1 (#697). Doesn't it work for you?

@natesilva
Copy link
Author

I can’t reproduce the error now. We are using v4.5.1 and I was able to reproduce it recently with a minimal test case, but now I can’t. 🤷🏽‍♂️ Closing, will reopen if it happens again.

@nolimitdev
Copy link

nolimitdev commented Jan 29, 2019

@natesilva
I found this bug today and I wanted to fill new issue with title Subscribe in cluster causes Unhandled error event after node is disconnected but I found this very similar issue. I think it is the same. I think this issue should be re-opened. I reproduced this error on ioredis v4.5.1 on linux debian and windows too with identical result so I will demonstrate for example result from windows.

It is not possible to avoid "Unhandled error event" in combination redis cluster + used subscribe + cluster node or all nodes goes down. There is no bug in non-cluster version but only in cluster version. I will show you results for non-cluster version (single node on port 6379) and cluster version (with 2 nodes on ports 7008 and 7009). I also tested two alternatives "A" without subscribe and "B" with subscribe.


Non-cluster (OK)

Start redis single node, then run node non-cluster.js and few seconds later shutdown node...

// file non-cluster.js
var ioredis = require('ioredis');
var redis = new ioredis({host: 'localhost', port: 6379});

redis.on('error', function(error) {
    console.log('Redis error:', error.message);
});

redis.on('ready', function() {
    console.log('Redis is ready');
});

// I will test "A" alternative without this subscribe code and "B" alternative with it
redis.subscribe('item', function(error, count) {
    if (error)
        console.log('Subscribe error:', error.message);
    else
        console.log('Subscribe count:', count);
});

// A - without subscribe:

Redis is ready
Redis error: read ECONNRESET
Redis error: connect ECONNREFUSED 127.0.0.1:6379
Redis error: connect ECONNREFUSED 127.0.0.1:6379
Redis error: connect ECONNREFUSED 127.0.0.1:6379
...

// B - with subscribe:

Redis is ready
Subscribe count: 1
Redis error: read ECONNRESET
Redis error: connect ECONNREFUSED 127.0.0.1:6379
Redis error: connect ECONNREFUSED 127.0.0.1:6379
Redis error: connect ECONNREFUSED 127.0.0.1:6379
...

There is no "Unhandled error event" in non-cluser version for both alternatives.


Cluster (BUG)

Start redis cluster, then run node cluster.js and few seconds later shutdown both cluster nodes...

// file cluster.js
var ioredis = require('ioredis');
var nodes = [{host: 'localhost', port: 7001}, {host: 'localhost', port: 7002}];
var redis = new ioredis.Cluster(nodes);

redis.on('error', function(error) {
    console.log('Redis cluster error:', error.message);
});

redis.on('ready', function() {
    console.log('Redis cluster is ready');
});

redis.on('node error', function(error) {
    console.log('Redis cluster node error:', error.message);
});

// I will test "A" alternative without this subscribe code and "B" alternative with it
redis.subscribe('item', function(error, count) {
    if (error)
        console.log('Subscribe error:', error.message);
    else
        console.log('Subscribe count:', count);
});

// A - without subscribe:

Redis cluster is ready
Redis cluster node error: timeout
Redis cluster node error: timeout
Redis cluster node error: timeout
Redis cluster node error: Connection is closed.
Redis cluster error: Failed to refresh slots cache.
Redis cluster node error: timeout
Redis cluster node error: timeout
Redis cluster error: Failed to refresh slots cache.
Redis cluster node error: timeout
Redis cluster node error: timeout
Redis cluster error: Failed to refresh slots cache.
Redis cluster node error: timeout
Redis cluster node error: timeout
...

No "Unhandled error event" in cluster version without subscribe.

// B - with subscribe:

Redis cluster is ready
Subscribe count: 1
Redis cluster node error: timeout
Redis cluster node error: timeout
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:7001
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:7001
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
Redis cluster node error: timeout
Redis cluster error: Failed to refresh slots cache.
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:7001
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:7001
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14)
Redis cluster node error: timeout
...

There is "Unhandled error event" in cluster version with subscribe alternative although all errors all correctly handled including that in subscribe callback. I think that it is unhandled by mistake somewhere in ioredis lib not by lib consumer. I also tried to listen for all cluster events but there is always this unhandled error.

@nolimitdev
Copy link

Maybe it will be usefull - similar Unhandled error events from linux - debian:

Redis cluster is ready
Subscribe count: 1
[ioredis] Unhandled error event: Error: write EPIPE
at exports._errnoException (util.js:1050:11)
at WriteWrap.afterWrite [as oncomplete] (net.js:814:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:7009
at Object.exports._errnoException (util.js:1050:11)
at exports._exceptionWithHostPort (util.js:1073:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1093:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:7009
at Object.exports._errnoException (util.js:1050:11)
at exports._exceptionWithHostPort (util.js:1073:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1093:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:7009
at Object.exports._errnoException (util.js:1050:11)
at exports._exceptionWithHostPort (util.js:1073:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1093:14)
[ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:7009
at Object.exports._errnoException (util.js:1050:11)
at exports._exceptionWithHostPort (util.js:1073:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1093:14)
...

luin added a commit that referenced this issue Jan 29, 2019
The errors will be caught by connection pool and a
new election will be made when the current subscriber
is lost, so we can safely ignore these errors.

Fix #768
luin added a commit that referenced this issue Jan 29, 2019
The errors will be caught by connection pool and a
new election will be made when the current subscriber
is lost, so we can safely ignore these errors.

Fix #768
ioredis-robot pushed a commit that referenced this issue Jan 29, 2019
## [4.6.1](v4.6.0...v4.6.1) (2019-01-29)

### Bug Fixes

* **Cluster:** ignore connection errors for subscriber. ([#790](#790)) ([f368c8a](f368c8a)), closes [#768](#768)
@ioredis-robot
Copy link
Collaborator

🎉 This issue has been resolved in version 4.6.1 🎉

The release is available on:

Your semantic-release bot 📦🚀

janus-dev87 added a commit to janus-dev87/ioredis-work that referenced this issue Mar 1, 2024
## [4.6.1](redis/ioredis@v4.6.0...v4.6.1) (2019-01-29)

### Bug Fixes

* **Cluster:** ignore connection errors for subscriber. ([#790](redis/ioredis#790)) ([f368c8a](redis/ioredis@f368c8a)), closes [#768](redis/ioredis#768)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants