Streams seem to stop working after a few hours

[KingScooty]

I've been testing it locally and on nodejitsu, but after a few hours, the stream seems to stop working. Is this because i'm not handling stream disconnections when a user disconnects by closing their tab/window ?

This is the code i'm using...

    twitter
        .stream('statuses/filter', {track: 'diablo'}, function(stream) {
            stream.on('data', function (data) {
                var encoded = strencode(data);
                var tweet = JSON.parse(encoded);
                var array = { 
                    "id": tweet.id,
                    "screen_name": tweet.user.screen_name,
                    "text" : tweet.text, 
                    "profile_image_url" : tweet.user.profile_image_url 
                };
                io.sockets.send(strencode(array));
            });
            stream.on('end', function (response) {
              // Handle a disconnection
            });
            stream.on('destroy', function (response) {
              // Handle a 'silent' disconnection from Twitter, no end/error event fired
            });
        });

[EdHubbell]

I was dealing with the same issue a week or 2 ago. I'm running my processes in forever.js, which will restart a node module when it shuts down. So with code like this:

  stream.on 'error', (method, code) ->
    errorTime = new Date()
    console.log 'Error: ' + code + ' ' + method + ' ' + errorTime.getHours() + ':' + errorTime.getMinutes() + ':' + errorTime.getSeconds()
    process.exit 1

  stream.on 'end', (res) ->
    endTime = new Date()
    console.log 'End' + ' ' + endTime.getHours() + ':' + endTime.getMinutes() + ':' + endTime.getSeconds()
    # Handle disconnect
    process.exit 1

  stream.on 'destroy', (res) ->
    destroyTime = new Date()
    console.log 'Destroy' + ' ' + destroyTime.getHours() + ':' + destroyTime.getMinutes() + ':' + destroyTime.getSeconds()
    # Handle 'silent' disconnect from Twitter, no end/error event fired
    process.exit 1

I'm pretty happy.

[mef]

I had the same problem, it is solved by @horixon's immortal-ntwitter: https://github.com/horixon/immortal-ntwitter
my stream is now running well.

[fender]

Is the immortal-ntwitter still needed to fix this? I saw some recent commits to ntwitter claiming reconnects have been improved?

[ram-nadella]

I have definitely been encountering this issue and it seems to silently stop because I have log statements in my destroy and end event handlers

My code looks like this

try {

    var twitter_client = new ntwitter(twitter_config);

    twitter_client.stream('statuses/filter', {'track': keywords_to_track}, function(stream) {
        stream.on('data', function (data) {
            redis_client.publish("ftweets", JSON.stringify(data));
        });

        stream.on('end', function (response) {
            console.time("Twitter stream end");
        });

        stream.on('destroy', function (response) {
            console.time("Twitter stream destroy");
        });

    });


} catch(ex) {
    console.time(ex);
}

I run this using forever, right after I start I verify it's working by connecting to the Redis server using redis-cli and issuing a MONITOR command, I can see the tweets coming in like crazy and then after a day or two I go back to check if it's still running and there is no activity and also when I do forever list it says the process is still running and there is nothing in the log file. Strange!

[horixon]

Could it be a stall?

"If 90 seconds pass with no data received, including newlines, disconnect and reconnect immediately according to the backoff strategies in the next section."

I haven't added this feature into my project immortal-ntwitter yet but just opened an issue for it.

[ghost]

I've been testing this, and those tests seem to confirm that it's indeed a stall. I reset a 90 second interval upon receiving data; the timeout expires and I'm not seeing any new tweets coming in.

[horixon]

Pull Request #109 does not handle reconnection of the stream. It only destroys the stream and emits an event. Its left up to the ntwitter consumer to recover in a way that is appropriate for there use case. One example of recovery is in immortal-twitter which reconnects the stream when its stalled.

[ram-nadella]

I ended up writing my own module to deal with the streaming part of the Twitter API (which is all I care about). Same API as ntwitter's .stream()

https://github.com/ram-nadella/tstream

It will automatically reconnect on disconnection, very naively at the moment, still a work in progress.

Emits an event for all the "Disconnect messages (disconnect)" as listed here:
https://dev.twitter.com/docs/streaming-apis/messages#Disconnect_messages_disconnect

What I've found is many instances of the Shutdown case "The feed was shutdown (possibly a machine restart)"

[ghost]

@horixon , your pull request #109 seems pretty close to the general philosophy of ntwitter of letting the user decide what to do with errors - hope we finally get to see some additional maintainers on this per #100...