-
Notifications
You must be signed in to change notification settings - Fork 638
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slack client closed, waiting for reconnect #203
Comments
Perhaps related, but I'm in a similar boat. Log messages:
|
Also experiencing the |
Seeing this as well:
Turned on debugging, the last action before the disconnect (by 10m 50s) was emitting a
Including the slack-attachment since it seems to frequently be the last thing in the logs |
I also have the same problem;
I see the code used to exit hubot but was updated at some point to not exit hubot. Since I run hubot with a supervisor, I'd actually prefer it exit -- since it would get restarted. But instead, it just sits until I manually restart with supervisor. |
Since Slack client seems to have troubles reconnecting occasionally (e.g. slackapi#203), this seems like the only way to solve it right now. I'm running Hubot using supervisor and in the case of Slack client dropping its connection it seems to me that having the Hubot process exit and get restarted from supervisor is the way to go.
Since Slack client seems to have troubles reconnecting occasionally (e.g. slackapi#203), this seems like the only way to solve it right now. I'm running Hubot using supervisor and in the case of Slack client dropping its connection it seems to me that having the Hubot process exit and get restarted from supervisor is the way to go.
Since Slack client seems to have troubles reconnecting occasionally (e.g. slackapi#203), this seems like the only way to solve it right now. I'm running Hubot using supervisor and in the case of Slack client dropping its connection it seems to me that having the Hubot process exit and get restarted from supervisor is the way to go.
From within the actual desktop client of Slack this evening I noticed it say something along the lines of "reconnecting...." and everything went grey. The client reconnected and we were back in business. Noticed that the bot was offline at that point and wouldn't you know this message appeared in the log.. [Mon Aug 24 2015 21:08:35 GMT-0700 (PDT)] INFO Slack client closed, waiting for reconnect I think it has something to do with Slack gettin funky with its protocols. Or maybe somebody just restarted some node that does some thing that some other thing needs. The days of high availability, amirite? |
Also having this issue.
|
Seeing the same thing
|
Probably we're affected this issue too. I really wish this bug is fixed in some way. For that matter, I can help. |
Just to add another voice to the chorus, I have this issue too:
|
We're running into this as well.
|
Same problem here: [Wed Nov 11 2015 11:20:21 GMT+0000 (UTC)] INFO Slack client closed, waiting for reconnect |
We came up with a 'workaround' for this - it appears that Hubot disconnects from Slack when there's been no activity visible to Hubot for a while. So, any messages in channels Hubot is in (doesn't have to be messages directed at Hubot, he sees everything). So, we just made sure that there's always some activity on Slack by making a channel containing just Hubot and Slackbot, and then asking using Slackbot's reminder functionality to have him post a reminder in the channel every hour. |
@shaundon giving this a whirl 👍🏼 |
@shaundon - we were just restarting hubort every hour ha... So it looks like the connector lacks a keep_alive... |
@shaundon couldn't use slashbots reminder since my hubot is in a private channel. Trying hubot-cron to see if i can replicate your fix |
I have slackbot send a reminder to the channel, not hubot itself. Works perfectly. |
@aensley my issue is that slackbot cannot send a reminder to a private channel |
Ah, I just made it a public channel. The only members are still slackbot and hubot. The extra channel in the list doesn't bother my team that much. |
Got the same issue here:
I'm running it on a server with a systemd service, so my temporary solution is to restart the hubot periodically via cron. |
Either exiting on disconnect (like #215) , or constantly pinging the bot user constantly (suggested #203 (comment)) both seem like workarounds to me, rather than fixing the problem. Best I can tell, this is an issue with either slack-client, or how hubot-slack sets up slack-client. |
Trying to trace this through:
Based on what people have reported, the second connection isn't actually receiving message. Setting I'd love to find a way to reliably reproduce this. It'd be pretty manual, but might be able to do something as simple as turn off networking for 10 seconds to trigger the ping timeouts. |
Actually, it looks like there's an environment variable specifically for the slack logger: |
@shaundon how would one remind every hour? I can't seem to find an example and slackbot doesn't understand my "every hour" very well.. as a sidenote, same problem here ofc |
I think just by setting 24 different reminders to go off daily. I'm not 100% as it was somebody else who set it up - I try not to use Slackbot very much for fear of making Hubot jealous. |
Hi, Is there a known workaround to this problem? I've looked through this thread and saw multiple leads but not a clear workaround. |
I run a script to check if the message
|
@lecoueyl You don't have to monitor it by yourself. You can set HUBOT_SLACK_EXIT_ON_DISCONNECT env variable to any non empty value and slack adapter is going to exit the process whenever there is disconnect, and forever will spawn it again. |
@lecoueyl What is |
@xaka actually this is unfortunately only mostly true. I have seen a few times recently my hubot gets disconnected and exits, but then when the new instance tries to reconnect it just hangs with an "unknown" connection error and I have to manually restart it. |
I think I have a fix for this. At least, it's a fix for this occurring when I had contacted Slack about debugging this by triggering a bunch of migrations on a test account and test bot, and @ofross was kind enough to sit with me while I debugged and he hit a button to cause the migration. I was able to reproduce it consistently easily enough, and the hard part was finding where to detect the upcoming migration. I had a workaround by changing hubot-slack, but I was worried the conditional I added it under was being used for reconnect when the ping was too old, and would cause double pings. That lead me to adding explicitly handling the migration events. I'm pretty sure you can install a version of node-slack-client in your hubot, and it will use that version rather than what hubot-slack specifies:
It's a little complicated to release this, because master has the 2.x series, and there's a lot of changes without a 1.4.x branch I can work against. I filed slackapi/node-slack-sdk#180 to figure that out. I still need to find a way to recreate the 'ping too old' condition to make sure that is fixed too. I am trying to figure out how to use toxiproxy to do it. |
@aensley yes I run it with forever. My script:
|
with debug enabled:
|
I run hubot with a custom script that keeps the bot online, just like forever.sh, but simpler :)
I set the But the problem is that sometimes it crashes with Any ideas ? Thanks |
@andreimavenhut Try doing Also, change line https://github.com/slackhq/hubot-slack/blob/master/src/slack.coffee#L54 to |
@anupdhml Tried that, here is the output:
The |
This PR should resolve most of the issues you are seeing: #309 Many of these disconnects are related to either a series of outages in November (https://status.slack.com/2015-11) or regular team migrations. We do regularly disconnect, but we have a number of mechanisms in place to help clients gracefully and quickly reconnect (or avoid the need to reconnect altogether), many of which are now handled in the latest version of Could any of you give this PR a spin and see if it helps with your reconnect issues? |
What do I put in the package.json to test the latest version? |
@marceloslacerda I believe the syntax would be: |
After the upgrade the problem stopped. |
Yep; we've been connected fine for 2 weeks now :) |
Outstanding! |
I've noticed that Hubot disconnects from Slack after a period of inactivity. I'm seeing this message in log but after that nothing happens and hubot no longer appears connected in Slack channels. At that point, only a manual restart will do.
[Sun Jun 21 2015 09:27:39 GMT+0300 (EEST)] INFO Slack client now connected
[Sun Jun 21 2015 09:29:36 GMT+0300 (EEST)] INFO Slack client closed, waiting for reconnect
How hubot-slack suppose to recover from this this state?
The text was updated successfully, but these errors were encountered: