Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZeroMQServer.Start: ZMQ Poller died #302

Closed
stefanmm opened this issue Oct 11, 2021 · 13 comments
Closed

ZeroMQServer.Start: ZMQ Poller died #302

stefanmm opened this issue Oct 11, 2021 · 13 comments
Labels
bug Something isn't working

Comments

@stefanmm
Copy link

Description
Program crashes when need to change theme - both from dark to light and light to dark. I am using stable Windows 11, latest version.

Expected behavior
When the time comes to change the theme, nothing happens. Icon disappears from the system tray, process is killed. This happens every day. To fix this, I start the program again and it switch theme as it should. I tried clean install and also installing via Windows store - nothing. Previous version (non "X" one) worked just fine, even on Windows 11.

Log Data
2021-10-11 16:58:43 | Error | ZeroMQServer.Start: ZMQ Poller died An existing connection was forcibly closed by the remote host.
config.yaml.txt
service.log

Commit Hash and Version
Commit: 9faf1f3
Version: 10.0.0.26

Screenshots

@stefanmm stefanmm added the bug Something isn't working label Oct 11, 2021
@Spiritreader
Copy link
Member

Spiritreader commented Oct 12, 2021

ZeroMQ uses TCP sockets to communicate.
Something on your system is causing these TCP connections to be closed.

Do you

  • regularily enable/disable your network adapters?
  • have an application running that is opening random tcp connections and terminates them ungracefully?

Update: You can switch to the beta channel and test a fix for ZMQ being able to escape controlled exception handling

Spiritreader added a commit that referenced this issue Oct 12, 2021
- updater can now downgrade beta to stable
- fix issue with disabled updater frequency textbox
- catch receive exception in Server.ReceiveReady (maybe addresses #302)
@stefanmm
Copy link
Author

I found what the problem is - VPN client. From time to time I am connecting to the VPN (for work) and as soon as I connect to the VPN "Auto Dark Mode" crashes. This happened just now on latest Beta update.

This is the error now (beta channel) when I connect to VPN:
ZeroMQServer.Start: ZMQ Poller died SocketException > An existing connection was forcibly closed by the remote host. > Boolean TryRecv(Int32, NetMQ.Core.Command ByRef)

@Spiritreader
Copy link
Member

I am unable to reproduce your issue, as such you would have to test out a few builds until I figure out what is causing this.
I've switched over the poller to run async on a NetMQ managed thread in an attempt to handle this gracefully (before brute-forcing it).

You can test out that new build if you want to (10.0.1.35)

Please, when testing, note the following:

  • It is very important that you test whether the NetMQ Poller is still responding after switching networks / connecting to your VPN
  • To do that, open the AutoDarkMode Shell (found in the install directory), and send an Alive Message.
    You should get "Response: Ok"
    image

@stefanmm
Copy link
Author

Ok I updated to the latest build (10.0.1.35) and the problem is still there.
Image 001
I will send you more details about VPN connection so you can replicate on your end.

@Spiritreader
Copy link
Member

I have made the zmq poll thread yet more resilient. In theory, it should now be able to auto-recreate sockets once they have become unavailable. I pushed this to .36

@stefanmm
Copy link
Author

Tried, there is still the problem. Error log:
2021-10-12 15:35:57 | Error | ZeroMQServer.Start: socket no longer available, acquiring new socket. error: SocketException > An existing connection was forcibly closed by the remote host. > Boolean TryRecv(Int32, NetMQ.Core.Command ByRef) 2021-10-12 15:35:57 | Error | ZeroMQServer.Start: removing and disposing socket failed: SocketException > An existing connection was forcibly closed by the remote host. > Int32 Send(Byte[]) 2021-10-12 15:35:57 | Info | ZeroMQServer.Start: socket bound to port: 13012
The problem is that the process is killed. So, even if you try to auto-recreate sockets it can't be done since the process doesn't exists anymore.

To replicate on your end:

  1. make sure Auto Night Mode is up and running
  2. install Windscribe VPN Windows client (you can try any other too)
  3. login with Free or Pro account (doesn't matter)
  4. connect to any free/pro network from the list if you do not have your own <- at this point Auto Night Mode crashes
  5. open Auto Night Mode again <- everything works again
  6. disconnect from the VPN network <- everything still works
  7. connect again to any network <- program crashes

@Spiritreader
Copy link
Member

Alright, I'll try to reproduce. Thanks!

Could you please attach the entire service.log file in the meantime?

@Spiritreader
Copy link
Member

Spiritreader commented Oct 12, 2021

image

Windscribe is telling me that it kills all sockets by default.
And moreso, it even kills sockets that target localhost (127.0.0.1) which aren't even connecting outbound to the internet.

I'm not sure if I can do anything about this if WindScribe is killing the AutoDarkModeSvc.exe process

@stefanmm
Copy link
Author

Ok, now it makes more sense. I never got that message, but after I saw this I remembered that there is one option related to that:
Image 003
I disabled it and now everything works.

I would never suspect that this could be a problem because the whole program was crashing. I will keep that option disabled, thank you for your help.

@Spiritreader
Copy link
Member

Spiritreader commented Oct 12, 2021

Update: I have continued my investigation, and it looks like this is a bug in the NetMQ library we're using. So ADM does crash after all.
I am checking out if I can find a fix for this.

Spiritreader added a commit that referenced this issue Oct 12, 2021
- switch NetMQ to named pipes
- should address (#302) as there is no more poller that could die
@Spiritreader
Copy link
Member

NetMQ has been replaced by Named Pipes.
You can turn the setting back on now (since Beta 10.0.0.39)

@stefanmm
Copy link
Author

After latest update it works! Thank you.

@Armin2208
Copy link
Member

Armin2208 commented Oct 12, 2021

damn! This is a big success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants