Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process-based parallelism for streaming only supported on Linux #14

Open
calum-chamberlain opened this issue Apr 20, 2021 · 0 comments
Open
Labels
bug Something isn't working

Comments

@calum-chamberlain
Copy link
Member

Describe the bug
When using thread-based parallelism to run a streamer and detector concurrently the detector can interfere with the streamer collecting data. #13 fixes this for Linux by moving to Process-based parallelism, however this implementation currently relies on using the fork method of starting Processes, which is only supported on Linux. Windows and MacOS do not have a safe fork method and use spawn instead. spawn does not work with the implmentation in #13 and an adjustment needs to be made to make this compatible with spawn.

To Reproduce
Attempt to use multiprocessing rather than multithreading in streamer.py.

Expected behavior
The streamer should be able to collect data regardless of what the rest of the system is doing.

Desktop (please complete the following information):

  • OS: Windows and Mac)S

Additional context
#13 was somewhat rushed through as a fix to issues experienced by me, and I do not use MacOS or Windows for this, so the patch was not tested on these systems until running through CI systems. The changes in #13 were quite significant, and I didn't have time to correct these for spawn once I had made them. This issue is here to serve both as a reminder to me, and for anyone experiencing this issue on MacOS and Windows to let me know that it is urgent for them.

@calum-chamberlain calum-chamberlain added the bug Something isn't working label Apr 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant