-
Notifications
You must be signed in to change notification settings - Fork 93
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Node (Network) Coordinated Subscriptions (part 1/3) - Centralise REP and PUB into single runtime server #4274
Node (Network) Coordinated Subscriptions (part 1/3) - Centralise REP and PUB into single runtime server #4274
Conversation
ce6fa2f
to
c871eb7
Compare
c871eb7
to
25157b1
Compare
25157b1
to
b66726c
Compare
0ad693c
to
988bde4
Compare
4b5a474
to
f06ca20
Compare
349bd08
to
0a9306d
Compare
9ff56e8
to
71da6cb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, need to go through this one very thoroughly and do some hands-on-testing, so far so good.
71da6cb
to
64bba86
Compare
64bba86
to
38eb201
Compare
Well that's nice 😁 |
38eb201
to
3a4fb6e
Compare
3549c54
to
25dd454
Compare
25dd454
to
80bcdd8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Works great, cheers.
I've poked and prodded this as best I'm able, works great.
|
||
Args: | ||
message (dict): message contents | ||
""" | ||
# TODO: If requested, coordinate publishing response/stream. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
# TODO: If requested, coordinate publishing response/stream. | |
# TODO: If requested, coordinate publishing response/stream. | |
# https://github.com/cylc/cylc-flow/issues/3329 |
This change:
This PR completes step one of the following changes needed for GraphQL subscriptions at the workflow:
In order to coordinate between PUB and REP sockets it makes sense to revive the single runtime server concept to initialize both and manage requests (REQ) for publishing (PUB) topics to subscribe (SUB) to.
This also results in less network code in the scheduler.
At present we have each network object (publisher, replier/server) instantiated by the scheduler in the main thread, then their startup/socket-bind happening in a separate thread:
![image](https://user-images.githubusercontent.com/11400777/160022599-d738a120-8880-4f56-b5c4-293f99b856b1.png)
This PR reduces the number of threads, with the publisher, subscriber coordinated in the same thread by the parent server object (instantiated by the scheduler still):
![image](https://user-images.githubusercontent.com/11400777/160021342-9e05a787-7ea6-4a43-8310-854f1cdfeda5.png)
Published articles/data/deltas are now put into a (threadsafe) queue, instead of called directly by the main thread...
Since the threading is setup by the scheduler now, the client no longer needs to import the threading library.
Running the complex workflow takes about the same time as the current master:
Requirements check-list
CONTRIBUTING.md
and added my name as a Code Contributor.setup.py
andconda-environment.yml
.