Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid polling in crew-powered pipelines #1183

Closed
wlandau opened this issue Nov 14, 2023 · 2 comments
Closed

Avoid polling in crew-powered pipelines #1183

wlandau opened this issue Nov 14, 2023 · 2 comments

Comments

@wlandau
Copy link
Member

wlandau commented Nov 14, 2023

Like the other pipeline algorithms in targets, https://github.com/ropensci/targets/blob/main/R/class_crew.R uses polling and exponential backoff to balance latency with CPU usage. nanonext condition variables may help us avoid polling altogether.

This issue depends on wlandau/crew#108, which depends on shikokuchuo/nanonext#20 and shikokuchuo/nanonext#21.

@wlandau wlandau self-assigned this Nov 14, 2023
@wlandau wlandau changed the title Avoid polling in crew integration Avoid polling crew-powered pipelines Nov 14, 2023
@wlandau wlandau changed the title Avoid polling crew-powered pipelines Avoid polling in crew-powered pipelines Nov 14, 2023
@wlandau wlandau added order: 3 and removed order: 1 labels Dec 4, 2023
@wlandau
Copy link
Member Author

wlandau commented Dec 5, 2023

Started this in branch 1183. The basic idea is there, but I think we need to throttle scaling using a time interval configurable by the user. I think we can create the infrastructure for this in crew, decouple it from any of the existing structures, and make it interact with the controller through the scale argument of push(), pop(), and wait().

@wlandau
Copy link
Member Author

wlandau commented Dec 5, 2023

Implemented.

@wlandau wlandau closed this as completed Dec 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant