Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a chunking option to GreedyScheduler. #66

Open
MasonProtter opened this issue Mar 4, 2024 · 0 comments
Open

Add a chunking option to GreedyScheduler. #66

MasonProtter opened this issue Mar 4, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@MasonProtter
Copy link
Member

MasonProtter commented Mar 4, 2024

Currently when you work with the GreedyScheduler, every item in the collection has to get put! into and take!en from a Channel which has a lot of overhead if individual iterations are fast.

We could get around this by adding an option to make the greedy scheduler operate on chunks of the the collection instead of items in the collection e.g.

tmapreduce(f, op, coll, scheduler=GreedySchedule(nchunks=100))

would be the equivalent of

tmapreduce(op, chunks(coll; n=100, split=:scatter), scheduler=GreedySchedule()) do inds
    mapreduce(f, op, view(coll, inds))
end
@MasonProtter MasonProtter added the enhancement New feature or request label Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants