[FEATURE REQUEST] Jobs aggregation

[krhubert]

I wonder if this feature is on your roadmap.

TL;DR;
Job aggregation allows you to enqueue multiple jobs successively, and have them passed to the Worker together rather than individually. The feature allows you to batch multiple successive operations into one.

You can see the usecase and the original PR:
hibiken/asynq#339

Final docs
https://github.com/hibiken/asynq/wiki/Task-aggregation

[nexovec]

Might be a duplicate of #453

[brandur]

This one's WIP and coming very soon.

[krhubert]

Let me close this one since it's a duplicate.

[bgentry]

@krhubert actually I wonder if you might be able to provide some more context on how you’re looking to use job aggregation / grouping / batching? I’ve read your linked issue on Asynq, not sure if that reflects your current use case. Mostly curious what timeframe you’re looking to batch over, whether and how you’re wanting to partition your batches, etc.

[krhubert]

@bgentry I'm unsure if I can share details in the public discussion, but I can give more specific overview in the private conversation. I can hop on a call to talk or join slack/discord if you have some. Let me know if that is something you are interested in.

Mostly curious what timeframe you’re looking to batch over,

My usecase is very different from what I shared in the issue, although the aggregate feature described in the issue solves the problem.

The timeframe I use right now:

1. Grace Period - 5 min
2. Max Delay - 15 min

GroupGracePeriod: The grace period is renewed whenever a task with the same group key is added to the group
GroupMaxDelay: The grace period has a configurable upper bound, user can optionally set maximum delay, after which Asynq will deliver the tasks to Handler regardless of the remaining grace period

whether and how you’re wanting to partition your batches, etc.

I don't partition them - If we think about the same partitioning - because to me it can happen before adding a job, and after aggregation.

I know this context might not give you full visibility.

[bgentry]

whenever a task with the same group key is added to the group

I think this maybe hints at what I was asking about partitioning. Essentially I'm trying to understand what criteria you want to use to group your jobs, whether it's merely about grouping any jobs of the same kind, or if you want to group on some other attrs.

I also sent you an email to try to connect another way about details you don't wish to post on GitHub. Cheers ✌️