perf(core): Batch workflow activation to speed up startup #13191
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ivov
commented
Feb 11, 2025
| @@ -533,7 +543,6 @@ export class ActiveWorkflowManager { | |||
| } | |||
|
|
|||
| if (shouldDisplayActivationMessage) { | |||
| this.logger.info(` - ${dbWorkflow.display()}`); | |||
There was a problem hiding this comment.
Move next to its pair to prevent logs out of order:
================================
Start Active Workflows:
================================
- "My workflow 70" (ID: rriRxvVT6Y1YcUnq))
- "My workflow 71" (ID: UNlbMpRODEH9OkW2))
=> Started
=> Started
Codecov ReportAll modified and coverable lines are covered by tests ✅ 📢 Thoughts on this report? Let us know! |
n8n-assistant
bot
added
core
tomi
reviewed
Feb 11, 2025
Comment on lines
429
to
466
| const activationPromises = batch.map(async (dbWorkflow) => { | ||
| try { | ||
| const wasActivated = await this.add(dbWorkflow.id, activationMode, dbWorkflow, { | ||
| shouldPublish: false, | ||
| }); | ||
| this.logger.info(' => Started'); | ||
| } | ||
| } catch (error) { | ||
| this.errorReporter.error(error); | ||
| this.logger.info( | ||
| ' => ERROR: Workflow could not be activated on first try, keep on trying if not an auth issue', | ||
| ); | ||
| if (wasActivated) { | ||
| this.logger.info(` - ${dbWorkflow.display()})`); | ||
| this.logger.info(' => Started'); | ||
| this.logger.debug(`Successfully started workflow ${dbWorkflow.display()}`, { | ||
| workflowName: dbWorkflow.name, | ||
| workflowId: dbWorkflow.id, | ||
| }); | ||
| } | ||
| } catch (error) { | ||
| this.errorReporter.error(error); | ||
| this.logger.info( | ||
| ` => ERROR: Workflow ${dbWorkflow.display()} could not be activated on first try, keep on trying if not an auth issue`, | ||
| ); | ||
|
|
||
| this.logger.info(` ${error.message}`); | ||
| this.logger.error( | ||
| `Issue on initial workflow activation try of ${dbWorkflow.display()} (startup)`, | ||
| { | ||
| workflowName: dbWorkflow.name, | ||
| workflowId: dbWorkflow.id, | ||
| }, | ||
| ); | ||
| this.logger.info(` ${error.message}`); | ||
| this.logger.error( | ||
| `Issue on initial workflow activation try of ${dbWorkflow.display()} (startup)`, | ||
| { | ||
| workflowName: dbWorkflow.name, | ||
| workflowId: dbWorkflow.id, | ||
| }, | ||
| ); | ||
|
|
||
| // eslint-disable-next-line @typescript-eslint/no-unsafe-argument | ||
| this.executeErrorWorkflow(error, dbWorkflow, 'internal'); | ||
| // eslint-disable-next-line @typescript-eslint/no-unsafe-argument | ||
| this.executeErrorWorkflow(error, dbWorkflow, 'internal'); | ||
|
|
||
| // do not keep trying to activate on authorization error | ||
| // eslint-disable-next-line @typescript-eslint/no-unsafe-call | ||
| if (error.message.includes('Authorization')) continue; | ||
| // do not keep trying to activate on authorization error | ||
| // eslint-disable-next-line @typescript-eslint/no-unsafe-call | ||
| if (error.message.includes('Authorization')) return; | ||
|
|
||
| this.addQueuedWorkflowActivation('init', dbWorkflow); | ||
| } | ||
| this.addQueuedWorkflowActivation('init', dbWorkflow); | ||
| } | ||
| }); |
There was a problem hiding this comment.
Could we extract this into a separate method to make this more readable? Either the creation of a batch or activation of a single WF
Comment on lines
426
to
427
| for (let i = 0; i < dbWorkflows.length; i += activationBatchSize) { | ||
| const batch = dbWorkflows.slice(i, i + activationBatchSize); |
There was a problem hiding this comment.
Using e.g. lodash chunk would be clearer than implementing the chunking ourselves
4 tasks
|
|
n8n
|
||||||||||||||||||||||||||||
| Project |
n8n
|
| Branch Review |
batch-activation
|
| Run status |
|
| Run duration | 04m 36s |
| Commit |
|
| Committer | Iván Ovejero |
| View all properties for this run ↗︎ | |
| Test results | |
|---|---|
Failures
|
0
|
Flaky
|
1
|
Pending
|
5
|
Skipped
|
0
|
Passing
|
436
|
| View all changes introduced in this branch ↗︎ | |
|
✅ All Cypress E2E specs passed |
Merged
|
Got released with |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
During startup we activate workflows one by one, which can be slow for instances with many workflows to activate. This PR introduces support for batching workflow activations via
N8N_WORKFLOW_ACTIVATION_BATCH_SIZEso users can choose how much of this work to parallelize based on their use cases.Related Linear tickets, Github issues, and Community forum posts
n/a
Review / Merge checklist
release/backport(if the PR is an urgent fix that needs to be backported)