refactor(core): Map out pubsub messages (no-changelog) #10566
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
n8n-assistant
bot
added
core
tomi
reviewed
Aug 27, 2024
Correct, we do not await the result of messages pushed into a channel. Publishers send them and they're eventually handled by subscribers. |
tomi
approved these changes
Aug 28, 2024
n8n
|
||||||||||||||||||||||||||||
| Project |
n8n
|
| Branch Review |
map-out-pubsub-commands
|
| Run status |
|
| Run duration | 04m 40s |
| Commit |
|
| Committer | Iván Ovejero |
| View all properties for this run ↗︎ | |
| Test results | |
|---|---|
Failures
|
0
|
Flaky
|
0
|
Pending
|
0
|
Skipped
|
0
|
Passing
|
419
|
| View all changes introduced in this branch ↗︎ | |
|
✅ All Cypress E2E specs passed |
MiloradFilipovic
added a commit
that referenced
this pull request
Aug 28, 2024
* master: ci: Fix provenance generation during NPM publish (no-changelog) (#10586) feat(core): Expose queue metrics for Prometheus (#10559) refactor(core): Map out pubsub messages (no-changelog) (#10566) fix: Fix scenario prefix not being passed (no-changelog) (#10575) refactor(core): Convert `verbose` to `debug` logs (#10574)
|
Got released with |
MiloradFilipovic
added a commit
that referenced
this pull request
Aug 28, 2024
* master: (24 commits) feat(core): Switch to MJML for email templates (#10518) fix(Gmail Trigger Node): Don't return date instances, but date strings instead (#10582) fix(core): Deduplicate sentry events using error stacktraces instead (no-changelog) (#10590) feat(editor): Implement new app layout (#10548) refactor(core): Standardize filename casing for services and Public API (no-changelog) (#10579) 🚀 Release 1.57.0 (#10587) fix(editor): Add tooltips to workflow history button (#10570) ci: Fix provenance generation during NPM publish (no-changelog) (#10586) feat(core): Expose queue metrics for Prometheus (#10559) refactor(core): Map out pubsub messages (no-changelog) (#10566) fix: Fix scenario prefix not being passed (no-changelog) (#10575) refactor(core): Convert `verbose` to `debug` logs (#10574) refactor(core): Use `@/databases/` instead of `@db/` (no-changelog) (#10573) ci: Fix destroy benchmark env workflow (no-changelog) (#10572) feat: Add benchmarking of pooled sqlite (no-changelog) (#10550) refactor(editor): User journey link to n8n.io (#10331) fix(Wait Node): Prevent waiting until invalid date (#10523) refactor(core): Standardize filename casing for controllers and databases (no-changelog) (#10564) refactor(core): Allow custom types on getCredentials (no-changelog) (#10567) fix(editor): Scale output item selector input width with value (#10555) ... # Conflicts: # packages/editor-ui/src/stores/assistant.store.ts
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In scaling mode, we have two pubsub channels for mains, workers, and webhooks to communicate: the command channel (main to workers, main to mains, main to webhooks) and the response channel (worker to mains).
This pubsub setup has two big areas for improvement:
OrchestrationHandlerService,OrchestrationHandlerMainService,OrchestrationHandlerWebhookService,OrchestrationHandlerWorkerService,OrchestrationWebhookService,OrchestrationWorkerService,OrchestrationService,RedisService,RedisServicePubSubSubscriber,RedisServicePubSubPublisher,RedisServiceBaseReceiver,RedisServiceBaseSender,RedisServiceBase,RedisClientService. Our use case does not require this much complexity, so we should aggressively simplify all this to make it easier to reason about.main/handleCommandMessageMain.ts,main/handleWorkerResponseMessageMain.ts,worker/handleCommandMessageWorker.ts,webhook/handleCommandMessageWebhook.tsWe should break all this down into individually unit-testable class methods, e.g. following the model intelemetry-event-relay.tsandlog-streaming-event-relay.ts. This will also decouple channel subscription from message handling.To start improving this, specifically the handler side, we can...
This initial PR maps out all pubsub messages and payloads for all interactions between mains, workers, and webhooks. Next steps will be...
main.message-handler.ts,worker.message-handler.ts, andwebhook.message-handler.ts,EventServiceand handle it in the relevant handler class, in a unit-tested method, and