feat(core): Support bidirectional communication between specific mains and specific workers

[ivov]

This PR allows specific a main to message one or more specific workers via a pubsub channel, and for a worker to message specific one or more specific mains via another pubsub channel. This is needed for Val's code execution agents setup - in scaling mode, workers will need to request script execution jobs from mains, and mains will need to send workers the results of those jobs completed by agents so that workers can continue executing.

Context

Scaling mode relies on two pubsub channels for mains and workers to communicate:

• the command channel
  • for a main to send commands to workers outside the Bull setup, e.g. to ask workers for worker IDs and statuses, to ask workers to reload the license, to ask workers to restart the event bus, to ask workers to reload external secrets providers, etc.
  • for a main to message peers in a multi-main setup, e.g. to add or remove triggers and pollers during leadership transition, to reflect workflow activation events in the UI (enabled, disabled, failed to enable), to relay execution lifecycle events for test webhooks if the main that handled the webhook differs from the main that created the webhook, to install or remove community packages based on what occurred in other mains, etc.
• the worker response channel
  • for a worker to send back results of commands, only needed by worker view right now

Implementation

In our current setup, we have

• worker/handleCommandMessageWorker.ts for worker actions in reaction to commands from the main process via the command channel,
• main/handleCommandMessageMain.ts for main process actions in reaction to messages received from peers via the command channel, and
• main/handleWorkerResponseMessageMain.ts for main process actions in reaction to responses from workers via the worker response channel.

Main-to-main and worker-to-main communications already support targeting specific recipients, so this PR extends the existing setup to support targeting specific recipients in main-to-worker communications. Please note the existing pubsub implementation needs heavy rework - this PR extends it as little as possible to unblock the agents initiative, until we have time to improve these foundations.

Testing

To test:

• override the queueModeId for a worker and for a main,
• add the below snippets to message a specific worker from a main and to message a specific main from a worker,
• add logs at handleCommandMessageWorker and handleWorkerResponseMessageMain to see the messages, and
• start both main and worker.

// start.ts - at end of initOrchestration
const redisPublisher = await Container.get(RedisService).getPubSubPublisher();
await redisPublisher.publishToCommandChannel({
	command: 'getId',
	payload: {
		id: '123',
	},
	targets: ['some-worker'], // main: message specific worker
});

// worker.ts - at end of initOrchestration
const redisPublisher = await Container.get(RedisService).getPubSubPublisher();
await redisPublisher.publishToWorkerChannel({
	workerId: this.queueModeId,
	command: 'getId',
	payload: {
		id: '123',
	},
	targets: ['some-main'], // worker: message specific main
});

[github-actions]

⚠️ Some Cypress E2E specs are failing, please fix them before merging

[cypress]

---

Test summary

397 • 0 • 0 • 0 • 1

---

Run details

Project	n8n
Status	Passed
Commit	9b7c874
Started	Aug 20, 2024 10:15 AM
Ended	Aug 20, 2024 10:19 AM
Duration	04:43 💡
OS	Linux Debian -
Browser	Electron 118

View run in Cypress Cloud ➡️

---

Flakiness

	e2e/14-mapping.cy.ts		1
1	Data mapping > maps expressions from table header

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Cloud

[github-actions]

✅ All Cypress E2E specs passed

[janober]

Got released with n8n@1.56.0