Offloading Consumers to Leaf Cluster for Mirrored Stream – Hub Cluster Load Impact?

[mwain]

I’ve been testing the creation of thousands of ephemeral consumers in a leaf node/cluster against a mirrored stream, with the goal of offloading consumer load from the central hub cluster. However, I’ve noticed an unexpected increase in CPU and messages in/out on the central hub cluster, and I would like clarification on whether this behaviour is expected or if there’s a way to better offload consumers?

Test setup:

• Shared account, APP
• Hub cluster, 3 nodes. Domain hub
• Leaf cluster, 3 nodes, using APP account. Domain leaf-0
• Stream s1 created in the Hub cluster (APP account)
• Mirrored stream s1-mirror created in the leaf cluster (APP account)

While testing with 100k ephemeral consumers created in the leaf cluster against the mirrored stream (s1-mirror), I noticed an increase in CPU usage and message traffic in/out of the central hub cluster. I had expected that consumers on the leaf cluster wouldn’t impact the hub/central cluster in this manner.

Checking the trace logs of a hub node i see this:

[89362] 2024/10/18 19:00:37.048199 [TRC] 127.0.0.1:61776 - rid:17 - <<- [LS+ cluster-leaf-0 APP _INBOX.IgO56QhsrUCmBNlDEPdhVT]
[89362] 2024/10/18 19:00:37.048265 [TRC] 127.0.0.1:61776 - rid:17 - <<- [LS+ cluster-leaf-0 APP _INBOX.IgO56QhsrUCmBNlDEPdhZN.*]
[89362] 2024/10/18 19:00:37.049614 [TRC] 127.0.0.1:61797 - lid:24 - <<- [LS+ $JS.FC.s1-mirror.mVi0FA3i.*]
[89362] 2024/10/18 19:00:37.049677 [TRC] 127.0.0.1:61776 - rid:17 - ->> [LS+ cluster-leaf-0 APP $JS.FC.s1-mirror.mVi0FA3i.*]
[89362] 2024/10/18 19:00:37.049701 [TRC] 127.0.0.1:4233 - rid:28 - ->> [LS+ cluster-leaf-0 APP $JS.FC.s1-mirror.mVi0FA3i.*]
[89362] 2024/10/18 19:00:37.278775 [TRC] 127.0.0.1:4223 - rid:22 - <<- [RMSG $NRG.AE.S-R3F-dZGfFLDO $NRG.R.SF5vgty0 42]

It seems that the hub cluster is being notified of the inbox of a consumer in the leaf cluster. Is this expected behavior? I was under the impression that the leaf cluster would handle consumers locally without placing additional load on the hub cluster.

Question:

• Is this behaviour expected when using mirrored streams and leaf clusters, specifically the increased load on the hub cluster?
• Is there a way to offload consumer load entirely to the leaf clusters without affecting the hub cluster?