feat: Autosharding API for (relay) subscriptions

[chair28980]

Currently the only way to manipulate a relay node's pubsub topic subscriptions is through an explicit subscribe call with the pubsub topic as argument or via static configuration when setting up the node. This task tracks work to add a subscribe method to relevant Relay APIs allowing applications to provide desired content topics as (relay) subscribe arguments, while the node translates those to relay subscriptions for specific shards. This task should involve a mechanism where an application is only notified of new messages on subscribed content topics, instead of all messages on the subscribed shards.

Priority: Critical for launch

[SionoiS]

Conundrum: If subscribed to a shard and then unsubbing from a content topic on the same shard should the pubsub topic be unsubbed from?

For content topic subscriptions to not override pubsub subscriptions some state must be kept but where?

[SionoiS]

Leaving aside my architectural experiment.

When content and pubsub topic are used there's a case where unsubscribing removes other subscriptions.

In GossipSub there's a way to unsubscribe a specific handler on a topic but to use this feature I need to keep track of all handlers.
I not sure where to keep the state needed for this.

This does not belong in waku_node but maybe waku_cache would be the place?

[SionoiS]

Weekly Update

• achieved: Added content topic subscriptions for relay with autosharding via REST & JSONRPC APIs
• next: PRs feedback

[SionoiS]

Weekly Update

• achieved: Refactored and simplified the core logic
• next: More PR feedback

[SionoiS]

Weekly Update

• achieved: many PR fixes,
• blocker: explicit subscriptions in js-waku tests

[SionoiS]

Weekly Update

• achieved: fixed js-waku nwaku interop test
• blocker: js-waku PR not merged

[SionoiS]

Weekly Update

• achieved: PR merged DONE!