Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat : autoshard relay api #807

Merged
merged 30 commits into from
Oct 20, 2023
Merged

feat : autoshard relay api #807

merged 30 commits into from
Oct 20, 2023

Conversation

chaitanyaprem
Copy link
Collaborator

@chaitanyaprem chaitanyaprem commented Oct 16, 2023

Description

Update Relay API to support autosharding per #674.
Following nwaku waku-org/nwaku#1983
Some refactoring is done in order to achieve this.
High level idea is to use ContentFilter for subscription and unsubscription.

Changes

  • Broadcaster is mandatory for relay to work
  • Modify relay Subscribe API to take in a contentFilter in order to subscribe to specific pubSubTopic and/or list of contentTopics
  • If subscription is for a specific content-Topic, filter messages delivered to subscription as per filter
  • Update Unsubscribe API
  • Unregister for subscriptions to be handled in broadcaster
  • Fix and update examples
  • Need to assess impact on ArraySubscriptions and refactor it
  • Implement new RPC API's to support contentTopics(Autosharding) based APIs
  • Optimize Relay code to not have msgChannel and rather increase buffersize for underlying pubsub subscription itself
  • Support more than 1 subscription for a pubSubTopic (uniqueness can be based on subID)
  • Test new autoshard relay RPC
  • Add unit tests in relay for new subscription

Note: Added few unit tests, will track a separate PR for unit tests as this PR is becoming too big.

Tests

  • Tested all new relay RPC's by running 2 nodes locally
  • Ran all existing unit tests
  • Ran service node locally

@status-im-auto
Copy link

status-im-auto commented Oct 16, 2023

Jenkins Builds

Click to see older builds (148)
Commit #️⃣ Finished (UTC) Duration Platform Result
e0e4ccd #1 2023-10-16 17:19:18 ~1 min linux 📄log
✔️ e0e4ccd #1 2023-10-16 17:20:03 ~1 min nix-flake 📄log
✔️ e0e4ccd #1 2023-10-16 17:22:21 ~4 min android 📦tgz
✔️ e0e4ccd #1 2023-10-16 17:23:50 ~5 min ios 📦tgz
✖️ e0e4ccd #1 2023-10-16 17:28:28 ~10 min tests 📄log
c15da7e #2 2023-10-16 17:21:58 ~27 sec linux 📄log
✔️ c15da7e #2 2023-10-16 17:23:27 ~1 min nix-flake 📄log
✔️ c15da7e #2 2023-10-16 17:25:35 ~3 min android 📦tgz
✔️ c15da7e #2 2023-10-16 17:27:19 ~5 min ios 📦tgz
✖️ c15da7e #2 2023-10-16 17:38:57 ~10 min tests 📄log
✖️ 5fa8512 #3 2023-10-16 19:45:04 ~22 sec tests 📄log
✖️ 5fa8512 #3 2023-10-16 19:45:08 ~25 sec tests 📄log
5fa8512 #3 2023-10-16 19:45:08 ~26 sec linux 📄log
✔️ 5fa8512 #3 2023-10-16 19:46:35 ~1 min nix-flake 📄log
✔️ 5fa8512 #3 2023-10-16 19:47:55 ~3 min android 📦tgz
✔️ 5fa8512 #3 2023-10-16 19:49:33 ~4 min ios 📦tgz
✖️ f4ffbbc #4 2023-10-16 21:18:14 ~22 sec tests 📄log
✖️ f4ffbbc #4 2023-10-16 21:18:18 ~26 sec tests 📄log
f4ffbbc #4 2023-10-16 21:18:21 ~28 sec linux 📄log
✔️ f4ffbbc #4 2023-10-16 21:19:44 ~1 min nix-flake 📄log
✔️ f4ffbbc #4 2023-10-16 21:21:42 ~3 min android 📦tgz
✔️ f4ffbbc #4 2023-10-16 21:21:51 ~3 min ios 📦tgz
✖️ fc35d3a #5 2023-10-16 21:30:57 ~25 sec tests 📄log
fc35d3a #5 2023-10-16 21:31:50 ~1 min linux 📄log
✖️ fc35d3a #5 2023-10-16 21:31:57 ~1 min tests 📄log
✔️ fc35d3a #5 2023-10-16 21:32:23 ~1 min nix-flake 📄log
✔️ fc35d3a #5 2023-10-16 21:33:44 ~3 min android 📦tgz
✔️ fc35d3a #5 2023-10-16 21:35:03 ~4 min ios 📦tgz
de83bbf #6 2023-10-16 21:52:47 ~24 sec linux 📄log
✖️ de83bbf #6 2023-10-16 21:53:32 ~1 min tests 📄log
✖️ de83bbf #6 2023-10-16 21:53:38 ~1 min tests 📄log
✔️ de83bbf #6 2023-10-16 21:54:16 ~1 min nix-flake 📄log
✔️ de83bbf #6 2023-10-16 21:55:56 ~3 min android 📦tgz
✔️ de83bbf #6 2023-10-16 21:57:09 ~4 min ios 📦tgz
✖️ 06dd50a #7 2023-10-16 22:00:52 ~1 min tests 📄log
✔️ 06dd50a #7 2023-10-16 22:01:39 ~1 min nix-flake 📄log
✔️ 06dd50a #7 2023-10-16 22:02:03 ~2 min linux 📦deb
✖️ 06dd50a #7 2023-10-16 22:02:18 ~2 min tests 📄log
✔️ 06dd50a #7 2023-10-16 22:03:22 ~3 min android 📦tgz
✔️ 06dd50a #7 2023-10-16 22:04:35 ~4 min ios 📦tgz
✔️ b4b9e3d #8 2023-10-16 22:15:30 ~1 min linux 📦deb
✔️ b4b9e3d #8 2023-10-16 22:15:46 ~1 min nix-flake 📄log
✔️ b4b9e3d #8 2023-10-16 22:17:04 ~3 min android 📦tgz
✔️ b4b9e3d #8 2023-10-16 22:18:37 ~4 min ios 📦tgz
✖️ b4b9e3d #8 2023-10-16 22:19:40 ~5 min tests 📄log
✔️ 7a4db68 #9 2023-10-16 22:21:50 ~1 min linux 📦deb
✔️ 7a4db68 #9 2023-10-16 22:22:38 ~1 min nix-flake 📄log
✔️ 7a4db68 #9 2023-10-16 22:23:54 ~3 min android 📦tgz
✔️ 7a4db68 #9 2023-10-16 22:25:02 ~4 min ios 📦tgz
✖️ 7a4db68 #9 2023-10-16 22:26:24 ~5 min tests 📄log
✔️ 7a4db68 #9 2023-10-16 22:47:10 ~2 min tests 📄log
✔️ 7a4db68 #10 2023-10-16 23:35:02 ~2 min tests 📄log
✔️ 4bcd91f #10 2023-10-16 23:41:23 ~1 min linux 📦deb
✔️ 4bcd91f #10 2023-10-16 23:41:30 ~1 min nix-flake 📄log
✔️ 4bcd91f #11 2023-10-16 23:43:19 ~3 min tests 📄log
✔️ 4bcd91f #10 2023-10-16 23:43:22 ~3 min android 📦tgz
✔️ 4bcd91f #10 2023-10-16 23:43:37 ~3 min tests 📄log
✔️ 4bcd91f #10 2023-10-16 23:45:36 ~5 min ios 📦tgz
✔️ d732304 #11 2023-10-17 19:30:52 ~1 min linux 📦deb
✔️ d732304 #11 2023-10-17 19:31:43 ~1 min nix-flake 📄log
✔️ d732304 #12 2023-10-17 19:32:03 ~2 min tests 📄log
✔️ d732304 #11 2023-10-17 19:32:39 ~2 min tests 📄log
✔️ d732304 #11 2023-10-17 19:33:28 ~3 min android 📦tgz
✔️ d732304 #11 2023-10-17 19:34:49 ~5 min ios 📦tgz
✔️ 83adec1 #12 2023-10-17 19:33:14 ~2 min linux 📦deb
✔️ 83adec1 #12 2023-10-17 19:33:38 ~1 min nix-flake 📄log
✔️ 83adec1 #12 2023-10-17 19:34:18 ~3 min ios 📦tgz
✔️ 83adec1 #13 2023-10-17 19:35:48 ~3 min tests 📄log
✔️ 83adec1 #12 2023-10-17 19:36:19 ~3 min tests 📄log
✔️ 83adec1 #12 2023-10-17 19:37:17 ~3 min android 📦tgz
✔️ 84cbd85 #13 2023-10-17 19:59:46 ~1 min nix-flake 📄log
✔️ 84cbd85 #13 2023-10-17 20:00:27 ~2 min linux 📦deb
✔️ 84cbd85 #13 2023-10-17 20:01:18 ~3 min android 📦tgz
✔️ 84cbd85 #13 2023-10-17 20:01:40 ~3 min tests 📄log
✔️ 84cbd85 #14 2023-10-17 20:01:44 ~3 min tests 📄log
✔️ 84cbd85 #13 2023-10-17 20:02:36 ~4 min ios 📦tgz
✔️ 6009fae #14 2023-10-17 21:15:11 ~1 min linux 📦deb
✔️ 6009fae #14 2023-10-17 21:15:55 ~1 min nix-flake 📄log
✔️ 6009fae #15 2023-10-17 21:16:08 ~2 min tests 📄log
✔️ 6009fae #14 2023-10-17 21:16:41 ~2 min tests 📄log
✔️ 6009fae #14 2023-10-17 21:17:38 ~3 min android 📦tgz
✔️ 6009fae #14 2023-10-17 21:18:38 ~4 min ios 📦tgz
✔️ 6dc99b0 #15 2023-10-18 02:16:17 ~1 min nix-flake 📄log
✔️ 6dc99b0 #15 2023-10-18 02:16:42 ~2 min linux 📦deb
✔️ 6dc99b0 #15 2023-10-18 02:16:42 ~2 min tests 📄log
✔️ 6dc99b0 #15 2023-10-18 02:18:26 ~4 min android 📦tgz
✔️ 6dc99b0 #15 2023-10-18 02:19:21 ~4 min ios 📦tgz
✖️ 6dc99b0 #16 2023-10-18 02:20:10 ~5 min tests 📄log
✔️ 460716d #16 2023-10-18 02:35:48 ~1 min linux 📦deb
✔️ 460716d #16 2023-10-18 02:36:36 ~1 min nix-flake 📄log
✔️ 460716d #17 2023-10-18 02:36:38 ~1 min tests 📄log
✔️ 460716d #16 2023-10-18 02:38:07 ~3 min tests 📄log
✔️ 460716d #16 2023-10-18 02:38:29 ~3 min android 📦tgz
✔️ 460716d #16 2023-10-18 02:38:54 ~4 min ios 📦tgz
✔️ a435055 #17 2023-10-18 03:04:59 ~1 min linux 📦deb
✔️ a435055 #18 2023-10-18 03:05:42 ~1 min tests 📄log
✔️ a435055 #17 2023-10-18 03:05:42 ~1 min nix-flake 📄log
✔️ a435055 #17 2023-10-18 03:06:08 ~2 min tests 📄log
✔️ a435055 #17 2023-10-18 03:07:13 ~3 min android 📦tgz
✔️ a435055 #17 2023-10-18 03:07:45 ~3 min ios 📦tgz
✔️ edf5cd2 #18 2023-10-18 19:10:51 ~1 min linux 📦deb
✔️ edf5cd2 #18 2023-10-18 19:12:07 ~2 min nix-flake 📄log
✔️ edf5cd2 #19 2023-10-18 19:13:20 ~3 min tests 📄log
✔️ edf5cd2 #18 2023-10-18 19:13:35 ~3 min android 📦tgz
✔️ edf5cd2 #18 2023-10-18 19:15:04 ~5 min ios 📦tgz
✖️ edf5cd2 #18 2023-10-18 19:15:39 ~5 min tests 📄log
✔️ cf0c1a3 #19 2023-10-18 21:27:45 ~1 min linux 📦deb
✔️ cf0c1a3 #20 2023-10-18 21:28:29 ~1 min tests 📄log
✔️ cf0c1a3 #19 2023-10-18 21:28:34 ~1 min nix-flake 📄log
✔️ cf0c1a3 #19 2023-10-18 21:28:58 ~2 min tests 📄log
✔️ cf0c1a3 #19 2023-10-18 21:30:17 ~3 min android 📦tgz
✔️ cf0c1a3 #19 2023-10-18 21:30:54 ~4 min ios 📦tgz
✔️ 4fc5134 #20 2023-10-18 22:57:20 ~1 min linux 📦deb
✔️ 4fc5134 #20 2023-10-18 22:58:11 ~1 min nix-flake 📄log
✔️ 4fc5134 #21 2023-10-18 22:58:12 ~1 min tests 📄log
✔️ 4fc5134 #20 2023-10-18 22:58:38 ~2 min tests 📄log
✔️ 4fc5134 #20 2023-10-18 23:00:02 ~3 min android 📦tgz
✔️ 4fc5134 #20 2023-10-18 23:00:51 ~4 min ios 📦tgz
✔️ b4426b2 #21 2023-10-19 00:28:46 ~1 min linux 📦deb
✔️ b4426b2 #22 2023-10-19 00:29:31 ~1 min tests 📄log
✔️ b4426b2 #21 2023-10-19 00:29:31 ~1 min nix-flake 📄log
✔️ b4426b2 #21 2023-10-19 00:29:59 ~2 min tests 📄log
✔️ b4426b2 #21 2023-10-19 00:31:36 ~3 min android 📦tgz
✔️ b4426b2 #21 2023-10-19 00:33:24 ~5 min ios 📦tgz
✔️ d7610b9 #22 2023-10-19 19:59:00 ~1 min nix-flake 📄log
✔️ d7610b9 #22 2023-10-19 19:59:50 ~2 min linux 📦deb
✔️ d7610b9 #23 2023-10-19 20:00:36 ~3 min tests 📄log
✔️ d7610b9 #22 2023-10-19 20:00:53 ~3 min tests 📄log
✔️ d7610b9 #22 2023-10-19 20:01:31 ~4 min android 📦tgz
✔️ d7610b9 #22 2023-10-19 20:02:06 ~4 min ios 📦tgz
4c22e60 #23 2023-10-20 16:30:46 ~19 sec linux 📄log
✖️ 4c22e60 #24 2023-10-20 16:30:47 ~19 sec tests 📄log
✖️ 4c22e60 #23 2023-10-20 16:30:51 ~21 sec tests 📄log
✖️ 4c22e60 #23 2023-10-20 16:31:20 ~53 sec nix-flake 📄log
4c22e60 #23 2023-10-20 16:32:44 ~2 min android 📄log
4c22e60 #23 2023-10-20 16:33:52 ~3 min ios 📄log
✔️ 9b3a0b8 #25 2023-10-20 17:39:16 ~1 min tests 📄log
✔️ 9b3a0b8 #24 2023-10-20 17:39:29 ~2 min nix-flake 📄log
✔️ 9b3a0b8 #24 2023-10-20 17:39:44 ~2 min tests 📄log
✔️ 9b3a0b8 #24 2023-10-20 17:39:49 ~2 min linux 📦deb
✔️ 9b3a0b8 #24 2023-10-20 17:41:32 ~4 min android 📦tgz
✔️ 9b3a0b8 #24 2023-10-20 17:42:32 ~5 min ios 📦tgz
✔️ b1f1b0b #25 2023-10-20 18:30:42 ~1 min linux 📦deb
✔️ b1f1b0b #25 2023-10-20 18:31:28 ~1 min nix-flake 📄log
✔️ b1f1b0b #25 2023-10-20 18:31:52 ~2 min tests 📄log
✔️ b1f1b0b #25 2023-10-20 18:33:27 ~3 min android 📦tgz
✔️ b1f1b0b #26 2023-10-20 18:33:46 ~4 min tests 📄log
✔️ b1f1b0b #25 2023-10-20 18:33:57 ~4 min ios 📦tgz
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 277721a #26 2023-10-20 18:53:23 ~1 min linux 📦deb
✔️ 277721a #27 2023-10-20 18:54:05 ~1 min tests 📄log
✔️ 277721a #26 2023-10-20 18:54:10 ~1 min nix-flake 📄log
✔️ 277721a #26 2023-10-20 18:54:32 ~2 min tests 📄log
✔️ 277721a #26 2023-10-20 18:55:45 ~3 min android 📦tgz
✔️ 277721a #26 2023-10-20 18:56:09 ~3 min ios 📦tgz
✔️ 7f9eb84 #27 2023-10-20 19:08:58 ~1 min linux 📦deb
✔️ 7f9eb84 #27 2023-10-20 19:09:45 ~1 min nix-flake 📄log
✔️ 7f9eb84 #28 2023-10-20 19:09:50 ~1 min tests 📄log
✔️ 7f9eb84 #27 2023-10-20 19:10:10 ~2 min tests 📄log
✔️ 7f9eb84 #27 2023-10-20 19:11:30 ~3 min android 📦tgz
✔️ 7f9eb84 #27 2023-10-20 19:12:16 ~4 min ios 📦tgz

@chaitanyaprem chaitanyaprem changed the title Feat : autoshard relay api feat : autoshard relay api Oct 16, 2023
@chaitanyaprem chaitanyaprem self-assigned this Oct 17, 2023
Copy link
Member

@richard-ramos richard-ramos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is looking great so far!

waku/v2/protocol/legacy_filter/waku_filter_test.go Outdated Show resolved Hide resolved
waku/v2/protocol/content_filter.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/waku_relay.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/subscription.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/subscription.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/broadcast.go Outdated Show resolved Hide resolved
@chaitanyaprem chaitanyaprem marked this pull request as ready for review October 18, 2023 23:54
cmd/waku/server/rpc/relay.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/waku_relay_test.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/broadcast.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/waku_relay.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/waku_relay.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/waku_relay.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/waku_relay.go Outdated Show resolved Hide resolved
waku/v2/protocol/relay/waku_relay.go Outdated Show resolved Hide resolved
cmd/waku/node.go Outdated Show resolved Hide resolved
Copy link
Member

@richard-ramos richard-ramos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥

@chaitanyaprem chaitanyaprem merged commit b5be83a into master Oct 20, 2023
2 checks passed
@chaitanyaprem chaitanyaprem deleted the feat/relay-api-autoshard branch October 20, 2023 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants