Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: store v3 #2431

Merged
merged 2 commits into from
Apr 25, 2024
Merged

feat: store v3 #2431

merged 2 commits into from
Apr 25, 2024

Conversation

SionoiS
Copy link
Contributor

@SionoiS SionoiS commented Feb 14, 2024

Description

The new Store protocol version 3 will work side by side with the old version for some time.

I copied all the code from v2 as a starting point. Store v2 is now known as legacy store.

Store v3 order results a bit differently than legacy. Sorted by time and then by hash (was digest previously).
Everything remain the same except now we have the ability to query by hashes too 🎉

Changes

  • New Store v3 protocol and client logic
  • New de/serialization according to the WIP specs
  • update tests
  • rest api
  • Store v3 and archive tie in logic

Tracking #2425

Copy link

github-actions bot commented Feb 14, 2024

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2431-rln-v2-true

Built from ba28042

@SionoiS SionoiS self-assigned this Feb 20, 2024
Copy link
Contributor

@jm-clius jm-clius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Direction lgtm! Added a couple of comments. Will also be making some minor changes to the draft RFC, but will keep you informed about those. :)

waku/waku_store_v3/client.nim Outdated Show resolved Hide resolved
waku/node/waku_node.nim Outdated Show resolved Hide resolved
waku/node/waku_node.nim Outdated Show resolved Hide resolved
waku/waku_store_v3/client.nim Outdated Show resolved Hide resolved
waku/waku_store_v3/common.nim Outdated Show resolved Hide resolved
waku/waku_store_v3/rpc_codec.nim Outdated Show resolved Hide resolved
waku/waku_store_v3/protocol.nim Outdated Show resolved Hide resolved
@SionoiS
Copy link
Contributor Author

SionoiS commented Feb 28, 2024

N.B. tests can't pass before archive is updated and store v3 is tied to it.

@SionoiS SionoiS force-pushed the feat--store-v3 branch 2 times, most recently from 15603b1 to 75177d9 Compare March 12, 2024 20:29
Copy link

This PR may contain changes to database schema of one of the drivers.

If you are introducing any changes to the schema, make sure the upgrade from the latest release to this change passes without any errors/issues.

Please make sure the label release-notes is added to make sure upgrade instructions properly highlight this change.

@SionoiS SionoiS marked this pull request as ready for review March 20, 2024 12:57
Copy link
Contributor

@chaitanyaprem chaitanyaprem left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at high level only Store protocol changes.

Will do the rest again as i have to understand a lot of existing code to understand changes :)

waku/waku_store_legacy/client.nim Show resolved Hide resolved
waku/waku_store/client.nim Outdated Show resolved Hide resolved
proc encode*(index: PagingIndexRPC): ProtoBuffer =
## Encode an Index object into a ProtoBuffer
## returns the resultant ProtoBuffer
proc encode*(req: StoreQueryRequest): ProtoBuffer =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General Question:

We are manually encoding protobufs, seems error-prone and high maintenance.
Can't we use something like https://github.com/PMunch/protobuf-nim to generate the nim code?

Copy link
Contributor Author

@SionoiS SionoiS Mar 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should but if I'm not mistaken we use proto2 and not 3?

edit: nvm we do use v3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this can be taken up as an improvement as part of separate PR.

cc @Ivansete-status

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very interesting @chaitanyaprem ! I think that is something we could look at but I don't see super urgency for now as we don't change the protobuf implementation very often

wdyt @jm-clius ?

Copy link
Contributor

@jm-clius jm-clius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! The only blocking comment I have is that repeated fields are packed in proto3 (see #2509). Longer term, we should probably write our own library as wrapper for minprotobuf or use another.

apps/wakunode2/app.nim Outdated Show resolved Hide resolved
waku/waku_store/rpc_codec.nim Show resolved Hide resolved
Copy link
Collaborator

@Ivansete-status Ivansete-status left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! I think is getting a good shape!
I've added some comments that I hope you find useful.

Additionally:

  1. Shall we keep both legacy and new store testing? I have the impression that we are reducing the legacy-store testing coverage.
  2. Given that there are many changes, I think we need to validate that it is working properly from the Status point of view. @richard-ramos can help us on that matter :)

tests/node/test_wakunode_store.nim Show resolved Hide resolved
tests/node/test_wakunode_store.nim Show resolved Hide resolved
tests/waku_store/test_resume.nim Outdated Show resolved Hide resolved
waku/node/waku_node.nim Outdated Show resolved Hide resolved
waku/waku_api/rest/store/handlers.nim Outdated Show resolved Hide resolved
waku/waku_core/message/digest.nim Show resolved Hide resolved
waku/waku_store/client.nim Show resolved Hide resolved
waku/waku_store/common.nim Outdated Show resolved Hide resolved
waku/waku_store/common.nim Show resolved Hide resolved
proc encode*(index: PagingIndexRPC): ProtoBuffer =
## Encode an Index object into a ProtoBuffer
## returns the resultant ProtoBuffer
proc encode*(req: StoreQueryRequest): ProtoBuffer =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very interesting @chaitanyaprem ! I think that is something we could look at but I don't see super urgency for now as we don't change the protobuf implementation very often

wdyt @jm-clius ?

Copy link
Contributor

@jm-clius jm-clius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, based on proto3 discussion with @SionoiS

Copy link

github-actions bot commented Apr 23, 2024

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2431-rln-v1

Built from 197d62d

Copy link

github-actions bot commented Apr 23, 2024

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2431-rln-v2

Built from 197d62d

Copy link
Collaborator

@Ivansete-status Ivansete-status left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for such a great PR! 💯

Just added minor comment :)

proc sendHistoryQueryRPC(
w: WakuStoreClient, req: HistoryQuery, peer: RemotePeerInfo
): Future[HistoryResult] {.async, gcsafe.} =
let connOpt = await w.peerManager.dialPeer(peer, WakuStoreCodec)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe sth to check in the future but I can't see why dialing on every query. I think it would be better to separate the dialIn from query operations :)

Copy link
Contributor Author

@SionoiS SionoiS Apr 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would you envision as an alternative? Dialing for each request makes this part of the code stateless which simplify things.

Do you mean separate for code organization purposes?

I made some changes in waku/waku_store/client.nim

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants