Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: supporting meta field in store #2609

Merged
merged 17 commits into from
May 6, 2024
Merged

Conversation

gabrielmer
Copy link
Contributor

@gabrielmer gabrielmer commented Apr 19, 2024

Description

Adding support for saving the meta field in the db.

Changes

  • supporting inserting and querying meta field in sqlite
  • supporting inserting and querying meta field in postgres
  • added unit tests

Issue

closes #2432

Copy link

This PR may contain changes to database schema of one of the drivers.

If you are introducing any changes to the schema, make sure the upgrade from the latest release to this change passes without any errors/issues.

Please make sure the label release-notes is added to make sure upgrade instructions properly highlight this change.

Copy link

github-actions bot commented Apr 19, 2024

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2609-rln-v2-false

Built from aafb959

@gabrielmer gabrielmer added the release-notes Issue/PR needs to be evaluated for inclusion in release notes highlights or upgrade instructions label Apr 22, 2024
@gabrielmer
Copy link
Contributor Author

Currently, messages sent before the migration appear with meta field NULL while the ones after the migration are either empty (but not NULL) or with a value. Not sure about this inconsistency. If meta field is empty, should we set it to NULL everywhere? or just as empty string?

image

cc @NagyZoltanPeter @Ivansete-status

@NagyZoltanPeter
Copy link
Contributor

Currently, messages sent before the migration appear with meta field NULL while the ones after the migration are either empty (but not NULL) or with a value. Not sure about this inconsistency. If meta field is empty, should we set it to NULL everywhere? or just as empty string?

image cc @NagyZoltanPeter @Ivansete-status

Hm, from pure DB perspective I vote for set it to NULL if meta has no value.

@gabrielmer gabrielmer force-pushed the supporting-meta-field-in-store branch from 6e07898 to 8b09c0f Compare April 26, 2024 08:01
Copy link

github-actions bot commented Apr 26, 2024

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2609-rln-v1

Built from 8066dfb

Copy link

github-actions bot commented Apr 26, 2024

You can find the image built from this PR at

quay.io/wakuorg/nwaku-pr:2609-rln-v2

Built from 8066dfb

@@ -34,6 +34,7 @@ proc msgHash*(pubSubTopic: string, msg: WakuMessage): array[32, byte] =
ctx.update(msg.payload)
ctx.update(msg.contentTopic.toBytes())
ctx.update(msg.timestamp.uint64.toBytes(Endianness.littleEndian))
# ctx.update(msg.meta) meta is not included in the message hash, as the signature goes in the meta field
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment addresses the doubt if meta field should be included in the hash.
Not sure if to delete the comment or not, it's ugly but I feel that it can be useful

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opinion there's no need to add meta to hash calculation but better to confirm with @jm-clius

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as the signature goes in the meta field What do you mean? Which signature we talking here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm referring to the signed validators feature. In order to validate a message, looks like it expects the message hash to be signed and the signature to be attached to the WakuMessage's meta field.

let recoveredSignature = SkSignature.fromRaw(msg.meta)
if recoveredSignature.isOk():
if recoveredSignature.get.verify(msgHash, protectedTopic.key):
outcome = errors.ValidationResult.Accept

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then yes indeed you can't add meta to the hash otherwise it would break this feature.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please note that the hash for the signed validator feature is an application-level message hash. It is different from the internal Waku Message hash.
For the signed validator feature, the meta field cannot be included (as pointed out, because it contains the very signature over the application-level hash). This is specified here: https://github.com/vacp2p/rfc-index/blob/main/status/raw/simple-scaling.md#dos-protection
For Waku Message hashes (the ones used in the Store and elsewhere) the meta field forms an essential part in order to allow applications to differentiate messages that may be exactly similar in all other fields: https://github.com/vacp2p/rfc-index/blob/main/waku/standards/core/14/message.md#deterministic-message-hashing

@@ -31,11 +31,11 @@ type PostgresDriver* = ref object of ArchiveDriver
const InsertRowStmtName = "InsertRow"
const InsertRowStmtDefinition = # TODO: get the sql queries from a file
"""INSERT INTO messages (id, messageHash, storedAt, contentTopic, payload, pubsubTopic,
version, timestamp) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) ON CONFLICT DO NOTHING;"""
version, timestamp, meta) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, CASE WHEN $9 = '' THEN NULL ELSE $9 END) ON CONFLICT DO NOTHING;"""
Copy link
Contributor Author

@gabrielmer gabrielmer Apr 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CASE statement is only added in the postgres statements, it doesn't seem to be supported in sqlite

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we get the data out of the DB do we process '' the same as NULL ? If not this is a problem.

Copy link
Contributor Author

@gabrielmer gabrielmer Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great point! So the drivers have to cast the values to the expected Nim data type. So when retrieving the value from the db and setting it to a Nim object, both NULL and '' are converted to an empty seq of bytes

@gabrielmer gabrielmer force-pushed the supporting-meta-field-in-store branch from f98a103 to a1e2110 Compare April 29, 2024 12:36
@gabrielmer gabrielmer requested review from Ivansete-status, SionoiS and NagyZoltanPeter and removed request for Ivansete-status and SionoiS April 29, 2024 12:36
@gabrielmer gabrielmer marked this pull request as ready for review April 30, 2024 07:43
Copy link
Collaborator

@Ivansete-status Ivansete-status left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for it!
The only interesting point of simplification would be the migration script. I wonder if we could skip the "partitions" part in that migration script and only focus on the actual partitioned table schema change

Comment on lines +20 to +34
DO $$
DECLARE
min_storedAt numeric;
max_storedAt numeric;
min_storedAtSeconds integer = 0;
max_storedAtSeconds integer = 0;
partition_name TEXT;
create_partition_stmt TEXT;
BEGIN
SELECT MIN(storedAt) into min_storedAt
FROM messages_backup;

SELECT MAX(storedAt) into max_storedAt
FROM messages_backup;

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the impression that it would be possible to just change the partitioned-table schema and that will automatically reflect that change in the current existing partitions. If that is the case, I think we can simplify the migration logic :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just tried it without the partition part (45a35e7) but the migration failed :/
So for now I think it's better to leave it this way

@@ -34,6 +34,7 @@ proc msgHash*(pubSubTopic: string, msg: WakuMessage): array[32, byte] =
ctx.update(msg.payload)
ctx.update(msg.contentTopic.toBytes())
ctx.update(msg.timestamp.uint64.toBytes(Endianness.littleEndian))
# ctx.update(msg.meta) meta is not included in the message hash, as the signature goes in the meta field
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my opinion there's no need to add meta to hash calculation but better to confirm with @jm-clius

Copy link
Contributor

@SionoiS SionoiS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -34,6 +34,7 @@ proc msgHash*(pubSubTopic: string, msg: WakuMessage): array[32, byte] =
ctx.update(msg.payload)
ctx.update(msg.contentTopic.toBytes())
ctx.update(msg.timestamp.uint64.toBytes(Endianness.littleEndian))
# ctx.update(msg.meta) meta is not included in the message hash, as the signature goes in the meta field
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as the signature goes in the meta field What do you mean? Which signature we talking here?

@@ -31,11 +31,11 @@ type PostgresDriver* = ref object of ArchiveDriver
const InsertRowStmtName = "InsertRow"
const InsertRowStmtDefinition = # TODO: get the sql queries from a file
"""INSERT INTO messages (id, messageHash, storedAt, contentTopic, payload, pubsubTopic,
version, timestamp) VALUES ($1, $2, $3, $4, $5, $6, $7, $8) ON CONFLICT DO NOTHING;"""
version, timestamp, meta) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, CASE WHEN $9 = '' THEN NULL ELSE $9 END) ON CONFLICT DO NOTHING;"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we get the data out of the DB do we process '' the same as NULL ? If not this is a problem.

@gabrielmer gabrielmer merged commit a46d445 into master May 6, 2024
14 of 15 checks passed
@gabrielmer gabrielmer deleted the supporting-meta-field-in-store branch May 6, 2024 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release-notes Issue/PR needs to be evaluated for inclusion in release notes highlights or upgrade instructions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat: add WakuMessage's meta field to db schema
5 participants