Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[3 / 5] Move crypto checks in the approval-distribution #4928

Merged
merged 22 commits into from
Sep 2, 2024

Conversation

alexggh
Copy link
Contributor

@alexggh alexggh commented Jul 2, 2024

Prerequisite

This is part of the work to further optimize the approval subsystems, if you want to understand the full context start with reading #4849 (comment),

Description

This PR contain changes, so that the crypto checks are performed by the approval-distribution subsystem instead of the approval-voting one. The benefit for these, is twofold:

  1. Approval-distribution won't have to wait every single time for the approval-voting to finish its job, so the work gets to be pipelined between approval-distribution and approval-voting.

  2. By running in parallel multiple instances of approval-distribution as described here [5 / 5] Introduce approval-voting-parallel #4849 (comment), this significant body of work gets to run in parallel.

Changes:

  1. When approval-voting send ApprovalDistributionMessage::NewBlocks it needs to pass the core_index and candidate_hash of the candidates.
  2. ApprovalDistribution needs to use RuntimeInfo to be able to fetch the SessionInfo from the runtime.
  3. Move approval-voting logic that checks VRF assignment into approval-distribution
  4. Move approval-voting logic that checks vote is correctly signed into approval-distribution
  5. Plumb approval-distribution and approval-voting tests to support the new logic.

Benefits

Even without parallelisation the gains are significant, for example on my machine if we run approval subsystem bench for 500 validators and 100 cores and trigger all 89 tranches of assignments and approvals, the system won't fall behind anymore because of late processing of messages.

Before change
Chain selection approved  after 11500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a

After change

Chain selection approved  after 5500 ms hash=0x0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a

TODO:

  • Run on versi.
  • Update parachain host documentation.

@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-1-5 branch from 78708d7 to 406d11c Compare July 3, 2024 12:15
@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-3-5 branch from 88e15da to 7c968e1 Compare July 3, 2024 12:16
@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-1-5 branch from 406d11c to 2281330 Compare July 5, 2024 11:36
@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-3-5 branch from 7c968e1 to 5fe747c Compare July 5, 2024 11:36
@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6635844

@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-3-5 branch from 5fe747c to e4f883e Compare July 5, 2024 13:13
@alexggh alexggh marked this pull request as ready for review July 5, 2024 13:37
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-1-5 branch from 2281330 to 8dbb088 Compare July 16, 2024 12:39
@alexggh alexggh force-pushed the alexaggh/approval-voting-parallel-3-5 branch from e4f883e to 14727c5 Compare July 16, 2024 12:40
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
…l-1-5' into alexaggh/approval-voting-parallel-3-5
- rename check_and_import in import.
- refactor un-needed variable.

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh alexggh added T0-node This PR/Issue is related to the topic “node”. T8-polkadot This PR/Issue is related to/affects the Polkadot network. labels Jul 26, 2024
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Base automatically changed from alexaggh/approval-voting-parallel-1-5 to master July 29, 2024 10:49
Copy link
Contributor

@sandreim sandreim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work @alexggh 🚀 . Left some comments and suggestions.

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh
Copy link
Contributor Author

alexggh commented Aug 14, 2024

PR ran in versi for a few days without any issues!

@sandreim
Copy link
Contributor

Anything preventing us from merging it ?

Copy link
Member

@ordian ordian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This optimization makes sense and great work 🚀
Left a few minor remarks

@alexggh
Copy link
Contributor Author

alexggh commented Aug 27, 2024

Anything preventing us from merging it ?

The only thing missing now is a security audit, @patriciobcs any thoughts when this PR will get its turn ?

@patriciobcs
Copy link

Anything preventing us from merging it ?

The only thing missing now is a security audit, @patriciobcs any thoughts when this PR will get its turn ?

Will be done by 13th September.

…ting

Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
@alexggh alexggh added this pull request to the merge queue Sep 2, 2024
Merged via the queue into master with commit 6b854ac Sep 2, 2024
182 of 185 checks passed
@alexggh alexggh deleted the alexaggh/approval-voting-parallel-3-5 branch September 2, 2024 09:33
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
T0-node This PR/Issue is related to the topic “node”. T8-polkadot This PR/Issue is related to/affects the Polkadot network.
Projects
Status: In progress
Status: Completed
Development

Successfully merging this pull request may close these issues.

7 participants