Shared validator pubkey #5883
Merged
Shared validator pubkey #5883
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR allows sharing the pubkey data between validators by using a thread-local cache for pubkey data, netting about a 400mb mem usage reduction on holesky due to us keeping 3 permanent + several ephemeral state copies in memory at all times and each state copy holding a full validator. The PR also introduces a hash cache for the key which gives ~14% speedup for a full state `hash_tree_root` - the key makes up for a large part of the `Validator` htr time. Finally, the time it takes to copy a state goes down as well from ~80m ms to ~60, for reasons similar to htr. We use a `ptr` even if a `ref` could in theory have been used - there is not much practical benefit to a `ref` (given it's mutable) while a `ptr` is cheaper and easier to copy (when copying temporary states). We could go further and cache a cooked pubkey but it turns out this is quite intrusive - in all the relevant places, we're already using a cooked key from the immutable validator data so there are no immediate performance gains of doing so while managing the compressed -> cooked key mapping would become more difficult - something for a future PR perhaps.
|
draft until post-deneb |
|
#5887 might let this build |
etan-status
reviewed
Feb 21, 2024
etan-status
reviewed
Feb 21, 2024
Comment on lines
+557
to
+559
| # This should never happen but we guard against it in case a | ||
| # default-initialized Validator instance makes it through the other safety | ||
| # nets |
There was a problem hiding this comment.
wondering if in such a hypothetical scenario, a log + exit wouldn't be more suitable. it starts injecting nasty bugs into surrounding logic
There was a problem hiding this comment.
using a zero-inited pubkey is the current behavior in this scenario (it's used in test cases)
There was a problem hiding this comment.
shouldn't it then return the correct hash_tree_root of the zero validator instead of zero hash?
etan-status
approved these changes
Feb 21, 2024
Co-authored-by: Etan Kissling <etan@status.im>
etan-status
added a commit
that referenced
this pull request
Apr 15, 2024
The current implementation of the validator key cache as introduced in #5883 leads to issues when compiling with `--gc:arc`. Namely, the assert in `injectdestructors.nim` > `destructiveMoveVar` is triggered: ```nim assert n.kind != nkSym or not hasDestructor(c, n.sym.typ) ``` `cached == nkSym`, and `n.sym.typ == ref HashedValidatorPubKeyItem` with `hasDestructor(c, n.sym.typ) == true`. Inlining the `addr ...[]` avoids the problem and allows `--gc:arc` compilation as part of LC wasm demo project. Compilation command: ```sh nim c \ -d:disable_libbacktrace \ -d:disableMarchNative \ -d:disableLTO \ -d:emscripten \ -d:release \ -d:useGcAssert \ -d:useSysAssert \ --debuginfo:off \ --nimcache:nimcache \ --os:linux \ --cpu:wasm32 \ --cc:clang \ --clang.exe:emcc \ --clang.linkerexe:emcc \ --gc:arc \ --exceptions:goto \ --define:noSignalHandler \ --define:danger \ --panics:on \ --passC:-fpic \ --passL:-Os \ --passL:-fpic \ --passC:'-pthread' \ --passL:'-pthread' \ --passC:'-sASSERTIONS' \ --passL:'-sASSERTIONS' \ --passC:'-sINITIAL_MEMORY=256MB' \ --passL:'-sINITIAL_MEMORY=256MB' \ --passC:'-sSTACK_SIZE=128MB' \ --passL:'-sSTACK_SIZE=128MB' \ --passC:'-sUSE_PTHREADS=1' \ --passL:'-sUSE_PTHREADS=1' \ --passC:'-sPTHREAD_POOL_SIZE_STRICT=0' \ --passL:'-sPTHREAD_POOL_SIZE_STRICT=0' \ --passL:'-sEXPORTED_FUNCTIONS="[_free, _malloc, _NimMain, _ETHRandomNumberCreate, _ETHConsensusConfigCreateFromYaml, _ETHConsensusConfigGetConsensusVersionAtEpoch, _ETHBeaconStateCreateFromSsz, _ETHBeaconStateDestroy, _ETHBeaconStateCopyGenesisValidatorsRoot, _ETHRootDestroy, _ETHForkDigestsCreateFromState, _ETHBeaconClockCreateFromState, _ETHBeaconClockGetSlot, _ETHLightClientStoreCreateFromBootstrap, _ETHLightClientStoreDestroy, _kETHLcSyncKind_UpdatesByRange, _kETHLcSyncKind_FinalityUpdate, _kETHLcSyncKind_OptimisticUpdate, _ETHLightClientStoreGetNextSyncTask, _ETHLightClientStoreGetMillisecondsToNextSyncTask, _ETHLightClientStoreProcessUpdatesByRange, _ETHLightClientStoreProcessFinalityUpdate, _ETHLightClientStoreProcessOptimisticUpdate, _ETHLightClientStoreGetFinalizedHeader, _ETHLightClientStoreIsNextSyncCommitteeKnown, _ETHLightClientStoreGetOptimisticHeader, _ETHLightClientStoreGetSafetyThreshold, _ETHLightClientHeaderCreateCopy, _ETHLightClientHeaderDestroy, _ETHLightClientHeaderCopyBeaconRoot, _ETHLightClientHeaderGetBeacon, _ETHBeaconBlockHeaderGetSlot, _ETHBeaconBlockHeaderGetProposerIndex, _ETHBeaconBlockHeaderGetParentRoot, _ETHBeaconBlockHeaderGetStateRoot, _ETHBeaconBlockHeaderGetBodyRoot, _ETHLightClientHeaderCopyExecutionHash, _ETHLightClientHeaderGetExecution, _ETHExecutionPayloadHeaderGetParentHash, _ETHExecutionPayloadHeaderGetFeeRecipient, _ETHExecutionPayloadHeaderGetStateRoot, _ETHExecutionPayloadHeaderGetReceiptsRoot, _ETHExecutionPayloadHeaderGetLogsBloom, _ETHExecutionPayloadHeaderGetPrevRandao, _ETHExecutionPayloadHeaderGetBlockNumber, _ETHExecutionPayloadHeaderGetGasLimit, _ETHExecutionPayloadHeaderGetGasUsed, _ETHExecutionPayloadHeaderGetTimestamp, _ETHExecutionPayloadHeaderGetExtraDataBytes, _ETHExecutionPayloadHeaderGetBaseFeePerGas, _ETHExecutionPayloadHeaderGetBlobGasUsed, _ETHExecutionPayloadHeaderGetExcessBlobGas, _ETHExecutionBlockHeaderCreateFromJson, _ETHExecutionBlockHeaderDestroy, _ETHExecutionBlockHeaderGetTransactionsRoot, _ETHExecutionBlockHeaderGetWithdrawalsRoot, _ETHTransactionsCreateFromJson, _ETHTransactionsDestroy, _ETHTransactionsGetCount, _ETHTransactionsGet, _ETHTransactionGetHash, _ETHTransactionGetFrom, _ETHTransactionGetNonce, _ETHTransactionGetMaxPriorityFeePerGas, _ETHTransactionGetMaxFeePerGas, _ETHTransactionGetGas, _ETHTransactionIsCreatingContract, _ETHTransactionGetTo, _ETHTransactionGetValue, _ETHTransactionGetInputBytes, _ETHTransactionGetBytes, _ETHTransactionGetEip6493Root, _ETHTransactionGetEip6493Bytes, _ETHTransactionGetNumEip6493SnappyBytes, _ETHReceiptsCreateFromJson, _ETHReceiptsDestroy, _ETHReceiptsGet, _ETHReceiptHasStatus, _ETHReceiptGetBytes, _ETHReceiptGetEip6493Bytes, _ETHReceiptGetNumEip6493SnappyBytes]"' \ --passL:'-sEXPORTED_RUNTIME_METHODS="[lengthBytesUTF8, stringToNewUTF8]"' \ --passL:'-Wl,--no-entry' \ --noMain:on \ --passL:'-o libnimbus_lc.js' \ nimbus-eth2/beacon_chain/libnimbus_lc/libnimbus_lc.nim ```
etan-status
added a commit
that referenced
this pull request
Apr 16, 2024
The current implementation of the validator key cache as introduced in #5883 leads to issues when compiling with `--gc:arc`. Namely, the assert in `injectdestructors.nim` > `destructiveMoveVar` is triggered: ```nim assert n.kind != nkSym or not hasDestructor(c, n.sym.typ) ``` `cached == nkSym`, and `n.sym.typ == ref HashedValidatorPubKeyItem` with `hasDestructor(c, n.sym.typ) == true`. Inlining the `addr ...[]` avoids the problem and allows `--gc:arc` compilation as part of LC wasm demo project. Compilation command: ```sh nim c \ -d:disable_libbacktrace \ -d:disableMarchNative \ -d:disableLTO \ -d:emscripten \ -d:release \ -d:useGcAssert \ -d:useSysAssert \ --debuginfo:off \ --nimcache:nimcache \ --os:linux \ --cpu:wasm32 \ --cc:clang \ --clang.exe:emcc \ --clang.linkerexe:emcc \ --gc:arc \ --exceptions:goto \ --define:noSignalHandler \ --define:danger \ --panics:on \ --passC:-fpic \ --passL:-Os \ --passL:-fpic \ --passC:'-pthread' \ --passL:'-pthread' \ --passC:'-sASSERTIONS' \ --passL:'-sASSERTIONS' \ --passC:'-sINITIAL_MEMORY=256MB' \ --passL:'-sINITIAL_MEMORY=256MB' \ --passC:'-sSTACK_SIZE=128MB' \ --passL:'-sSTACK_SIZE=128MB' \ --passC:'-sUSE_PTHREADS=1' \ --passL:'-sUSE_PTHREADS=1' \ --passC:'-sPTHREAD_POOL_SIZE_STRICT=0' \ --passL:'-sPTHREAD_POOL_SIZE_STRICT=0' \ --passL:'-sEXPORTED_FUNCTIONS="[_free, _malloc, _NimMain, _ETHRandomNumberCreate, _ETHConsensusConfigCreateFromYaml, _ETHConsensusConfigGetConsensusVersionAtEpoch, _ETHBeaconStateCreateFromSsz, _ETHBeaconStateDestroy, _ETHBeaconStateCopyGenesisValidatorsRoot, _ETHRootDestroy, _ETHForkDigestsCreateFromState, _ETHBeaconClockCreateFromState, _ETHBeaconClockGetSlot, _ETHLightClientStoreCreateFromBootstrap, _ETHLightClientStoreDestroy, _kETHLcSyncKind_UpdatesByRange, _kETHLcSyncKind_FinalityUpdate, _kETHLcSyncKind_OptimisticUpdate, _ETHLightClientStoreGetNextSyncTask, _ETHLightClientStoreGetMillisecondsToNextSyncTask, _ETHLightClientStoreProcessUpdatesByRange, _ETHLightClientStoreProcessFinalityUpdate, _ETHLightClientStoreProcessOptimisticUpdate, _ETHLightClientStoreGetFinalizedHeader, _ETHLightClientStoreIsNextSyncCommitteeKnown, _ETHLightClientStoreGetOptimisticHeader, _ETHLightClientStoreGetSafetyThreshold, _ETHLightClientHeaderCreateCopy, _ETHLightClientHeaderDestroy, _ETHLightClientHeaderCopyBeaconRoot, _ETHLightClientHeaderGetBeacon, _ETHBeaconBlockHeaderGetSlot, _ETHBeaconBlockHeaderGetProposerIndex, _ETHBeaconBlockHeaderGetParentRoot, _ETHBeaconBlockHeaderGetStateRoot, _ETHBeaconBlockHeaderGetBodyRoot, _ETHLightClientHeaderCopyExecutionHash, _ETHLightClientHeaderGetExecution, _ETHExecutionPayloadHeaderGetParentHash, _ETHExecutionPayloadHeaderGetFeeRecipient, _ETHExecutionPayloadHeaderGetStateRoot, _ETHExecutionPayloadHeaderGetReceiptsRoot, _ETHExecutionPayloadHeaderGetLogsBloom, _ETHExecutionPayloadHeaderGetPrevRandao, _ETHExecutionPayloadHeaderGetBlockNumber, _ETHExecutionPayloadHeaderGetGasLimit, _ETHExecutionPayloadHeaderGetGasUsed, _ETHExecutionPayloadHeaderGetTimestamp, _ETHExecutionPayloadHeaderGetExtraDataBytes, _ETHExecutionPayloadHeaderGetBaseFeePerGas, _ETHExecutionPayloadHeaderGetBlobGasUsed, _ETHExecutionPayloadHeaderGetExcessBlobGas, _ETHExecutionBlockHeaderCreateFromJson, _ETHExecutionBlockHeaderDestroy, _ETHExecutionBlockHeaderGetTransactionsRoot, _ETHExecutionBlockHeaderGetWithdrawalsRoot, _ETHTransactionsCreateFromJson, _ETHTransactionsDestroy, _ETHTransactionsGetCount, _ETHTransactionsGet, _ETHTransactionGetHash, _ETHTransactionGetFrom, _ETHTransactionGetNonce, _ETHTransactionGetMaxPriorityFeePerGas, _ETHTransactionGetMaxFeePerGas, _ETHTransactionGetGas, _ETHTransactionIsCreatingContract, _ETHTransactionGetTo, _ETHTransactionGetValue, _ETHTransactionGetInputBytes, _ETHTransactionGetBytes, _ETHTransactionGetEip6493Root, _ETHTransactionGetEip6493Bytes, _ETHTransactionGetNumEip6493SnappyBytes, _ETHReceiptsCreateFromJson, _ETHReceiptsDestroy, _ETHReceiptsGet, _ETHReceiptHasStatus, _ETHReceiptGetBytes, _ETHReceiptGetEip6493Bytes, _ETHReceiptGetNumEip6493SnappyBytes]"' \ --passL:'-sEXPORTED_RUNTIME_METHODS="[lengthBytesUTF8, stringToNewUTF8]"' \ --passL:'-Wl,--no-entry' \ --noMain:on \ --passL:'-o libnimbus_lc.js' \ nimbus-eth2/beacon_chain/libnimbus_lc/libnimbus_lc.nim ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR allows sharing the pubkey data between validators by using a thread-local cache for pubkey data, netting about a 400mb mem usage reduction on holesky due to us keeping 3 permanent + several ephemeral state copies in memory at all times and each state copy holding a full validator set.
The PR also introduces a hash cache for the key which gives ~14% speedup for a full state
hash_tree_root- the key makes up for a large part of theValidatorhtr time.Finally, the time it takes to copy a state goes down as well from ~80m ms to ~60, for reasons similar to htr.
We use a
ptreven if arefcould in theory have been used - there is not much practical benefit to aref(given it's mutable) while aptris cheaper and easier to copy (when copying temporary states).We could go further and cache a cooked pubkey but it turns out this is quite intrusive - in all the relevant places, we're already using a cooked key from the immutable validator data so there are no immediate performance gains of doing so while managing the compressed -> cooked key mapping would become more difficult - something for a future PR perhaps.
./ncli --print-times hashTreeRoot capella_state state-8028127-81e89e6c-a37fb323.ssz a37fb32350d0d8ef9ae665e28036ec9843dd9c8a9b520acb6b68c8065bec5191 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 118.682, 0.000, 118.682, 118.682, 1, Load file 1000.351, 0.000, 1000.351, 1000.351, 1, Compute./ncli --print-times hashTreeRoot capella_state state-8028127-81e89e6c-a37fb323.ssz a37fb32350d0d8ef9ae665e28036ec9843dd9c8a9b520acb6b68c8065bec5191 All time are ms Average, StdDev, Min, Max, Samples, Test Validation is turned off meaning that no BLS operations are performed 420.763, 0.000, 420.763, 420.763, 1, Load file 870.934, 0.000, 870.934, 870.934, 1, Compute