feat: Dump state of every epoch to S3 #8661
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Tags on state parts look like this: |
|
cc: @mm-near |
mm-near
approved these changes
Mar 8, 2023
| // Create a connection to S3. | ||
| let s3_bucket = config.client_config.state_sync_s3_bucket.clone(); | ||
| let s3_region = config.client_config.state_sync_s3_region.clone(); | ||
| let bucket = s3::Bucket::new( |
There was a problem hiding this comment.
here we assume that credentials are magically set in the environment?
There was a problem hiding this comment.
Yes, added a comment.
Added a suggestion to an error message.
Will also add to the documentation.
| // The actual dumping of state to S3. | ||
| tracing::info!(target: "state_sync_dump", shard_id, ?epoch_id, epoch_height, %sync_hash, %state_root, parts_dumped, num_parts, "Creating parts and dumping them"); | ||
| let mut res = None; | ||
| for part_id in parts_dumped..num_parts { |
There was a problem hiding this comment.
nit: I'd move this code (until 242) into separate method
There was a problem hiding this comment.
Moved as much as I can, because I can't move all of it.
The function will need to be async because it calls put_object. But passing chain to an async function is not an option, because Chain isn't Send.
| /// for example 'STATE_SYNC_DUMP:2' for shard_id=2. | ||
| fn state_sync_dump_progress_key(shard_id: ShardId) -> Vec<u8> { | ||
| let mut key = b"STATE_SYNC_DUMP:".to_vec(); | ||
| key.extend(shard_id.to_le_bytes()); |
There was a problem hiding this comment.
I also wonder if this key shouldn't contain the epoch id information
There was a problem hiding this comment.
Not including epoch id in the key, because only one epoch dump per shard is possible at a time.
I can think of adding epoch id to the key for the purpose of storing the history of dumping epochs to external storage, but doesn't seem necessary.
nikurt
added a commit
to nikurt/nearcore
that referenced
this pull request
Mar 15, 2023
* Start a thread per shard to do the dumping * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior. * No changes to the behavior of the node if those options are not enabled in `config.json`. * State is persisted to RocksDB such that restarts of the node are well handled. * Some useful metrics are exported. * The node assumes it's the only node in the this and all alternative universes that does the dumping. * * Unclear how to use multiple nodes to complete the dump faster * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags * * Do we even need tags?
nikurt
added a commit
to nikurt/nearcore
that referenced
this pull request
Mar 15, 2023
* Start a thread per shard to do the dumping * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior. * No changes to the behavior of the node if those options are not enabled in `config.json`. * State is persisted to RocksDB such that restarts of the node are well handled. * Some useful metrics are exported. * The node assumes it's the only node in the this and all alternative universes that does the dumping. * * Unclear how to use multiple nodes to complete the dump faster * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags * * Do we even need tags?
nikurt
added a commit
to nikurt/nearcore
that referenced
this pull request
Mar 20, 2023
* Start a thread per shard to do the dumping * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior. * No changes to the behavior of the node if those options are not enabled in `config.json`. * State is persisted to RocksDB such that restarts of the node are well handled. * Some useful metrics are exported. * The node assumes it's the only node in the this and all alternative universes that does the dumping. * * Unclear how to use multiple nodes to complete the dump faster * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags * * Do we even need tags?
nikurt
added a commit
to nikurt/nearcore
that referenced
this pull request
Mar 23, 2023
* Start a thread per shard to do the dumping * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior. * No changes to the behavior of the node if those options are not enabled in `config.json`. * State is persisted to RocksDB such that restarts of the node are well handled. * Some useful metrics are exported. * The node assumes it's the only node in the this and all alternative universes that does the dumping. * * Unclear how to use multiple nodes to complete the dump faster * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags * * Do we even need tags?
nikurt
added a commit
to nikurt/nearcore
that referenced
this pull request
Apr 6, 2023
* Start a thread per shard to do the dumping * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior. * No changes to the behavior of the node if those options are not enabled in `config.json`. * State is persisted to RocksDB such that restarts of the node are well handled. * Some useful metrics are exported. * The node assumes it's the only node in the this and all alternative universes that does the dumping. * * Unclear how to use multiple nodes to complete the dump faster * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags * * Do we even need tags?
nikurt
added a commit
to nikurt/nearcore
that referenced
this pull request
Apr 13, 2023
* Start a thread per shard to do the dumping * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior. * No changes to the behavior of the node if those options are not enabled in `config.json`. * State is persisted to RocksDB such that restarts of the node are well handled. * Some useful metrics are exported. * The node assumes it's the only node in the this and all alternative universes that does the dumping. * * Unclear how to use multiple nodes to complete the dump faster * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags * * Do we even need tags?
nikurt
added a commit
that referenced
this pull request
Apr 14, 2023
* Start a thread per shard to do the dumping * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior. * No changes to the behavior of the node if those options are not enabled in `config.json`. * State is persisted to RocksDB such that restarts of the node are well handled. * Some useful metrics are exported. * The node assumes it's the only node in the this and all alternative universes that does the dumping. * * Unclear how to use multiple nodes to complete the dump faster * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags * * Do we even need tags?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYconfig.jsonspecify bothconfig.state_sync.s3_bucketandconfig.state_sync.s3_regionto enable the new behavior.config.json.