-
Notifications
You must be signed in to change notification settings - Fork 622
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: State Sync from External Storage #8789
feat: State Sync from External Storage #8789
Commits on Apr 13, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 0039c20 - Browse repository at this point
Copy the full SHA 0039c20View commit details -
Configuration menu - View commit details
-
Copy full SHA for e9cb32c - Browse repository at this point
Copy the full SHA e9cb32cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 95ff6dc - Browse repository at this point
Copy the full SHA 95ff6dcView commit details -
Configuration menu - View commit details
-
Copy full SHA for c3a0112 - Browse repository at this point
Copy the full SHA c3a0112View commit details -
Configuration menu - View commit details
-
Copy full SHA for 85a8ded - Browse repository at this point
Copy the full SHA 85a8dedView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f0d69c - Browse repository at this point
Copy the full SHA 4f0d69cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 556248f - Browse repository at this point
Copy the full SHA 556248fView commit details -
Configuration menu - View commit details
-
Copy full SHA for b15dd00 - Browse repository at this point
Copy the full SHA b15dd00View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3e17aba - Browse repository at this point
Copy the full SHA 3e17abaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8105005 - Browse repository at this point
Copy the full SHA 8105005View commit details -
Configuration menu - View commit details
-
Copy full SHA for cb2731e - Browse repository at this point
Copy the full SHA cb2731eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 98cd7cf - Browse repository at this point
Copy the full SHA 98cd7cfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 712ca55 - Browse repository at this point
Copy the full SHA 712ca55View commit details -
Configuration menu - View commit details
-
Copy full SHA for bc953ea - Browse repository at this point
Copy the full SHA bc953eaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 61671f5 - Browse repository at this point
Copy the full SHA 61671f5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0667d8b - Browse repository at this point
Copy the full SHA 0667d8bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0063fa8 - Browse repository at this point
Copy the full SHA 0063fa8View commit details -
Configuration menu - View commit details
-
Copy full SHA for f9b7f24 - Browse repository at this point
Copy the full SHA f9b7f24View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2195391 - Browse repository at this point
Copy the full SHA 2195391View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b1f311 - Browse repository at this point
Copy the full SHA 8b1f311View commit details -
feat: use split store in the view client (near#8656)
- use split store in the view client - when configured - removed the get_store(Temperature) method and replaced usages with safer get_cold_store and get_hot_store - added SplitStorageConfig and used it to configure a few things
Configuration menu - View commit details
-
Copy full SHA for b224412 - Browse repository at this point
Copy the full SHA b224412View commit details -
feat: Dump state of every epoch to S3 (near#8661)
* Start a thread per shard to do the dumping * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior. * No changes to the behavior of the node if those options are not enabled in `config.json`. * State is persisted to RocksDB such that restarts of the node are well handled. * Some useful metrics are exported. * The node assumes it's the only node in the this and all alternative universes that does the dumping. * * Unclear how to use multiple nodes to complete the dump faster * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags * * Do we even need tags?
Configuration menu - View commit details
-
Copy full SHA for 5846e1a - Browse repository at this point
Copy the full SHA 5846e1aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4cb5139 - Browse repository at this point
Copy the full SHA 4cb5139View commit details -
Configuration menu - View commit details
-
Copy full SHA for afcce1b - Browse repository at this point
Copy the full SHA afcce1bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8408ce7 - Browse repository at this point
Copy the full SHA 8408ce7View commit details -
Configuration menu - View commit details
-
Copy full SHA for c61ff2a - Browse repository at this point
Copy the full SHA c61ff2aView commit details -
Configuration menu - View commit details
-
Copy full SHA for b682424 - Browse repository at this point
Copy the full SHA b682424View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f43f8c - Browse repository at this point
Copy the full SHA 4f43f8cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 161a357 - Browse repository at this point
Copy the full SHA 161a357View commit details -
Configuration menu - View commit details
-
Copy full SHA for 020d7f9 - Browse repository at this point
Copy the full SHA 020d7f9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4910fcc - Browse repository at this point
Copy the full SHA 4910fccView commit details -
Configuration menu - View commit details
-
Copy full SHA for c2fa4a0 - Browse repository at this point
Copy the full SHA c2fa4a0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 14e202f - Browse repository at this point
Copy the full SHA 14e202fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0da5f3b - Browse repository at this point
Copy the full SHA 0da5f3bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 610e820 - Browse repository at this point
Copy the full SHA 610e820View commit details -
Configuration menu - View commit details
-
Copy full SHA for 296dda3 - Browse repository at this point
Copy the full SHA 296dda3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 11e965a - Browse repository at this point
Copy the full SHA 11e965aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 95c1ad5 - Browse repository at this point
Copy the full SHA 95c1ad5View commit details -
Configuration menu - View commit details
-
Copy full SHA for dc89915 - Browse repository at this point
Copy the full SHA dc89915View commit details -
Configuration menu - View commit details
-
Copy full SHA for f14e8aa - Browse repository at this point
Copy the full SHA f14e8aaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 15d1fbb - Browse repository at this point
Copy the full SHA 15d1fbbView commit details -
Configuration menu - View commit details
-
Copy full SHA for f390b19 - Browse repository at this point
Copy the full SHA f390b19View commit details -
Configuration menu - View commit details
-
Copy full SHA for 89134c5 - Browse repository at this point
Copy the full SHA 89134c5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5a1acff - Browse repository at this point
Copy the full SHA 5a1acffView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4a9aa86 - Browse repository at this point
Copy the full SHA 4a9aa86View commit details -
Configuration menu - View commit details
-
Copy full SHA for 56dc868 - Browse repository at this point
Copy the full SHA 56dc868View commit details -
Configuration menu - View commit details
-
Copy full SHA for 74773e0 - Browse repository at this point
Copy the full SHA 74773e0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 81d511f - Browse repository at this point
Copy the full SHA 81d511fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6db2638 - Browse repository at this point
Copy the full SHA 6db2638View commit details -
Configuration menu - View commit details
-
Copy full SHA for 51dbd55 - Browse repository at this point
Copy the full SHA 51dbd55View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6022323 - Browse repository at this point
Copy the full SHA 6022323View commit details -
Configuration menu - View commit details
-
Copy full SHA for cca8bdc - Browse repository at this point
Copy the full SHA cca8bdcView commit details -
Configuration menu - View commit details
-
Copy full SHA for b800bdc - Browse repository at this point
Copy the full SHA b800bdcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1cd3a72 - Browse repository at this point
Copy the full SHA 1cd3a72View commit details -
fix: apply
log_summary_style
to state sync phase message (near#8735)* Use colour for state sync phase display * fix * fix * debug verbosity * fix
Configuration menu - View commit details
-
Copy full SHA for b319c51 - Browse repository at this point
Copy the full SHA b319c51View commit details -
[Runtime Epoch Split] (3/n) Add ability to get Arc<EpochManagerAdapte…
…r> out of an &RuntimeWithEpochManagerAdapter. (near#8768)
Configuration menu - View commit details
-
Copy full SHA for 6b09e0d - Browse repository at this point
Copy the full SHA 6b09e0dView commit details -
feat: Disable state sync by default because it's unreliable (near#8730)
Add a config option `state_sync_enabled`. Default value is `false`. If the option is `true`, state sync works as usual, but prints this message every 10 seconds: ``` Mar 15 07:31:11 nikurt-4 neard[48582]: 2023-03-15T07:31:11.352458Z WARN stats: The node is syncing its State. The current implementation of this mechanism is known to be unreliable. It may never complete, or fail randomly and corrupt the DB. Mar 15 07:31:11 nikurt-4 neard[48582]: Suggestions: Mar 15 07:31:11 nikurt-4 neard[48582]: * Download a recent data snapshot and restart the node. Mar 15 07:31:11 nikurt-4 neard[48582]: * Disable state sync in the config. Add `"state_sync_enabled": false` to `config.json`. Mar 15 07:31:11 nikurt-4 neard[48582]: A better implementation of State Sync is work in progress. ``` If the option is `false`, the node proceeds to download and apply blocks. Started a node from a month-old snapshot. After about 24 hours of running I see this, which confirms that the Block Sync is enabled and State Sync wasn't enabled: ``` Mar 15 07:32:27 nikurt-3 neard[36289]: 2023-03-15T07:32:27.565700Z INFO stats: #118440788 Downloading blocks 4.70% (1951625 left; at 118440788) 23 peers ⬇ 638 kB/s ⬆ 86.9 kB/s 2.00 bps 70.7 Tgas/s CPU: 62%, Mem: 4.27 GB ``` Fix near#8719
Configuration menu - View commit details
-
Copy full SHA for fdaa79e - Browse repository at this point
Copy the full SHA fdaa79eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 89297d4 - Browse repository at this point
Copy the full SHA 89297d4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 699954f - Browse repository at this point
Copy the full SHA 699954fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1244ac3 - Browse repository at this point
Copy the full SHA 1244ac3View commit details -
Configuration menu - View commit details
-
Copy full SHA for ad7ded8 - Browse repository at this point
Copy the full SHA ad7ded8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6691bf7 - Browse repository at this point
Copy the full SHA 6691bf7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1cc6f05 - Browse repository at this point
Copy the full SHA 1cc6f05View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ad29cd - Browse repository at this point
Copy the full SHA 0ad29cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 438e343 - Browse repository at this point
Copy the full SHA 438e343View commit details -
chore: update changelog with 1.32.1 and 1.32.2 (near#8869)
Not entirely clear if we want minor releases tracked in changelog, do we? If so here is the PR for 1.32.1 and 1.32.2.
Configuration menu - View commit details
-
Copy full SHA for ea0b82a - Browse repository at this point
Copy the full SHA ea0b82aView commit details -
feat: override NetworkConfig from JSON config (near#8871)
Added config.experimental.network_config_overrides field. It contains the overrides for the currently default values from NetworkConfig. The JSON config override is done before the CLI overrides.
Configuration menu - View commit details
-
Copy full SHA for e3baa8e - Browse repository at this point
Copy the full SHA e3baa8eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5a0a4fb - Browse repository at this point
Copy the full SHA 5a0a4fbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 526225d - Browse repository at this point
Copy the full SHA 526225dView commit details -
Configuration menu - View commit details
-
Copy full SHA for f5d9545 - Browse repository at this point
Copy the full SHA f5d9545View commit details -
Configuration menu - View commit details
-
Copy full SHA for cc49285 - Browse repository at this point
Copy the full SHA cc49285View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8067add - Browse repository at this point
Copy the full SHA 8067addView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4c53eac - Browse repository at this point
Copy the full SHA 4c53eacView commit details -
Configuration menu - View commit details
-
Copy full SHA for cd34b4a - Browse repository at this point
Copy the full SHA cd34b4aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8f2c9b4 - Browse repository at this point
Copy the full SHA 8f2c9b4View commit details -
Configuration menu - View commit details
-
Copy full SHA for faa8a10 - Browse repository at this point
Copy the full SHA faa8a10View commit details -
Configuration menu - View commit details
-
Copy full SHA for cfce87c - Browse repository at this point
Copy the full SHA cfce87cView commit details -
Configuration menu - View commit details
-
Copy full SHA for e71a0ea - Browse repository at this point
Copy the full SHA e71a0eaView commit details -
Configuration menu - View commit details
-
Copy full SHA for b01050f - Browse repository at this point
Copy the full SHA b01050fView commit details -
Configuration menu - View commit details
-
Copy full SHA for b61749a - Browse repository at this point
Copy the full SHA b61749aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 594cafa - Browse repository at this point
Copy the full SHA 594cafaView commit details -
Configuration menu - View commit details
-
Copy full SHA for a9debc5 - Browse repository at this point
Copy the full SHA a9debc5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 00f70d4 - Browse repository at this point
Copy the full SHA 00f70d4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0855002 - Browse repository at this point
Copy the full SHA 0855002View commit details -
Configuration menu - View commit details
-
Copy full SHA for b09200c - Browse repository at this point
Copy the full SHA b09200cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 27423e5 - Browse repository at this point
Copy the full SHA 27423e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2a7454f - Browse repository at this point
Copy the full SHA 2a7454fView commit details
Commits on Apr 14, 2023
-
Configuration menu - View commit details
-
Copy full SHA for b29d2b3 - Browse repository at this point
Copy the full SHA b29d2b3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6500c42 - Browse repository at this point
Copy the full SHA 6500c42View commit details