This repository has been archived by the owner on Apr 3, 2024. It is now read-only.
betanet #13
Merged
betanet #13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a few tests which test that `SnapshotHostInfo` is properly broadcast between peers. `SnapshotHostInfo` lets other peers know that a certain peer has a state snapshot at a specific point in time, which is later used for state sync. Nodes publish this information periodically and it should be broadcast to all nodes in the network. Fixes: #10172
Fuzzer building runs on rustc nightly, which can have a different set of warnings than rustc stable. Rather than introduce a new compilation step for all the PRs, just compile fuzzers without caring about warnings. Fixes a recent fuzzer building issue, probably caused by a nightly upgrade: https://github.com/near/nearcore/actions/runs/6929988512/job/18848791016
…ons (#10256) Functions `find_chunk_producer_for_forwarding()`, `find_validator_for_forwarding()` are not used much, one calls another, and they both call `chain.head()`. That seems unnecessary. Refactored to call `get_chunk_producer()` directly, and not worry about 3 layers of functions using different types of heads. See near/nearcore#10251 for context
We have now fully migrated off buildkite, so let’s remove the old pipeline from the repository :)
Community reports that sometimes it takes minutes for a transaction to be recorded on chain. My guess is that it takes that much time to record a transaction on chain, because it gets forwarded to nodes which either: * fail to produce their chunks * don't have time to process incoming tx RPCs before producing their chunks * the node producing a tx is out of sync and the set of validators is already too late All of these can be fixed by multicasting transactions further into the future. This PR is the simplest attempt to send transactions further into the future. Tested: Had a testnet node and submitted a tx that was supposed to be forwarded to chunk producers of the next 100 blocks. In practice it gets forwarded to 5 validators. Sending a message to a validator takes about 100µs for a trivial tx.
Building on top of near/nearcore#10249, further reduces use of `num_shards()` in chain.rs. Also removes some "premature" optimisations that I don't think would necessarily help in production and just add complexity to the code.
* Simplifies `state_sync_enabled` and `enable_multiline_logging`.
* Adjusts {min,max}_block_production_delay config options when
generating configs for testnet and mainnet.
* Simplifies config initialization logic.
* Deletes betanet and stakewars
Fixes https://rustsec.org/advisories/RUSTSEC-2023-0072, reported by the cargo-audit CI job
Adds support for partial mandates in chunk validator assignment. Currently only full mandates are considered. Assume `stake_per_mandate = 5`, then: - Valdiator `V1` with stake 12 holds 2 full mandates corresponding to a stake of 10. The remaining stake of 2 does not participate in validation. - Validator `V2` with stake 4 holds 0 full mandates and does not participate in validation at all. With support for partial mandates `V1` gets a partial mandate with weight 2 and `V2` gets a partial mandate of 4. So partial mandates allow to increase the number of validators and allows them to commit their entire stake to validation. ### Probability of shard corruption In [these simulations](https://near.zulipchat.com/#narrow/stream/407237-pagoda.2Fcore.2Fstateless-validation/topic/chunk.20validator.20assignment.20simulation.20with.20partial.20seats), the probability of shard corruption decreases as partial seats are included. In general, the effect of including partial seats depends on the distribution of (malicious) stake across validators and the stake required per mandate. ### `ValidatorMandatesAssignment` This type is introduced to more clearly document the result of sampling chunk validators and to avoid passing around a raw `Vec<HashMap<ValidatorId, _>>`. ### Shuffling of shard ids [This thread](mooori/sim-validator-assignment#12 (comment)) points out a bias towards small shard ids and brings up shard id shuffling as a way to fix it. For now shard ids are not shuffled in this PR, since I think it would make the diff harder to review. I would suggest to introduce shard id shuffling in a separate PR. This should not pose a risk as chunk validator assignment is not enabled in production.
<p>This PR was automatically created by Snyk using the credentials of a
real user.</p><br /><h3>Snyk has created this PR to upgrade
@types/react-dom from 18.2.14 to 18.2.15.</h3>
:information_source: Keep your dependencies up-to-date. This makes it
easier to fix existing vulnerabilities and to more quickly identify and
fix newly disclosed vulnerabilities when they affect your project.
<hr/>
- The recommended version is **1 version** ahead of your current
version.
- The recommended version was released **22 days ago**, on 2023-11-07.
<details>
<summary><b>Release notes</b></summary>
<br/>
<details>
<summary>Package name: <b>@types/react-dom</b></summary>
<ul>
<li>
<b>18.2.15</b> - 2023-11-07
</li>
<li>
<b>18.2.14</b> - 2023-10-18
</li>
</ul>
from <a
href="https://snyk.io/redirect/github/DefinitelyTyped/DefinitelyTyped/releases">@types/react-dom
GitHub release notes</a>
</details>
</details>
<hr/>
**Note:** *You are seeing this because you or someone else with access
to this repository has authorized Snyk to open upgrade PRs.*
For more information: <img
src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiI5MDM2MzdmYS0yNGU2LTRiZjktYjY1NC1hYzYzNzFiMDY5MDEiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjkwMzYzN2ZhLTI0ZTYtNGJmOS1iNjU0LWFjNjM3MWIwNjkwMSJ9fQ=="
width="0" height="0"/>
🧐 [View latest project
report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr)
🛠 [Adjust upgrade PR
settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr)
🔕 [Ignore this dependency or unsubscribe from future upgrade
PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=@types/react-dom&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)
<!---
(snyk:metadata:{"prId":"903637fa-24e6-4bf9-b654-ac6371b06901","prPublicId":"903637fa-24e6-4bf9-b654-ac6371b06901","dependencies":[{"name":"@types/react-dom","from":"18.2.14","to":"18.2.15"}],"packageManager":"npm","type":"auto","projectUrl":"https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr","projectPublicId":"98480bdc-d80b-4fd1-89d7-c4c56a706763","env":"prod","prType":"upgrade","vulns":[],"issuesToFix":[],"upgrade":[],"upgradeInfo":{"versionsDiff":1,"publishedDate":"2023-11-07T20:33:53.244Z"},"templateVariants":[],"hasFixes":false,"isMajorUpgrade":false,"isBreakingChange":false,"priorityScoreList":[]})
--->
Co-authored-by: snyk-bot <snyk-bot@snyk.io>
… (#10268)
The following code is currently used for comparing `boundary_account`
and `account_id`:
```rust
if boundary_account.cmp(account_id) == Greater {
...
}
```
IMO it's a bit confusing - it isn't immediately obvious whether this
means `boundary_account` > `account_id` or `account_id` >
`boundary_account`. I misread this line and because of that I made a
mistake in #10240.
Let's change it to something that is easier to read:
```rust
if account_id < boundary_account {
...
}
```
Add a command which allows to analyse gas usage on some part of the
blockchain.
The command goes over the specified blocks and gathers information about
how much
gas was used by each block, shard and account.
Then it displays an analysis of all the data.
The analysis contains:
* Total gas used
* Gas used on each shard
* Top 10 accounts by gas usage
* The optimal account by which a shard could be split into two halves
with similar gas usage
Help page:
```bash
$ ./neard database analyse-gas-usage --help
Analyse gas usage in a chosen sequnce of blocks
Usage: neard database analyse-gas-usage [OPTIONS]
Options:
--last-blocks <LAST_BLOCKS>
Analyse the last N blocks in the blockchain
--from-block-height <FROM_BLOCK_HEIGHT>
Analyse blocks from the given block height, inclusive
--to-block-height <TO_BLOCK_HEIGHT>
Analyse blocks up to the given block height, inclusive
-h, --help
Print help
```
Example usage:
```bash
# Analyse gas usage in the last 1000 blocks
cargo run --bin neard -- database analyse-gas-usage --last-blocks 1000
# Analyse gas usage in blocks with heights between 200 and 300 (inclusive)
./neard database analyse-gas-usage --from-block-height 200 --to-block-height 300
```
…s()` (#10271) Closes near/nearcore#10237
A pretty heavy handed rewrite of the resharding documentation. I'm open to some heavy handed suggestions ;) I'm not sure what's the best trade off between duplicating some content from the NEPs here and keeping in minimal with just links. There is some new content here, mainly the rollout section, I would love to hear some thoughts about this in particular from @gmilescu and @posvyatokum.
This introduces a hack in our fork of cargo-bolero, needed for proper fuzzer building. The reason is still the same as the patch previously applied: the current bolero way of auto-detecting fuzzers is suboptimal. All these patches should be able to go away with some work on cargo-bolero, but we have other priorities, and this hack seems to make the tarball builds work locally at least. With this, we run with two patches: - Ekleog-NEAR/bolero@877f199, which is a bit hacky but might deserve being included upstream were it not for backward compatibility reasons - Ekleog-NEAR/bolero@56da8e6, which is a real, bad hack, but seems to be enough to stop having to care about it for now, until we can go back to it and fix the underlying issue that fuzzer listing is currently not well-implemented. Its consequence is, any fuzzer in the `neard` crate will not actually run on our fuzzing infrastructure. We’ll have to fix it at some point.
This PR moves NightshadeRuntime tests module into a separate file without changing anything else. See near/nearcore#10275 for more context. In case having tests as a separate file is a good idea for the following reasons: - these are mostly integration-style tests with heavy setup (see `TestEnv`) and quite a lot of code, so to me it seems like it deserves to be in a separate file - the code is already in `mod.rs` file, so we don't need to create a separate directory structure.
…dation. (#10257) We discussed as a team earlier that we don't need these feature flags. Removing it to simplify further development of chunk validation.
<p>This PR was automatically created by Snyk using the credentials of a
real user.</p><br /><h3>Snyk has created this PR to upgrade @types/react
from 18.2.25 to 18.2.37.</h3>
:information_source: Keep your dependencies up-to-date. This makes it
easier to fix existing vulnerabilities and to more quickly identify and
fix newly disclosed vulnerabilities when they affect your project.
<hr/>
- The recommended version is **12 versions** ahead of your current
version.
- The recommended version was released **25 days ago**, on 2023-11-07.
<details>
<summary><b>Release notes</b></summary>
<br/>
<details>
<summary>Package name: <b>@types/react</b></summary>
<ul>
<li>
<b>18.2.37</b> - 2023-11-07
</li>
<li>
<b>18.2.36</b> - 2023-11-06
</li>
<li>
<b>18.2.35</b> - 2023-11-05
</li>
<li>
<b>18.2.34</b> - 2023-11-01
</li>
<li>
<b>18.2.33</b> - 2023-10-26
</li>
<li>
<b>18.2.32</b> - 2023-10-25
</li>
<li>
<b>18.2.31</b> - 2023-10-20
</li>
<li>
<b>18.2.30</b> - 2023-10-19
</li>
<li>
<b>18.2.29</b> - 2023-10-18
</li>
<li>
<b>18.2.28</b> - 2023-10-10
</li>
<li>
<b>18.2.27</b> - 2023-10-09
</li>
<li>
<b>18.2.26</b> - 2023-10-09
</li>
<li>
<b>18.2.25</b> - 2023-10-04
</li>
</ul>
from <a
href="https://snyk.io/redirect/github/DefinitelyTyped/DefinitelyTyped/releases">@types/react
GitHub release notes</a>
</details>
</details>
<hr/>
**Note:** *You are seeing this because you or someone else with access
to this repository has authorized Snyk to open upgrade PRs.*
For more information: <img
src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiJkYjA5OTM4YS0zZGY5LTRkMTYtYTM5MS00MDkyMzVkNTllMDUiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6ImRiMDk5MzhhLTNkZjktNGQxNi1hMzkxLTQwOTIzNWQ1OWUwNSJ9fQ=="
width="0" height="0"/>
🧐 [View latest project
report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr)
🛠 [Adjust upgrade PR
settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?utm_source=github&utm_medium=referral&page=upgrade-pr)
🔕 [Ignore this dependency or unsubscribe from future upgrade
PRs](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763/settings/integration?pkg=@types/react&utm_source=github&utm_medium=referral&page=upgrade-pr#auto-dep-upgrades)
<!---
(snyk:metadata:{"prId":"db09938a-3df9-4d16-a391-409235d59e05","prPublicId":"db09938a-3df9-4d16-a391-409235d59e05","dependencies":[{"name":"@types/react","from":"18.2.25","to":"18.2.37"}],"packageManager":"npm","type":"auto","projectUrl":"https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=upgrade-pr","projectPublicId":"98480bdc-d80b-4fd1-89d7-c4c56a706763","env":"prod","prType":"upgrade","vulns":[],"issuesToFix":[],"upgrade":[],"upgradeInfo":{"versionsDiff":12,"publishedDate":"2023-11-07T20:31:32.792Z"},"templateVariants":[],"hasFixes":false,"isMajorUpgrade":false,"isBreakingChange":false,"priorityScoreList":[]})
--->
Co-authored-by: snyk-bot <snyk-bot@snyk.io>
<p>This PR was automatically created by Snyk using the credentials of a
real user.</p><br /><h3>Snyk has created this PR to fix one or more
vulnerable packages in the `npm` dependencies of this project.</h3>
#### Changes included in this PR
- Changes to the following files to upgrade the vulnerable dependencies
to a fixed version:
- tools/debug-ui/package.json
- tools/debug-ui/package-lock.json
#### Vulnerabilities that will be fixed
##### With an upgrade:
Severity | Priority Score (*) | Issue | Breaking Change | Exploit
Maturity
:-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------
 | **661/1000** <br/> **Why?** Recently disclosed, Has a
fix available, CVSS 7.5 | Missing Release of Resource after Effective
Lifetime
<br/>[SNYK-JS-INFLIGHT-6095116](https://snyk.io/vuln/SNYK-JS-INFLIGHT-6095116)
| Yes | No Known Exploit
(*) Note that the real score may have changed since the PR was raised.
<details>
<summary><b>Commit messages</b></summary>
</br>
<details>
<summary>Package name: <b>react-query</b></summary>
The new version differs by 1 commits.</br>
<ul>
<li><a
href="https://snyk.io/redirect/github/TanStack/query/commit/357ec041a6fcc4a550f3df02c12ecc7bcdefbc05">357ec04</a>
v4 release (#3842)</li>
</ul>
<a
href="https://snyk.io/redirect/github/TanStack/query/compare/ef684205cb4890db904db5b387513fb9042e0bb6...357ec041a6fcc4a550f3df02c12ecc7bcdefbc05">See
the full diff</a>
</details>
</details>
Check the changes in this PR to ensure they won't cause issues with your
project.
------------
**Note:** *You are seeing this because you or someone else with access
to this repository has authorized Snyk to open fix PRs.*
For more information: <img
src="https://api.segment.io/v1/pixel/track?data=eyJ3cml0ZUtleSI6InJyWmxZcEdHY2RyTHZsb0lYd0dUcVg4WkFRTnNCOUEwIiwiYW5vbnltb3VzSWQiOiIyMzRmMTE2Yi0wYWFjLTRkOTMtYWRlZi0zMjNhMDgyMTQ2MzAiLCJldmVudCI6IlBSIHZpZXdlZCIsInByb3BlcnRpZXMiOnsicHJJZCI6IjIzNGYxMTZiLTBhYWMtNGQ5My1hZGVmLTMyM2EwODIxNDYzMCJ9fQ=="
width="0" height="0"/>
🧐 [View latest project
report](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=fix-pr)
🛠 [Adjust project
settings](https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=fix-pr/settings)
📚 [Read more about Snyk's upgrade and patch
logic](https://support.snyk.io/hc/en-us/articles/360003891078-Snyk-patches-to-fix-vulnerabilities)
[//]: #
(snyk:metadata:{"prId":"234f116b-0aac-4d93-adef-323a08214630","prPublicId":"234f116b-0aac-4d93-adef-323a08214630","dependencies":[{"name":"react-query","from":"3.39.3","to":"4.0.0"}],"packageManager":"npm","projectPublicId":"98480bdc-d80b-4fd1-89d7-c4c56a706763","projectUrl":"https://app.snyk.io/org/ecp88/project/98480bdc-d80b-4fd1-89d7-c4c56a706763?utm_source=github&utm_medium=referral&page=fix-pr","type":"auto","patch":[],"vulns":["SNYK-JS-INFLIGHT-6095116"],"upgrade":["SNYK-JS-INFLIGHT-6095116"],"isBreakingChange":true,"env":"prod","prType":"fix","templateVariants":["updated-fix-title","priorityScore"],"priorityScoreList":[661],"remediationStrategy":"vuln"})
---
**Learn how to fix vulnerabilities with free interactive lessons:**
🦉 [Learn about vulnerability in an interactive lesson of Snyk
Learn.](https://learn.snyk.io/?loc=fix-pr)
Co-authored-by: snyk-bot <snyk-bot@snyk.io>
…debug-ui (#10272) Bumps [@adobe/css-tools](https://github.com/adobe/css-tools) from 4.3.1 to 4.3.2. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/adobe/css-tools/blob/main/History.md"><code>@adobe/css-tools</code>'s changelog</a>.</em></p> <blockquote> <h1>4.3.2 / 2023-11-28</h1> <ul> <li>Fix redos vulnerability with specific crafted css string - CVE-2023-48631</li> <li>Fix Problem parsing with :is() and nested :nth-child() <a href="https://github.com/adobe/css-tools/issues/211">#211</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/adobe/css-tools/commits">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/near/nearcore/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Added some metrics from tests in mainnet and testnet. ref data mainnet https://nearinc.grafana.net/goto/aFBwfLHSg?orgId=1 ref data testnet https://nearinc.grafana.net/goto/XgBUBYNSg?orgId=1
Made no functional changes except: * Replaced a tuple with a struct * Introduced a function `made_enough_progress()` to make the code much simpler to understand. * Renamed a ban reason
Reported by @wacban. Debug printing results in multiline strings, which are hard to interpret. It also breaks tooling one might want to use to analyze logs. Example: ``` 2023-12-01T10:22:08.186809Z INFO network: failed to connect to ed25519:AF4j@91.226.87.212:54063 result=Err(tcp::Stream::connect() Caused by: deadline has elapsed) ``` I believe this is coming from the default `Debug` impl for `anyhow::Error`. Using `Display` impl should fix it as it concatenates the context via `:` instead. --------- Co-authored-by: nikurt <86772482+nikurt@users.noreply.github.com>
`SnapshotHostInfo` contains a list of shard ids for which a peer has a snapshot. Under normal circumstances this list is pretty small - the maximum size would be the total number of shards. But a malicious peer could craft a `SnapshotHostInfo` message with very large number of shards - millions of shard ids. This could cause problems on the receiver node, so there has to be limit on the number of shard ids in a message. Let's limit the number of shard ids in a single message to `MAX_SHARDS_PER_SNAPSHOT_HOST_INFO = 512`. It's a constant that describes how many shards a single peer can have snapshots for. A peer doesn't keep more state snapshots than the number of tracked shards, so this limit is reasonable. 512 shards ought to be enough for anyone. In an ideal world we could check the current number of shards and reject messages that contain more shard ids than the current number, but sadly this can't be implemented. The problem is that the receiving node might be behind the rest of the blockchain, and the latest information just isn't available, so it can't check what the current number of shards is. We could reject messages in such situations, but this would lead to loss of information. Limiting the number of shard ids to a constant number is an okay alternative.
make the resharding config dynamic
Merge pull request #8 from utnet-org/james-vrf-validators
init rpc for miner_chips_list
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
betanet init