Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unified scheduler no leak #19

Open
wants to merge 844 commits into
base: unified-scheduler-fallible-new-task
Choose a base branch
from

Conversation

ryoqun
Copy link
Owner

@ryoqun ryoqun commented Apr 29, 2024

Problem

Summary of Changes

Fixes #

@ryoqun ryoqun force-pushed the unified-scheduler-no-leak branch 2 times, most recently from 615bfc7 to a05124c Compare May 1, 2024 06:31
@ryoqun ryoqun force-pushed the unified-scheduler-fallible-new-task branch 7 times, most recently from 89461f5 to 27922d5 Compare May 2, 2024 14:19
apfitzge and others added 21 commits May 12, 2024 10:19
…and SubscriptionToken (solana-labs#1246)

* clippy: allow dead_code for SubscriptionToken

* clippy: allow dead_code for GraphVoteAccountModeError

* clippy: allow dead_code for DroppableTask

* remove the string in the GraphVoteAccountModeError

* abort the joinhandle when drop

* only trace JoinHandle in the drop

* add comment
* ci: source based coverage

* point to file explorer

* fix sanity

* break line

* pass @ arg from ci/coverage to script/coverage

* move source script to the top

* direct to the summary page

* explain why nightly

* don't rug orignal rustflag

* rearrange args order

* bring back the trace level logging

* remove _

* fix log level

* add reference to envs

* put cov result into target/cov

* move output to target/cov

* XXX hack buildkite pipeline

* XXX: test for no log

* Revert "XXX hack buildkite pipeline"

This reverts commit effb72e.

* Revert "XXX: test for no log"

This reverts commit 975482d.

* don't use trace log level temporarily

* only search coverage report in the main folder
…reds (solana-labs#1271)

With chained Merkle shreds which are signed by the retransmitter,
different retransmitter signatures does not make a shred duplicate.
…s#1308)

We were accidentally doing several thousands 4MB allocations - even
during incremental hash - which added up to a 8G+ memory spikes over ~2s
every ~30s.

Fix by using Vec::new() in the identity function. Empirically 98%+
reduces join arrays with less than 128 elements, and only the last few
reduces join large vecs. Because realloc does exponential growth we
don't see pathological reallocation but reduces do at most one realloc
(and often 0 because of exp growth).
program-runtime: modify sysvar cache impl

in prep for sol_sysvar_get, rework how SysvarCache stores data internally
we store account data directly, except also storing StakeHistory and SlotHashes as objects
these object representations can be removed after native programs are converted to bpf
…bs#1277)

* Change type in StartBlockHeightAndRewards to PartitionedStakeRewards and hoist conversion into calculation

* Make PartitionedStakeReward pub to match EpochRewardStatus

* Return PartitionedStakeRewards from calculate_stake_vote_rewards and update types accordingly

* Fixup tests to use PartitionedStakeReward

* Simplify Stake asserts/unwraps
…#1286)

* don't activate the hmac feature of libsecp256k1 in the workspace, but keep the other default features

* activate the hmac feature of libsecp256k1 in solana-sdk
…-dalek (solana-labs#1327)

* remove unnecessary activation of serde feature of curve25519-dalek

* zk-sdk actually does need serde in curve25519-dalek
* remove unused syn features

* add back extra-traits feature

* add back full feature to syn under sdk-macro
brooksprumo and others added 30 commits June 5, 2024 14:32
…#1608)

* Add failing test

* sdk: don't assume input key ordering matches tx when replacing signatures

* Refactor iterators

---------

Co-authored-by: Trent Nelson <trent@solana.com>
* make borsh optional in sdk and program

* fmt

* don't include borsh in dev-context-only-utils
* build(deps): bump async-trait from 0.1.77 to 0.1.80

Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.77 to 0.1.80.
- [Release notes](https://github.com/dtolnay/async-trait/releases)
- [Commits](dtolnay/async-trait@0.1.77...0.1.80)

---
updated-dependencies:
- dependency-name: async-trait
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update all Cargo files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump rustversion from 1.0.14 to 1.0.17

Bumps [rustversion](https://github.com/dtolnay/rustversion) from 1.0.14 to 1.0.17.
- [Release notes](https://github.com/dtolnay/rustversion/releases)
- [Commits](dtolnay/rustversion@1.0.14...1.0.17)

---
updated-dependencies:
- dependency-name: rustversion
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update all Cargo files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [parking_lot](https://github.com/Amanieu/parking_lot) from 0.12.2 to 0.12.3.
- [Changelog](https://github.com/Amanieu/parking_lot/blob/master/CHANGELOG.md)
- [Commits](Amanieu/parking_lot@0.12.2...0.12.3)

---
updated-dependencies:
- dependency-name: parking_lot
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* build(deps): bump regex from 1.10.3 to 1.10.4

Bumps [regex](https://github.com/rust-lang/regex) from 1.10.3 to 1.10.4.
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](rust-lang/regex@1.10.3...1.10.4)

---
updated-dependencies:
- dependency-name: regex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update all Cargo files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: HaoranYi <haoran.yi@solana.com>
* program-test: add program account helpers

* program-test: add test for Core BPF programs

* program-test: add BPF program fixture

* wrap into `add_upgradeable_program_to_genesis`

* wrap ELF loading into `add_upgradeable_program_to_genesis`

* cleanup
vote: hoist vote sender types up to runtime
…solana-labs#1549)

* Plumb new SplTokenAdditionalData everywhere

* account-decoder: Calculate ui amount with interest

* rpc / ledger: Populate interest bearing information

* rpc: Test interest-bearing config

* Deprecate `parse_token` for `parse_token_v2`

* Deprecate parse_account_data and AccountAdditionalData for v2

* Deprecate token_amount_to_ui_amount -> v2

* Make get_mint_owner_and_additional_data pub(crate)

* ledger: Revert changes, always use raw amount
Requirements are a bit date, add some recent recs
* build(deps): bump rustls from 0.21.11 to 0.21.12

Bumps [rustls](https://github.com/rustls/rustls) from 0.21.11 to 0.21.12.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](rustls/rustls@v/0.21.11...v/0.21.12)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update all Cargo files

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [syn](https://github.com/dtolnay/syn) from 2.0.58 to 2.0.66.
- [Release notes](https://github.com/dtolnay/syn/releases)
- [Commits](dtolnay/syn@2.0.58...2.0.66)

---
updated-dependencies:
- dependency-name: syn
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…itted transactions (solana-labs#1547)

* wire up scheduler to adjust actual loaded accounts size cost for committed transactions

* update tests
* SVM: add `execute_timings` to API return type

* SVM: message processor: rename `timings` to `execute_timings`

* SVM: remove `timings` mutable input
…s#1661)

* Retire stable-rust clippy for buggy redundant_clone

* Remove unused scripts/cargo-clippy-stable.sh

* Remove clippy-stable from .mergify.yml
* Clean idle pooled schedulers periodically

* Fix race in failure_{with,without}_extra_tx

* Improve test

* Use extract_if polyfill

* Write justfication of extract_if with minor opt.
…bs#1574)

* Gracefully abort scheduler on a panic in handlers

* Move test-related mention out of the tested code itself
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.