fix: increase trie cache size #5706
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Longarithm
added
the
A-transaction-runtime
pmnoxx
reviewed
Dec 8, 2021
| @@ -119,7 +119,7 @@ impl TrieStorage for TrieMemoryPartialStorage { | |||
|
|
|||
| /// Maximum number of cache entries. | |||
| #[cfg(not(feature = "no_cache"))] | |||
| const TRIE_MAX_CACHE_SIZE: usize = 10000; | |||
There was a problem hiding this comment.
Yes, I think Self(Arc::new(Mutex::new(SizedCache::with_size(TRIE_MAX_CACHE_SIZE)))), could be the problem.
Replacing it with SyncLruCache could help a little bit #5632
Once, this gets approved, I'll work on even better version, as there are a few other issues with it, that could cause big issues.
It is theoretically, possible, that the current implementation of TrieCache causes the issues.
I have a plan to fix it, but I need to wait for #5632 to be approved first.
To improve:
- use SyncLruCache, that uses a better implemented
LruCacheinstead ofcached. - Instead of using single
Mutex, use a shared mutex, so we have for example, 16 smaller caches, and the chances of hitting a mutex that's busy, are much lower. Latency, spikes can happen, if potentially, a thread is swapped out, while hitting a mutex, etc. - There, are a few other issues with the cache, though we need to wait for review first.
There was a problem hiding this comment.
Anyway, that's a bit out of topic, not blocking this PR. I think increase cache size is an excellent idea!
bowenwang1996
approved these changes
Dec 8, 2021
pmnoxx
approved these changes
Dec 8, 2021
mm-near
reviewed
Dec 8, 2021
mzhangmzz
approved these changes
Dec 8, 2021
bowenwang1996
pushed a commit
that referenced
this pull request
Dec 20, 2021
Increase trie cache size to mitigate receipts undercharging issue. ## Validity We can increase cache size to 50_000, so the theoretical occupied size is 50_000 * 4 (num_shards) * 4_000 = 800 MB. In reality, `/usr/bin/time -v ` shows **520 -> 1070 MB** RAM growth which seem to be caused by #5212. This change have an impact like 30 MB. I've additionally checked that cache is full on my runs. ## Speedup On the sample of 30 receipts we considered, we have the following characteristics: ``` default settings: summary time = 3.3s, avg undercharging = 101 increase state cache size: summary time = 0.67s, avg undercharging = 21 increase state cache size + trie cache size = 50k: summary time = 0.5s, avg undercharging = 11.5 increase state cache size + trie cache size = 100k + reduce max value size to 1k: summary time = 0.3s, avg undercharging = 6.9 ``` For now, let's just increase cache size. On this sample we reduce undercharging 2x and summary execution time 1.3x. ## Testing * checking that cache is full; * existing tests, * estimating `/usr/bin/time -v ./state-viewer apply_range --start_index 54051433 --end_index 54053438 --shard_id 3`
bowenwang1996
added a commit
that referenced
this pull request
Jan 3, 2022
This reverts commit f41190e.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Increase trie cache size to mitigate receipts undercharging issue.
Validity
We can increase cache size to 50_000, so the theoretical occupied size is 50_000 * 4 (num_shards) * 4_000 = 800 MB.
In reality,
/usr/bin/time -vshows 520 -> 1070 MB RAM growth which seem to be caused by #5212. This change have an impact like 30 MB.I've additionally checked that cache is full on my runs.
Speedup
On the sample of 30 receipts we considered, we have the following characteristics:
For now, let's just increase cache size. On this sample we reduce undercharging 2x and summary execution time 1.3x.
Testing
/usr/bin/time -v ./state-viewer apply_range --start_index 54051433 --end_index 54053438 --shard_id 3