Remove removed past PathHashes #7080
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
asdacap
reviewed
May 27, 2024
| @@ -104,11 +104,15 @@ public Task Execute(CancellationToken cancellationToken) | |||
| if (_logger.IsWarn) _logger.Warn($"Detected {pruningConfig.CacheMb}MB of pruning cache config. Pruning cache more than 2000MB is not recommended as it may cause long memory pruning time which affect attestation."); | |||
| } | |||
|
|
|||
| // 8 bytes Node pointer, (LinkedListNode 8 pointer + 8 pointer) + (HashAndTinyPath: 32 bytes for value + 8 bytes for path) + 32 bytes ValueHash256 | |||
| int KeyEntryMemorySize = 8 + (8 + 8) + (32 + 8) + 32; // 96 bytes | |||
There was a problem hiding this comment.
Can you double the default config also.
There was a problem hiding this comment.
Does it need 2.1M hashes?
There was a problem hiding this comment.
GC time is a function of number of objects (rather than size) I can redesign an Lru so it uses a giant array and indexes rather than pointers, if you want to go this high
There was a problem hiding this comment.
Yes, I've specifically, tune this amount.
There was a problem hiding this comment.
PR to eradicate the objects #7100
asdacap
reviewed
May 27, 2024
| { | ||
| TreePath fullPath = key.path.ToTreePath(); // Micro op to reduce double convert | ||
| if (CanRemove(key.addr, key.path, fullPath, prevHash, keyValuePair.Value)) | ||
| { | ||
| Metrics.RemovedNodeCount++; | ||
| writeBatch.Remove(key.addr, fullPath, prevHash); | ||
| _pastPathHash.Delete(key); |
There was a problem hiding this comment.
Will this make mem pruning slower, (since it does more work), or faster? (since it become smaller).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Otherwise after a couple prunes the cache quickly grows to the maximum 2.1M items; some of which will never be used again.
Types of changes
What types of changes does your code introduce?
Testing
Requires testing
If yes, did you write tests?
Notes on testing
Optional. Remove if not applicable.
Documentation
Requires documentation update
If yes, link the PR to the docs update or the issue with the details labeled
docs. Remove if not applicable.Requires explanation in Release Notes
If yes, fill in the details here. Remove if not applicable.
Remarks
Optional. Remove if not applicable.