feat: implement inlining for small values in flat storage #8988
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pugachAG
force-pushed
the
inline-flat-storage-values
branch
6 times, most recently
from
2a10dc8 to
c984bea
Compare
pugachAG
force-pushed
the
inline-flat-storage-values
branch
2 times, most recently
from
4ae35ac to
0a137ca
Compare
Longarithm
approved these changes
May 2, 2023
| #[derive(BorshSerialize, BorshDeserialize, Debug, Clone, PartialEq, Eq)] | ||
| pub enum FlatStateValue { | ||
| Ref(ValueRef), | ||
| // TODO(8243): add variant here for the inlined value | ||
| Inlined(Vec<u8>), |
There was a problem hiding this comment.
I have a little doubt if we should introduce it as a term in codebase. But it makes sense because our default behaviour is "not-inlined".
pugachAG
force-pushed
the
inline-flat-storage-values
branch
4 times, most recently
from
4111011 to
9629ed1
Compare
pugachAG
force-pushed
the
inline-flat-storage-values
branch
from
9629ed1 to
f3707c9
Compare
pugachAG
force-pushed
the
inline-flat-storage-values
branch
from
f3707c9 to
103ae59
Compare
pugachAG
added a commit
to pugachAG/nearcore
that referenced
this pull request
May 4, 2023
near-bulldozer bot
pushed a commit
that referenced
this pull request
May 4, 2023
…9005) This reverts #8988 as it results in node crashing: [zulip](https://near.zulipchat.com/#narrow/stream/345766-pagoda.2Fstorage.2Fflat-storage/topic/Canary.20crashes/near/355722627). The original PR changes data format for delta changes on disk, so binary with the new code failed to deserialise deltas written by the old code. I suggest we revert the change for now and then re-publish it along with the db migration.
nikurt
pushed a commit
that referenced
this pull request
May 4, 2023
Part of #8243. This PR implements value inlining MVP for state sync: * add `FlatStateValue::Inlined` variant to store inlined values as part of `FlatState` and `FlatStateChanges` on disk. * change flat storage API to return `FlatStateValue` instead of `ValueRef`. The following will be implemented separately: * Migration for existing `FlatState` values. This is required for state sync, but quite involved, so decided to keep it separately. * Inlining for cached flat state deltas: for now we keep those as `ValueRef`. * Using inlined values for transaction processing: for now we convert inlined values to `ValueRef`.
nikurt
pushed a commit
that referenced
this pull request
May 4, 2023
…9005) This reverts #8988 as it results in node crashing: [zulip](https://near.zulipchat.com/#narrow/stream/345766-pagoda.2Fstorage.2Fflat-storage/topic/Canary.20crashes/near/355722627). The original PR changes data format for delta changes on disk, so binary with the new code failed to deserialise deltas written by the old code. I suggest we revert the change for now and then re-publish it along with the db migration.
nikurt
pushed a commit
that referenced
this pull request
May 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #8243.
This PR implements value inlining MVP for state sync:
FlatStateValue::Inlinedvariant to store inlined values as part ofFlatStateandFlatStateChangeson disk.FlatStateValueinstead ofValueRef.The following will be implemented separately:
FlatStatevalues. This is required for state sync, but quite involved, so decided to keep it separately.ValueRef.ValueRef.