pageserver: move toward more efficient IndexPart serialization #4939
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will replace hand encode/decode done in various places.
The v2 format duplicates all layer names in a set and a map. Remove the `timeline_layers` from the structure, and demote it to just being serialized from `layer_metadata` keys: this prepares for a v3 format that removes the field entirely, which can be done after this version is fully deployed. Also clean up the IndexPart's fields to disentangle it from serialization: - Remove disk_consistent_lsn from IndexPart, as it only exists as a convenience to people looking at JSON. - Replace metadata_bytes with metadata, and do the serialization of this along with the struct as a whole. The unit test that tested v1 decode with missing_layers and inconsistent layer_metadata is removed, because all production data had already been rewritten to avoid that. It was already the case that an index_part with incomplete layer_metadata would fail to attach.
jcsp
added
c/storage/pageserver
jcsp
changed the title
1264 tests run: 1210 passed, 1 failed, 53 skipped (full report)Failures on Posgres 14
The comment gets automatically updated with the latest test results
7d80f9f at 2023-08-09T09:58:22.429Z :recycle: |
|
Doing this more gradually in #4972 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
IndexPart contains two redundant lists of layer names: a set of the names, and then a map of name to metadata.
It also has two places where serialization stuff peeks into the runtime view of the data:
metadata_bytes, we should just serialize/deserialize it like other serialized data, rather than having to do from_bytes/to_bytes separately.Summary of changes
The IndexPart structure removes
timeline_layersanddisk_persistent_lsn.metadata_bytesbecomes justmetadata. Explicit Serialize and Deserialize implementations map from the serialized structured to the cleaned up IndexPart.The encoding version remains v2, so that existing pageservers are forward compatible to what we write. The
timeline_layersin serialized version is synthesized from the keys oflayer_metadata. The next release after deploying this, we can bump the version to 3 and remove the field.Checklist before requesting a review
Checklist before merging