pageserver: lift decoding and interpreting of wal into wal_decoder #9487
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This one is a bit less obvious than the previous ones. I merged some of the logic that was previously in `WalIngest::ingest_record` to `WalIngest::ingest_xact_record`.
This is an odd one. It requires the current checkpoint value to decide what to do. That can't trivially be moved to the SK. It's possible with protocol change, but deferring decision for now. Hence, send the raw record and let the pageserver figure it out.
This will give us a nice evolution path when we want to add new actions for a record.
The goal of this commit is to make it clearer when we are ingesting the whole record versus when we are ingesting an action for the record. I also merged the VM bits clearing into one function since they were exactly the same.
We wish to have high level WAL decoding logic in `wal_decoder::decoder` module. For this we need the `Value` and `NeonWalRecord` types accessible there, so: 1. Move `Value` and `NeonWalRecord` to `pageserver_api::value` and `pageserver_api::record` respectively. I had to add a testing feature to `pageserver_api` to get this working due to `NeonWalRecord` test directives. 2. Get rid of `pageserver::repository` (follow up from (1)) 3. Move PG specific WAL record types to `postgres_ffi::record`. In theory they could live in `wal_decoder`, but it would create a circular dependency between `wal_decoder` and `postgres_ffi`. Long term it makes sennse for those types to be PG version specific, so that will work out nicely. 4. Move higher level WAL record types (to be ingested by pageserver) into `wal_decoder::models`
A new type is added to [`wal_decoder::models`], InterpretedWalRecord. This type contains everything that the pageserver requires in order to ingest a WAL record. The highlights are the `metadata_record` which is an optional special record type to be handled and `blocks` which stores key, value pairs to be persisted to storage. This type is produced by [`wal_decoder::models::InterpretedWalRecord::from_bytes`] from a raw PG wal record. The rest of this commit separates decoding and interpretation of the PG WAL record from its application in [`WalIngest::ingest_record`].
5202 tests run: 4995 passed, 0 failed, 207 skipped (full report)Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
d82df3b at 2024-10-23T16:11:04.803Z :recycle: |
This was referenced Oct 23, 2024
VladLazar
force-pushed
the
vlad/module-musical-chairs
branch
from
10f4b99 to
b87ac5b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
A new type is added to [
wal_decoder::models], InterpretedWalRecord. This type contains everything that the pageserver requires in order to ingest a WAL record. The highlights are themetadata_recordwhich is an optional special record type to be handled andblockswhich stores key, value pairs to be persisted to storage.This type is produced by
[
wal_decoder::models::InterpretedWalRecord::from_bytes] from a raw PG wal record.The rest of this commit separates decoding and interpretation of the PG WAL record from its application in [
WalIngest::ingest_record].Problem
Summary of changes
Checklist before requesting a review
Checklist before merging