Introduce a new pgcopydb internal message: ENDPOS. #321
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When reaching endpos in the middle of a transaction we should stop processing the stream, transform what we have, and apply up to the last full transaction, ignoring the partial one at the end. Because of the streaming approach taken by pgcopydb, the next best thing we can do is to ROLLBACK the last partial transaction while still being able to recognise that --endpos has been reached. Previously to this patch, pgcopydb would use the endpos LSN to forge a keepalive message, meaning that we would then update our replay_lsn and replication origin tracking to a position in the middle of a transaction we did not replay. In case when the user then restart pgcopydb with a new endpos then we would skip one transaction which was written off as already replayed. This patch fixes this situation.
It should be possible to reset endpos to a later point in time and resume replaying changes up to the new point in time, even when the previously set endpos did fell in the middle of a transaction.
When comparing startpos with a previously written JSON file we need to avoid writing the same LSN twice, but that computation/guard doesn't apply when starting CDC the first time from the replication slot: then we want to write starting at the startpos, not skip it.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When reaching endpos in the middle of a transaction we should stop processing the stream, transform what we have, and apply up to the last full transaction, ignoring the partial one at the end.
Because of the streaming approach taken by pgcopydb, the next best thing we can do is to ROLLBACK the last partial transaction while still being able to recognise that --endpos has been reached.
Previously to this patch, pgcopydb would use the endpos LSN to forge a keepalive message, meaning that we would then update our replay_lsn and replication origin tracking to a position in the middle of a transaction we did not replay.
In case when the user then restart pgcopydb with a new endpos then we would skip one transaction which was written off as already replayed. This patch fixes this situation.