Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ctlog: write pending tiles to protect against concurrent sequencers #1

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

ctlog: write pending tiles to protect against concurrent sequencers #1

wants to merge 1 commit into from

Commits on Mar 17, 2024

  1. ctlog: write pending tiles to protect against concurrent sequencers 

    If two sequencers (somehow) end up running at the same time, they
could scribble over each other's uploads to the backend storage.

Protect against that special case by writing the tiles in two steps. First, all
tiles are staged in a unique tar file (keyed by the hash of the new tree), and
only written to their final location after a successful lock update.

In the normal case, this will incur one extra upload and delete, adding one
write operation, and double the amount of data written to backend storage.

To make sure that the (rare) recovery code path is well-tested, use the same
code path in both the recovery and normal case.

A special IAM policy could allow deletes only of pending files.
    @jellevandenhooff
    jellevandenhooff committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    b89dbd8 View commit details
    Browse the repository at this point in the history