Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

closedts: allow serving follower after transfer for reads before lease start #35129

Closed
ajwerner opened this issue Feb 21, 2019 · 0 comments · Fixed by #35130
Closed

closedts: allow serving follower after transfer for reads before lease start #35129

ajwerner opened this issue Feb 21, 2019 · 0 comments · Fixed by #35130
Assignees
@ajwerner
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@ajwerner
Copy link
Contributor

When a lease transfer occurs, follower replicas become unable to server follower reads until a new closed timestamp update is received. It is unfortunate that a lease transfer leads to a period of time where a follower can no longer serve a read at a timestamp it was previously able to serve. @nvanbenschoten observes in #35120 (comment) that the lease transfer start time can effectively act as a closed timestamp for a given range. A range can determine if it can use a follower read by checking whether the requested timestamp falls before max(store-wide CT, lease start ts). While for the former case we need to ensure the MLAI, for the latter case we should be safe to read at time before the lease start so long as the follower replica currently knows of a lease which starts after the requested timestamp.

Describe the solution you'd like
Lease transfers do not prevent follower reads from timestamps which were previously safe.

Describe alternatives you've considered
One suggestion was that we could keep a history of leases and use that history to determine a valid previous closed timestamp .
@ajwerner ajwerner self-assigned this Feb 21, 2019
ajwerner added a commit to ajwerner/cockroach that referenced this issue Feb 21, 2019
@ajwerner
storage: use lease start time to determine follower read safety
184c63c 
This change obviously needs some testing. I'm putting it out to make sure I
understand the issue and proposed solution.

This change attempts to address a situation where a lease transfer may prevent
a follower read. This happens because after the lease transfer we no longer have
any information in the closedts storage regarding the range from the new
leaseholder.

Fixes cockroachdb#35129.

Release note: None
@ajwerner ajwerner mentioned this issue Feb 21, 2019
Merged
ajwerner added a commit to ajwerner/cockroach that referenced this issue Feb 21, 2019
@ajwerner
storage: use lease start time to determine follower read safety
f54ba93 
This change obviously needs some testing. I'm putting it out to make sure I
understand the issue and proposed solution.

This change recognizes that lease transfer start times have provide write
invariants equivalent to those provided by the closed timestamp and MLAI and
thus the maximum closed timestamp for a replica can be viewed as the maximum
timestamp of that of the current replica lease start time and the timestamp
provided by the closed timestamp provider. Utilizing this fact makes follower
reads as well as rangefeeds more robust to lease transfers. Before this change
lease transfers to different nodes might cause closed timestamp information for
a range to become unavailable leading to an inability for a range to serve a
read at a timestamp which was previously determined to be safe.

Fixes cockroachdb#35129.

Release note: None
@ajwerner ajwerner mentioned this issue Feb 26, 2019
Closed
@awoods187 awoods187 added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Mar 6, 2019
ajwerner added a commit to ajwerner/cockroach that referenced this issue Mar 8, 2019
@ajwerner
storage: use lease start time to determine follower read safety
b83eb99 
This change recognizes that lease transfer start times have provide write
invariants equivalent to those provided by the closed timestamp and MLAI and
thus the maximum closed timestamp for a replica can be viewed as the maximum
timestamp of that of the current replica lease start time and the timestamp
provided by the closed timestamp provider. Utilizing this fact makes follower
reads as well as rangefeeds more robust to lease transfers. Before this change
lease transfers to different nodes might cause closed timestamp information for
a range to become unavailable leading to an inability for a range to serve a
read at a timestamp which was previously determined to be safe.

Fixes cockroachdb#35129.

Release note: None
ajwerner added a commit to ajwerner/cockroach that referenced this issue Mar 12, 2019
@ajwerner
storage: use lease start time to determine follower read safety
7037b54 
This change recognizes that lease transfer start times have provide write
invariants equivalent to those provided by the closed timestamp and MLAI and
thus the maximum closed timestamp for a replica can be viewed as the maximum
timestamp of that of the current replica lease start time and the timestamp
provided by the closed timestamp provider. Utilizing this fact makes follower
reads as well as rangefeeds more robust to lease transfers. Before this change
lease transfers to different nodes might cause closed timestamp information for
a range to become unavailable leading to an inability for a range to serve a
read at a timestamp which was previously determined to be safe.

Fixes cockroachdb#35129.

Release note: None
craig bot pushed a commit that referenced this issue Mar 14, 2019
@ajwerner
Merge #35130
d8d92b7 
35130: storage: use lease start time to determine follower read safety r=ajwerner a=ajwerner

This change obviously needs some testing. I'm putting it out to make sure I
understand the issue and proposed solution.

This change attempts to address a situation where a lease transfer may prevent
a follower read. This happens because after the lease transfer we no longer have
any information in the closedts storage regarding the range from the new
leaseholder.

Fixes #35129.
Fixes #32980.

Release note: None

Co-authored-by: Andrew Werner <ajwerner@cockroachlabs.com>
@craig craig bot closed this as completed in #35130 Mar 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ajwerner ajwerner

Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

storage: use lease start time to determine follower read safety ajwerner/cockroach
2 participants
@ajwerner @awoods187