Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release-22.1: sqlliveness: add timeouts to heartbeats #88807

Merged
merged 3 commits into from
Sep 27, 2022
Merged

release-22.1: sqlliveness: add timeouts to heartbeats #88807

merged 3 commits into from
Sep 27, 2022

Conversation

aadityasondhi
Copy link
Collaborator

Backport:

Please see individual PRs for details.

Release justification: low-risk bugfix to existing functionality

/cc @cockroachdb/release

@blathers-crl
Copy link

blathers-crl bot commented Sep 27, 2022 •
edited by aadityasondhi
Loading

Thanks for opening a backport.

Please check the backport criteria before merging:

  • Patches should only be created for serious issues or test-only changes.
  • Patches should not break backwards-compatibility.
  • Patches should change as little code as possible.
  • Patches should not change on-disk formats or node communication protocols.
  • Patches should not add new functionality.
  • Patches must not add, edit, or otherwise modify cluster versions; or add version gates.
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
  • There is a high priority need for the functionality that cannot wait until the next release and is difficult to address in another way.
  • The new functionality is additive-only and only runs for clusters which have specifically “opted in” to it (e.g. by a cluster setting).
  • New code is protected by a conditional check that is trivial to verify and ensures that it only runs for opt-in clusters.
  • The PM and TL on the team that owns the changed code have signed off that the change obeys the above rules.

Add a brief release justification to the body of your PR to justify this backport.

Some other things to consider:

  • What did we do to ensure that a user that doesn’t know & care about this backport, has no idea that it happened?
  • Will this work in a cluster of mixed patch versions? Did we test that?
  • If a user upgrades a patch version, uses this feature, and then downgrades, what happens?

@cockroach-teamcity
Copy link
Member

This change is Reviewable

dhartunian and others added 2 commits September 27, 2022 10:43
@dhartunian @aadityasondhi
sqlliveness: add timeouts to heartbeats
409dec0 
Previously, sqlliveness heartbeat operations could block on the transactions
that were involved. This change introduces some timeouts of the length of the
heartbeat during the create and refresh operations.

Resolves cockroachdb#85541

Release note: None

Release justification: low-risk bugfix to existing functionality
@aadityasondhi
ccl: lint fixes in tenant_side_test.go
e6b1ec1 
Release note: None
@aadityasondhi aadityasondhi force-pushed the backport22.1-87533-88764 branch from d93dbf2 to 5dccb42 Compare September 27, 2022 14:46
@aadityasondhi aadityasondhi mentioned this pull request Sep 27, 2022
Closed
ajwerner
ajwerner approved these changes Sep 27, 2022
View reviewed changes
Copy link
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 5 of 5 files at r1, 2 of 2 files at r2, 2 of 2 files at r3, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @aadityasondhi)

@aadityasondhi aadityasondhi force-pushed the backport22.1-87533-88764 branch from 5dccb42 to abff60b Compare September 27, 2022 15:50
@aadityasondhi
sqlliveness: change heartbeat timeouts to be a fraction of TTL
25d0c32 
Previously, the heartbeat timeouts were being set as the length of the
heartbeat. This can cause issues when there is longer latency in the
system and caused flaky tests.

By making the heartbeat timout a fraction of the TTL, we are able to
avoid timing out too early and also ensure that we can retry at least
once before the seession expiry.

Ran test with `--stress` to verify it doesn't flake:
```
2559 runs so far, 0 failures, over 5m0s
```

Resolves: cockroachdb#88743

Release note: None
@aadityasondhi aadityasondhi force-pushed the backport22.1-87533-88764 branch from abff60b to 25d0c32 Compare September 27, 2022 17:36
@aadityasondhi aadityasondhi merged commit df4ea07 into cockroachdb:release-22.1 Sep 27, 2022
@aadityasondhi aadityasondhi deleted the backport22.1-87533-88764 branch September 27, 2022 21:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ajwerner ajwerner ajwerner approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@aadityasondhi @cockroach-teamcity @ajwerner @dhartunian