-
Notifications
You must be signed in to change notification settings - Fork 3.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
roachtest: tpcc/nodes=3/w=max failed #30886
Comments
SHA: https://github.com/cockroachdb/cockroach/commits/f78cc21bb4d7bd3a79e44b4608c04684edf28179 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=939960&tab=buildLog
|
@nvanbenschoten, could you take a look at this one? The error is an opaque "context deadline exceeded". Weird that this would just come up today (or did we rename tests?) |
I think this just shows that #30804 is actually fixed now and that we're hitting a new issue. |
One issue I see is that we're not passing a |
I can reproduce the failure without the |
Fixes cockroachdb#30886. Release note: None
SHA: https://github.com/cockroachdb/cockroach/commits/6a236533824c797948eaddbd6bedf5e12eee1a13 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=942507&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/bd9de99ff5569d9c21df027544b4fdb17ffc4a55 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=946966&tab=buildLog
|
@nvanbenschoten something new then? |
SHA: https://github.com/cockroachdb/cockroach/commits/dc0e73c728e533fdb3bec63e53eec174e920ff22 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=952220&tab=buildLog
|
Last one is a fluke, but the "context deadline exceeded" above might be new. |
SHA: https://github.com/cockroachdb/cockroach/commits/2215217e8ee38d28a14eb9fd2fe9af8b0b702e7d Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=958181&tab=buildLog
|
I can't remember what the latest with |
I'm hoping it's one of the stalls I'm seeing in #30064. If a write got blocked then eventually everything else would pile on behind it and also block. I'll revisit this issue once that one's fixed. |
Is your hypothesis that the |
I don't have a real theory about the context deadlines themselves, more about the stall we see right before it. If you feel up to it, looking into the source of the error would be very helpful. Thanks! |
I see this in the logs:
Suspicious at all? |
There's also this guy:
But I don't think that |
SHA: https://github.com/cockroachdb/cockroach/commits/b0a3aada2869f44bac4cf75e0c4b2d09659767f9 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1004113&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/b01468f592f6b12ba147e1c19ea6d8ad7484ff6e Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1007980&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/04f39007a9aecf14e0f989400127d8dd16067dbd Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1010528&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/eb8345b19cf6c15b3e4fcb9c156136357d83cb2d Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1015656&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/cfd53f62f038a06e34570f11ec02bc74dabd2aab Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1021046&tab=buildLog
|
cc @asubiotto |
SHA: https://github.com/cockroachdb/cockroach/commits/96f03c95d5078ebad7167c5cdb145e365978a008 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1022695&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/153fcf347cd6a53517f607ec1e6c7c14193640dd Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1052961&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/e7dc507fa0ecc7dc5ed597ca5c6cdeb48086428c Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1057350&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/1da87b86059c282c2b2821d6b5b481f17c89d98e Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1069764&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/3c80c5e06c4aa5ed25e1cc02b78037f6ec121939 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1072755&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/d6adf24cae788d7cd967feadae8e9c0388ce5273 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1079910&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/99064bf2797cce49e2d56d0324a3344cf2312aca Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1082456&tab=buildLog
|
Last few failures have been #31361. |
SHA: https://github.com/cockroachdb/cockroach/commits/766a53defc27940d41ae8987792ecd79e76d2b65 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1090948&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/3f22b976765a7cc58b97a9fd9866f2f26493c987 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1097785&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/94dcfdef9e9fff49bf85900c1c73603073cb9fab Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1101244&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/9f084ad576e85756c5c5a7e41335d9aa2d3eee30 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1112101&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/e10fb557b11b5ff1b8609aa963da23c37a1143c8 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1113854&tab=buildLog
|
Last failure was the same as #34025. |
SHA: https://github.com/cockroachdb/cockroach/commits/395d842feb97c5bd8cad2b32b71a5156c03061eb Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1115923&tab=buildLog
|
SHA: https://github.com/cockroachdb/cockroach/commits/b9bd958fccddc699d47eccbbec80db75c10eab46 Parameters: To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1122796&tab=buildLog
|
Fixes cockroachdb#34025. Fixes cockroachdb#33624. Fixes cockroachdb#33335. Fixes cockroachdb#33151. Fixes cockroachdb#33149. Fixes cockroachdb#34159. Fixes cockroachdb#34293. Fixes cockroachdb#32813. Fixes cockroachdb#30886. Fixes cockroachdb#34228. Fixes cockroachdb#34321. It is rare but possible for a replica to become a leaseholder but not learn about this until it applies a snapshot. Immediately upon the snapshot application's `ReplicaState` update, the replica will begin operating as a standard leaseholder. Before this change, leases acquired in this way would not trigger in-memory side-effects to be performed. This could result in a regression in the new leaseholder's timestamp cache compared to the previous leaseholder, allowing write-skew like we saw in cockroachdb#34025. This could presumably result in other anomalies as well, because all of the steps in `leasePostApply` were skipped. This PR fixes this bug by detecting lease updates when applying snapshots and making sure to react correctly to them. It also likely fixes the referenced issue. The new test demonstrated that without this fix, the serializable violation speculated about in the issue was possible. Release note (bug fix): Fix bug where lease transfers passed through Snapshots could forget to update in-memory state on the new leaseholder, allowing write-skew between read-modify-write operations.
34548: storage: apply lease change side-effects on snapshot recipients r=nvanbenschoten a=nvanbenschoten Fixes #34025. Fixes #33624. Fixes #33335. Fixes #33151. Fixes #33149. Fixes #34159. Fixes #34293. Fixes #32813. Fixes #30886. Fixes #34228. Fixes #34321. It is rare but possible for a replica to become a leaseholder but not learn about this until it applies a snapshot. Immediately upon the snapshot application's `ReplicaState` update, the replica will begin operating as a standard leaseholder. Before this change, leases acquired in this way would not trigger in-memory side-effects to be performed. This could result in a regression in the new leaseholder's timestamp cache compared to the previous leaseholder's cache, allowing write-skew like we saw in #34025. This could presumably result in other anomalies as well, because all of the steps in `leasePostApply` were skipped (as theorized by #34025 (comment)). This PR fixes this bug by detecting lease updates when applying snapshots and making sure to react correctly to them. It also likely fixes the referenced issue. The new test demonstrates that without this fix, the serializable violation speculated about in the issue was possible. Co-authored-by: Nathan VanBenschoten <nvanbenschoten@gmail.com>
SHA: https://github.com/cockroachdb/cockroach/commits/0f5a289d49105cd317a2713b71546d84c25547f6
Parameters:
To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=937459&tab=buildLog
The text was updated successfully, but these errors were encountered: