roachtest: cdc/sink-chaos/rangefeed=true failed [skipped] #36019
Assignees
Labels
C-test-failure
Broken test (automatically or manually discovered).
O-roachtest
O-robot
Originated from a bot.
Milestone
Comments
cockroach-teamcity
added
C-test-failure
danhhz
added a commit
to danhhz/cockroach
that referenced
this issue
Mar 25, 2019
…list
In the roachtests for crdb-chaos and sink-chaos we're seeing changefeeds
fail with surprising errors:
[NotLeaseHolderError] r681: replica (n1,s1):1 not lease holder; replica (n2,s2):2 is
descriptor not found
We'd like to avoid failing a changefeed unnecessarily, so when an error
bubbles up to the top level, we'd like to retry the distributed flow if
possible. We initially tried to whitelist which errors should cause the
changefeed to retry, but this turns out to be brittle, so this commit
switches to a blacklist. Any error that is expected to be permanent is
now marked with `MarkTerminalError` by the time it comes out of
`distChangefeedFlow`. Everything else should be logged loudly and
retried.
Touches cockroachdb#35974
Touches cockroachdb#36019
Release note: None
danhhz
added a commit
to danhhz/cockroach
that referenced
this issue
Mar 26, 2019
…list
In the roachtests for crdb-chaos and sink-chaos we're seeing changefeeds
fail with surprising errors:
[NotLeaseHolderError] r681: replica (n1,s1):1 not lease holder; replica (n2,s2):2 is
descriptor not found
We'd like to avoid failing a changefeed unnecessarily, so when an error
bubbles up to the top level, we'd like to retry the distributed flow if
possible. We initially tried to whitelist which errors should cause the
changefeed to retry, but this turns out to be brittle, so this commit
switches to a blacklist. Any error that is expected to be permanent is
now marked with `MarkTerminalError` by the time it comes out of
`distChangefeedFlow`. Everything else should be logged loudly and
retried.
Touches cockroachdb#35974
Touches cockroachdb#36019
Release note: None
danhhz
added a commit
to danhhz/cockroach
that referenced
this issue
Mar 26, 2019
…list
In the roachtests for crdb-chaos and sink-chaos we're seeing changefeeds
fail with surprising errors:
[NotLeaseHolderError] r681: replica (n1,s1):1 not lease holder; replica (n2,s2):2 is
descriptor not found
We'd like to avoid failing a changefeed unnecessarily, so when an error
bubbles up to the top level, we'd like to retry the distributed flow if
possible. We initially tried to whitelist which errors should cause the
changefeed to retry, but this turns out to be brittle, so this commit
switches to a blacklist. Any error that is expected to be permanent is
now marked with `MarkTerminalError` by the time it comes out of
`distChangefeedFlow`. Everything else should be logged loudly and
retried.
Touches cockroachdb#35974
Touches cockroachdb#36019
Release note: None
craig bot
pushed a commit
that referenced
this issue
Mar 27, 2019
36132: changefeedccl: switch high-level retry marker from whitelist to blacklist r=nvanbenschoten a=danhhz
In the roachtests for crdb-chaos and sink-chaos we're seeing changefeeds
fail with surprising errors:
[NotLeaseHolderError] r681: replica (n1,s1):1 not lease holder; replica (n2,s2):2 is
descriptor not found
We'd like to avoid failing a changefeed unnecessarily, so when an error
bubbles up to the top level, we'd like to retry the distributed flow if
possible. We initially tried to whitelist which errors should cause the
changefeed to retry, but this turns out to be brittle, so this commit
switches to a blacklist. Any error that is expected to be permanent is
now marked with `MarkTerminalError` by the time it comes out of
`distChangefeedFlow`. Everything else should be logged loudly and
retried.
Touches #35974
Touches #36019
Release note: None
Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
|
SHA: https://github.com/cockroachdb/cockroach/commits/17565100d1e7c66341e6db3e39bb66202958cb81 Parameters: To repro, try: Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1204567&tab=buildLog |
|
Looks like my constants in #36132 need tuning |
|
SHA: https://github.com/cockroachdb/cockroach/commits/d03a34e92d2ee558fb6aedb0709b733a1fab97f4 Parameters: To repro, try: Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1207666&tab=buildLog |
|
SHA: https://github.com/cockroachdb/cockroach/commits/a6b3c540b696002b2ed07036a657612995d6d1ab Parameters: To repro, try: Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1211353&tab=buildLog |
|
SHA: https://github.com/cockroachdb/cockroach/commits/668162cc99e4f3198b663b1abfa51858eeb3ccb8 Parameters: To repro, try: Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1212251&tab=buildLog |
|
SHA: https://github.com/cockroachdb/cockroach/commits/2851c7d56ee4966109691b5c48c73ec8d4cc9847 Parameters: To repro, try: Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1215354&tab=buildLog |
|
SHA: https://github.com/cockroachdb/cockroach/commits/c6df752eefe4609b8a5bbada0955f79a2cfb790e Parameters: To repro, try: Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1217763&tab=buildLog |
danhhz
changed the title
danhhz
added a commit
to danhhz/cockroach
that referenced
this issue
Apr 15, 2019
For a while, the cdc/crdb-chaos and cdc/sink-chaos roachtests have been failing because an error that should be marked as retryable wasn't. As a result of the discussion in cockroachdb#35974, I tried switching from a whitelist (retryable error) to a blacklist (terminal error) in cockroachdb#36132, but on reflection this doesn't seem like a great idea. We added a safety net to prevent false negatives from retrying indefinitely but it was immediately apparent that this meant we needed to tune the retry loop parameters. Better is to just do the due diligence of investigating the errors that should be retried and retrying them. The commit is intended for backport into 19.1 once it's baked for a bit. Closes cockroachdb#35974 Closes cockroachdb#36018 Closes cockroachdb#36019 Closes cockroachdb#36432 Release note (bug fix): `CHANGEFEED` now retry instead of erroring in more situations
craig bot
pushed a commit
that referenced
this issue
Apr 16, 2019
36804: sql/sem/pretty: use left alignment for column names in CREATE r=knz a=knz
Before:
```
CREATE TABLE t (
name STRING,
id INT8
NOT NULL
PRIMARY KEY
)
```
After:
```
CREATE TABLE t (
name STRING,
id INT8
NOT NULL
PRIMARY KEY
)
```
36852: changefeedccl: switch retryable errors back to a whitelist r=nvanbenschoten a=danhhz
For a while, the cdc/crdb-chaos and cdc/sink-chaos roachtests have been
failing because an error that should be marked as retryable wasn't. As a
result of the discussion in #35974, I tried switching from a whitelist
(retryable error) to a blacklist (terminal error) in #36132, but on
reflection this doesn't seem like a great idea. We added a safety net to
prevent false negatives from retrying indefinitely but it was
immediately apparent that this meant we needed to tune the retry loop
parameters. Better is to just do the due diligence of investigating the
errors that should be retried and retrying them.
The commit is intended for backport into 19.1 once it's baked for a bit.
Closes #35974
Closes #36018
Closes #36019
Closes #36432
Release note (bug fix): `CHANGEFEED` now retry instead of erroring in
more situations
36872: coldata: fix Slice when slicing up to batch.Length() r=yuzefovich a=asubiotto
A panic occured because we weren't treating the end slice index as
exclusive, resulting in an out of bounds panic when attempting to slice
the nulls slice.
Release note: None
Co-authored-by: Raphael 'kena' Poss <knz@cockroachlabs.com>
Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
Co-authored-by: Alfonso Subiotto Marqués <alfonso@cockroachlabs.com>
danhhz
added a commit
to danhhz/cockroach
that referenced
this issue
Apr 24, 2019
For a while, the cdc/crdb-chaos and cdc/sink-chaos roachtests have been failing because an error that should be marked as retryable wasn't. As a result of the discussion in cockroachdb#35974, I tried switching from a whitelist (retryable error) to a blacklist (terminal error) in cockroachdb#36132, but on reflection this doesn't seem like a great idea. We added a safety net to prevent false negatives from retrying indefinitely but it was immediately apparent that this meant we needed to tune the retry loop parameters. Better is to just do the due diligence of investigating the errors that should be retried and retrying them. The commit is intended for backport into 19.1 once it's baked for a bit. Closes cockroachdb#35974 Closes cockroachdb#36018 Closes cockroachdb#36019 Closes cockroachdb#36432 Release note (bug fix): `CHANGEFEED` now retry instead of erroring in more situations
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
SHA: https://github.com/cockroachdb/cockroach/commits/dfa23c01e4ea39b19ca8b2e5c8a4e7cf9b9445f4
Parameters:
To repro, try:
Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1189954&tab=buildLog
The text was updated successfully, but these errors were encountered: