Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rpc: TestRemoteOffsetUnhealthy failed #121238

Closed
cockroach-teamcity opened this issue Mar 27, 2024 · 1 comment · Fixed by #121279
Closed

rpc: TestRemoteOffsetUnhealthy failed #121238

cockroach-teamcity opened this issue Mar 27, 2024 · 1 comment · Fixed by #121279
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Milestone
23.2

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Mar 27, 2024 •
edited by cockroach-jira-scripts
Loading

rpc.TestRemoteOffsetUnhealthy failed with artifacts on release-23.2 @ 42d8d22934b3dd6988e6d66bdbfce6ed216a231c:

=== RUN   TestRemoteOffsetUnhealthy
    context_test.go:1167: initial connection heartbeat failed: operation "conn heartbeat" timed out after 162ms (given timeout 100ms): grpc: context deadline exceeded [code 4/DeadlineExceeded]
E240327 19:59:44.514839 436 2@rpc/peer.go:601  [rnode=2,raddr=127.0.0.1:32775,class=default,rpc] 19  failed connection attempt (last connected 0s ago): operation "conn heartbeat" timed out after 162ms (given timeout 100ms): grpc: context deadline exceeded [code 4/DeadlineExceeded]
--- FAIL: TestRemoteOffsetUnhealthy (0.41s)

Parameters:

  • TAGS=bazel,gss
  • stress=true
Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-37149
@cockroach-teamcity cockroach-teamcity added branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. T-kv KV Team labels Mar 27, 2024
@cockroach-teamcity cockroach-teamcity added this to the 23.2 milestone Mar 27, 2024
@kvoli
Copy link
Collaborator

kvoli commented Mar 28, 2024

Duplicate of #116898.

@kvoli kvoli added X-duplicate Closed as a duplicate of another issue. P-3 Issues/test failures with no fix SLA and removed release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Mar 28, 2024
@kvoli kvoli mentioned this issue Mar 28, 2024
Merged
craig bot pushed a commit that referenced this issue Mar 28, 2024
@kvoli
Merge #121279
fbae232 
121279: rpc: expect eventual connection in remote offset unhealthy test r=arulajmani a=kvoli

`TestRemoteOffsetUnhealthy` manually connects a grid of peers to later assert on the remote offset between rpc connections. The initial connection setup can timeout under resource constraints.

Retry the initial connection when setting up the test.

Fixes: #121238
Fixes: #116898
Release note: None

Co-authored-by: Austen McClernon <austen@cockroachlabs.com>
@craig craig bot closed this as completed in 2fe2f25 Mar 28, 2024
blathers-crl bot pushed a commit that referenced this issue Mar 28, 2024
@kvoli
rpc: expect eventual connection in remote offset unhealthy test
7e46ae8 
`TestRemoteOffsetUnhealthy` manually connects a grid of peers to later
assert on the remote offset between rpc connections. The initial
connection setup can timeout under resource constraints.

Retry the initial connection when setting up the test.

Fixes: #121238
Fixes: #116898
Release note: None
@blathers-crl blathers-crl bot mentioned this issue Mar 28, 2024
Merged
blathers-crl bot pushed a commit that referenced this issue Mar 28, 2024
@kvoli
rpc: expect eventual connection in remote offset unhealthy test
997a118 
`TestRemoteOffsetUnhealthy` manually connects a grid of peers to later
assert on the remote offset between rpc connections. The initial
connection setup can timeout under resource constraints.

Retry the initial connection when setting up the test.

Fixes: #121238
Fixes: #116898
Release note: None
@blathers-crl blathers-crl bot mentioned this issue Mar 28, 2024
Merged
@github-project-automation github-project-automation bot moved this to Closed in KV Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.2 Used to mark GA and release blockers, technical advisories, and bugs for 23.2 C-test-failure Broken test (automatically or manually discovered). O-robot Originated from a bot. P-3 Issues/test failures with no fix SLA T-kv KV Team X-duplicate Closed as a duplicate of another issue.
Projects
No open projects
KV
Status: Closed
Milestone
23.2
Development

Successfully merging a pull request may close this issue.

rpc: expect eventual connection in remote offset unhealthy test kvoli/cockroach
2 participants
@cockroach-teamcity @kvoli