roachtest: replicate/wide failed

[cockroach-teamcity]

roachtest.replicate/wide failed with artifacts on master @ e7b15ebaed9c14668ade0a7827a5525aedef1ab0:

test artifacts and logs in: /artifacts/replicate/wide/run_1
(test_impl.go:291).Fatal: pq: unable to look up descriptor for n7

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=1 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-22048

Epic CRDB-16746

[cockroach-teamcity]

roachtest.replicate/wide failed with artifacts on master @ 9c21578450e395c83a1dc0df7090296fef06e006:

test artifacts and logs in: /artifacts/replicate/wide/run_1
(test_impl.go:291).Fatal: pq: unable to look up descriptor for n7

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=1 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[aliher1911]

Looks like either a flake because we don't handle transient restart errors or some underlying behaviour change.

Test fails because query for unavailable ranges fails after cluster is restarted with 6 nodes out of 9:

cockroach/pkg/cmd/roachtest/tests/allocator.go

Line 416 in 89dd056

if err := db.QueryRow(query).Scan(&unavailable, &underReplicated); err != nil {

N7 is the node removed from a cluster.

[cockroach-teamcity]

roachtest.replicate/wide failed with artifacts on master @ 8a5cb5109d6b7e122e996a44f940bfd61698cd81:

test artifacts and logs in: /artifacts/replicate/wide/run_1
(test_impl.go:291).Fatal: pq: unable to look up descriptor for n7

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=1 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[aliher1911]

Seem broken by #92787
@arulajmani you reviewed that PR, any thoughts?

[arulajmani]

That PR effectively reverts the changes made in #92293 to introduce a new end-point, instead of using Gossip for NodeDescriptors.

Looking at crdb_internal.go before #92293, we constructed a node locality map as such:

cockroach/pkg/sql/crdb_internal.go

Lines 3669 to 3677 in 8f8cc70

	// Map node descriptors to localities
	descriptors, err := getAllNodeDescriptors(p)
	if err != nil {
	return nil, nil, err
	}
	nodeIDToLocality := make(map[roachpb.NodeID]roachpb.Locality)
	for _, desc := range descriptors {
	nodeIDToLocality[desc.NodeID] = desc.Locality
	}

Note that getAllNodeDescriptors also gets descriptors through gossip, so the change in #92787 to use GetNodeDescriptor continues to use the same source.

However, it seems like we inadvertently changed the behaviour around here:

cockroach/pkg/sql/crdb_internal.go

Lines 3732 to 3735 in 8f8cc70

	replicaLocality := nodeIDToLocality[replica.NodeID].String()
	if err := replicaLocalityArr.Append(tree.NewDString(replicaLocality)); err != nil {
	return nil, err
	}

Before these two PRs went in, we would include an empty locality if the node descriptor wasn't found in Gossip, as there wouldn't be an entry in the nodeIDToLocality map. In #92787 we inadvertently started erroring out if we couldn't find the descriptor in Gossip.

It should be easy enough to bring back the old behaviour. cc @ecwall, assigning this to you given you authored the 2 patches linked above. I think all we need to do here is swallow the error from GetNodeDescriptor and use an empty string as the locality.

---

One thing that isn't clear to me is why we're not able to retrieve this descriptor from Gossip, given our TTL of 2 hours. Might be worth trying to understand this a bit more.

[cockroach-teamcity]

roachtest.replicate/wide failed with artifacts on master @ 146556e19f5e4fdc8c3e6a623b280cc33aee4d18:

test artifacts and logs in: /artifacts/replicate/wide/run_1
(test_impl.go:291).Fatal: pq: unable to look up descriptor for n7

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=1 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.replicate/wide failed with artifacts on master @ 8165e3974c10e88b6ae11c6255872ea16f3a67e3:

test artifacts and logs in: /artifacts/replicate/wide/run_1
(test_impl.go:291).Fatal: pq: unable to look up descriptor for n7

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=1 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.replicate/wide failed with artifacts on master @ ec095bc2fdbe4e518b076db20e4920fab67222bf:

test artifacts and logs in: /artifacts/replicate/wide/run_1
(test_impl.go:291).Fatal: pq: unable to look up descriptor for n7

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=1 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

This test on roachdash | Improve this report!