Skip to content

Commit

Permalink
ccl/sqlproxyccl: fix TestWatchPods under stressrace
Browse files Browse the repository at this point in the history
Fixes #69220.
Regression from #67452.

In #67452, we omitted DRAINING pods from the tenant directory. Whenever a pod
goes into the DRAINING state, the pod watcher needs time to update the
directory. Not waiting for that while calling EnsureTenantAddr produces a
stale result. This commit updates TestWatchPods by polling on EnsureTenantAddr
until that has been completed.

Release note: None
  • Loading branch information
jaylim-crl committed Dec 16, 2021
1 parent ae17f3d commit 1120d60
Showing 1 changed file with 10 additions and 7 deletions.
17 changes: 10 additions & 7 deletions pkg/ccl/sqlproxyccl/tenant/directory_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -101,9 +101,10 @@ func TestWatchPods(t *testing.T) {

// Ensure that all addresses have been cleared from the directory, since
// it should only return RUNNING addresses.
addrs, err := dir.LookupTenantAddrs(ctx, tenantID)
require.NoError(t, err)
require.Empty(t, addrs)
require.Eventually(t, func() bool {
addrs, _ := dir.LookupTenantAddrs(ctx, tenantID)
return len(addrs) == 0
}, 10*time.Second, 100*time.Millisecond)

// Now shut the tenant directory down.
processes := tds.Get(tenantID)
Expand All @@ -120,10 +121,12 @@ func TestWatchPods(t *testing.T) {
require.Equal(t, addr, pod.Addr)
require.Equal(t, tenant.DELETING, pod.State)

require.Eventually(t, func() bool {
addrs, _ := dir.LookupTenantAddrs(ctx, tenantID)
return len(addrs) == 0
}, 10*time.Second, 100*time.Millisecond)
// We know that the directory should have been emptied earlier since we
// don't add DRAINING pods to the directory, so putting the pod into the
// DELETING state should not make a difference.
addrs, err := dir.LookupTenantAddrs(ctx, tenantID)
require.NoError(t, err)
require.Empty(t, addrs)

// Resume tenant again by a direct call to the directory server
_, err = tds.EnsurePod(ctx, &tenant.EnsurePodRequest{tenantID.ToUint64()})
Expand Down

0 comments on commit 1120d60

Please sign in to comment.