Bug Report: Flaky test TestGatewayBufferingWhileReparenting

[GuptaManan100]

Overview of the Issue

The test TestGatewayBufferingWhileReparenting has been flaky in main.

Reproduction Steps

Run TestGatewayBufferingWhileReparenting over and over.

Binary Version

main

Operating System and Environment details

-

Log Fragments

No response

[GuptaManan100]

As far as I can tell, there is an underlying issue that can happen.

When we try to run a query, we first use the healthcheck to find the tablets which are serving -

tablets := gw.hc.GetHealthyTabletStats(target)

If we find no such tablets, we eventually use the keyspace event watcher to find whether the primary is serving or not -

primary, notServing := kev.PrimaryIsNotServing(target)

The information of healthchecks that come from the vttablets are first digested by the healthchecker in vtgates. keyspace event watcher registers for the notifications of these changes with the health-check, but those notifications are processed asynchronously.

This can lead to a situation where the primary tablet becomes non-serving, which is updated in the health-check, but not as yet in the keyspace-event-watcher.

In this case, we would infer from the health check that there are no healthy tablets, but keyspace event watcher would say that there is a serving primary tablet. This causes the loop to run again because we find ourselves in an inconsistant state. However, we have no wait before retyring the same checks, so it is possible for us to run into the same issue in the next iteration too, until the keyspace-event-watcher is able to process the health-check update. This sometimes leads to this error "inconsistent state detected, primary is serving but initially found no available tablet" surfacing out and causing the test to fail.