[db] wait for token to propagate

[FredKSchott]

Changes

• A new attempt to solve this problem: [db] Pass preferred region for token management #10428
• That PR did solve the problem when each region had a single machine running, but failed to account for replication across multiple machines in the same region.
• In practice, Machine A in region sjc could create your token and Machine B (also in region sjc) could attempt to use it before it's been propagated. Pinning to region sjc did not solve the problem in that case.
• The new solution removes the complexity of the original one, and now just embraces replication as something to wait for vs. trying to build around and solve for. It adds a delay on token creation greater than the maximum time our tokens take to propagate (2sec of wait vs. 1600ms of maximum token propagation). In the near future, we should be able to get this down closer to 1sec of wait time.

Untitled.mov

Testing

• Tested manually

Docs

• N/A

[changeset-bot]

🦋 Changeset detected

Latest commit: 05db6f1

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[matthewp]

Approved to get this fix out the door. Would love to discuss trying to figure out a deterministic solution for the future.