testutils: capture Side-Eye snapshots in SucceedsSoon

[asg0451]

Make testutils.SucceedsSoon() capture a snapshot of the test process
with Side-Eye when the deadline expires. The idea is to help with
debugging flaky unit tests, both locally and on CI: the snapshot should
help us figure out where the operation that the test was waiting on has
gotten stuck.

The snapshot is generated if the SIDE_EYE_TOKEN environment variable is
set (it should be set to our organization's API token). The test log will
have a link to app.side-eye.io for visualizing the collected snapshot.

In the future, perhaps we can use Side-Eye to also collect execution
traces and CPU profiles for even more information.

Epic: None
Release note: None

[cockroach-teamcity]

This change is 

[asg0451]

thought: maybe it would be better if this was opt-in for tests

[stevendanna]

thought: maybe it would be better if this was opt-in for tests

I wonder if perhaps the place to put this is into some function that you can defer. It could look at (*testing.T).Failed() and then capture the snapshot if the environment variable is set and if the test failed. This would make it easy to drop this into a test that uses other means of waiting for things as well. The tricky bit would be that you would want to make sure you always placed the defer after any defered stopper Stops.

Out of curiosity, would it be possible to take the snapshot but save it to disk rather than send it to side-eye directly? Then perhaps side-eye could allow us to upload the snapshot if we end up actually caring about it?