Introduce make_fake_batch() to avoid racy caches and redact more otel trace details

[garypen]

It transpires that there are multiple race conditions in our tests which makes snapshotting traces problematic. This problem is particularly acute when we are testing batch requests, because of the way we build up our batch requests. The most significant impact has been on the otel traces tests which have been very flaky.

A number of the tests expose subtle races in their expectations around span creation. This is more complex and warrants a detailed explanation.

We already have a mechanism for generating a query from a fake supergraph query builder. When we added support for batching, we leveraged this functionality to convert a simple single request into a batch request by doing something (conceptually at least) like this:

map the request, so that we clone it and wrap it in an array which now contains two identical requests.

That is simple and easy to implement, so we've used it in a few places in our tests.

However, when the router executes....

At various places in the router we check if items are cached to decide whether or not to do some processing. That processing may involve the generation of a tracing span. If we don't do the processing, because, let's say, data is in a cache, then we now have a potential race condition, because not doing the processing means the various snapshots that we rely on in our tests will (or won't) match.

This, in a nutshell, are the problems that this PR fixes.

1. Make items in a batch different enough that caching effects are avoided.
2. Redact various details so that sequencing is not as much of an issue in the otel traces tests

This results in a lot of updated snapshots, but the changes aren't actually that significant.

Unblocks: #5644

[router-perf]

CI performance tests

• const - Basic stress test that runs with a constant number of users
• demand-control-instrumented - A copy of the step test, but with demand control monitoring and metrics enabled
• demand-control-uninstrumented - A copy of the step test, but with demand control monitoring enabled
• enhanced-signature - Enhanced signature enabled
• events - Stress test for events with a lot of users and deduplication ENABLED
• events_big_cap_high_rate - Stress test for events with a lot of users, deduplication enabled and high rate event with a big queue capacity
• events_big_cap_high_rate_callback - Stress test for events with a lot of users, deduplication enabled and high rate event with a big queue capacity using callback mode
• events_callback - Stress test for events with a lot of users and deduplication ENABLED in callback mode
• events_without_dedup - Stress test for events with a lot of users and deduplication DISABLED
• events_without_dedup_callback - Stress test for events with a lot of users and deduplication DISABLED using callback mode
• extended-reference-mode - Extended reference mode enabled
• large-request - Stress test with a 1 MB request payload
• no-tracing - Basic stress test, no tracing
• reload - Reload test over a long period of time at a constant rate of users
• step-jemalloc-tuning - Clone of the basic stress test for jemalloc tuning
• step-local-metrics - Field stats that are generated from the router rather than FTV1
• step-with-prometheus - A copy of the step test with the Prometheus metrics exporter enabled
• step - Basic stress test that steps up the number of users over time
• xlarge-request - Stress test with 10 MB request payload
• xxlarge-request - Stress test with 100 MB request payload

[timbotnik]

This seems fine - you are more of an expert here wrt batching. It would be nice if we could have some of the tests not rely on batching and keep a tighter match on important span attributes (the signature is one of the only "must have" pieces of the Apollo OTel trace ingestor, and I noticed in this PR we are now redacting it from snapshots).

[garypen]

This seems fine - you are more of an expert here wrt batching. It would be nice if we could have some of the tests not rely on batching and keep a tighter match on important span attributes (the signature is one of the only "must have" pieces of the Apollo OTel trace ingestor, and I noticed in this PR we are now redacting it from snapshots).

I know it's not ideal. I think we'd have to have a different version of the assert_report macro for batching vs non-batching. The non-batching wouldn't redact the additional details. That might be good enough.

I'll try it out later.

[garypen]

I modified the macro to only perform the additional redaction if the test is a batch test. I think that's a nice compromise and keeps the tighter match that @timbotnik requested above.