otel: check attribute for all calls: trace_id

[deniseli]

If you click on any event here , you'll see a trace_id field under Info > otel. Our friendly customer team needs to use trace_id to trace verb-to-verb calls.

The existing trace_id that gets logged by default is probably the one described here.

Separately, our request headers contain a trace_id that can definitely be used to track verb-to-verb calls.

We do not know if the default trace_id field provided by otel (i.e. what we have now in Datadog) is actually correctly instrumented such that it can be used to trace verb-to-verb calls. We also do not know if otel's trace_id is the same as our request headers' trace_id.

The task here is to:

1. Find answers to these 2 unknowns.
2. If the existing trace_id does not enable tracing of verb-to-verb calls, then:
   1. we need to either:
      1. Fix it so that the spans we send to otel correctly cover the full call stack
      2. OR, start logging our request headers' trace_id as a new attribute
   2. Cross reference with otel: chat with our friendly customer team to figure out what they expect to see/work in DataDog that isn't WAI #2022 to make sure the users are seeing the same thing as us. Or if they don't realize it's broken and we do, we should give them a heads up (so they're aware until we've fixed it)
3. Separately, we should check if any request header data (including trace_id) is persisted by default. We should also take a quick pass over to make sure we aren't exporting any sensitive data (e.g. auth tokens).

[safeer]

Fixed in #2144