Errors not properly reported to Datadog

[Meemaw]

Describe the bug
When using otel datadog exporter, errors are not properly sent to Datadog. In the UI you can see there is an error, but its missing stack trace, description etc.

This is how it looks in router:

This is how it looks with a service that properly reports errors:

Expected behavior
Errors should be sent to Datadog in a way Datadog expects them.

[BrynCooke]

Possibly related to #3226. We may not be creating events at the correct place right now. But we need to investigatge.

[sbehrends]

Not exactly matches but we are also facing some OpenTelemetry issues here.

Temporally @MatthiasLangGrover resolved it by creating a rhai script to log the errors

fn subgraph_service(service, subgraph) {
    let response_callback = |response| {
        if !response.body.errors.is_empty() {
            for error in response.body.errors {
                log_error(`Error received from subgraph (${subgraph}): ${error.message}`);
            }
        }
    };
    service.map_response(response_callback);
}

[BrynCooke]

It looks like there is an issue for events that was introduced in: #2999

In particular this optimization assumes that the otel layer is not interested in events, which is not the case.

[BrynCooke]

After removing the prefiltering I can see that via OTLP events are transmitted to Datadog:

But it's not clear how to link that to the Datadog UI.

The Datadog exporter fared even worst with no events sent at all.

I think we need to do the following:

1. Reinstate events being sent to the otel layer
2. Then have a sustained effort to try and make sure the DataDog integration is good and we have appropriate instructions for users to set everything up.

[Geal]

@Meemaw since #4102 it is possible to set the dd.trace_id option on span attributes to get the correlation id https://www.apollographql.com/docs/router/configuration/telemetry/instrumentation/spans#span-configuration-example
could you check if that is enough to get the log/span correlation working now?