Opni logging should collect traces

[dbason]

This will involve:

1. Implementing Export methods for traces in the otel forwarder
2. Adding a trace pipeline to the otel preprocessor config
3. Implementing an Export method for traces in the gateway
4. Adding a trace pipline to the otel config for logging.

[jaehnri]

@dbason I have a few questions 😅

1. I've understood that the logging plugin has a forwarder responsible for sending data to the upstream cluster via OTLP. In the upstream, the component responsible for receiving these OTLP logs is this supportagent?
   

   opni/pkg/supportagent/shipper/otlp.go

   Lines 21 to 23 in 13d8a9f

   	type otlpShipper struct {
   	otlpShipperOptions
   	client collogspb.LogsServiceClient

2. Is the following the preprocessing and OpenSearch ingestion pipeline?
   

   opni/pkg/resources/preprocessor/workloads.go

   Lines 93 to 98 in 13d8a9f

   	pipelines:
   	logs:
   	receivers: ["otlp"]
   	processors: ["resource", "attributes", "transform"]
   	exporters: ["opensearch"]
   	`))

3. Why are there otel collectors both downstream and upstream?

[dbason]

The supportagent is something different, that's a standalone tool. I'll ping you and we can set up something to go over the artchitecture.

[jaehnri]

As of now, there is no support for traces in our custom OpenSearch exporter:
https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter

However, there is ongoing development for it (see open-telemetry/opentelemetry-collector-contrib#23611), and traces, specifically, should be added soon (see open-telemetry/opentelemetry-collector-contrib#24540).

For that matter, we planned to pause the development of this feature for now and come back when at least the traces are available.