Export marshalers option in kafkaexporter to make it extensible

[mmynk]

Component(s)

exporter/kafka

Describe the issue you're reporting

I'm trying to extend the existing kafkaexporter by passing a FactoryOption to override the default marshalers but I don't know how to instantiate one because the param to that is not exported.

This is what I wish to do:

func NewFactory() exporter.Factory {
	return kafkaexporter.NewFactory(kafkaExporter.WithMetricsMarshalers(metricsMarshalers()))  // export WithMetricsMarshalers in kafkaexporter
}

But since withMetricsMarshalers is private in kafkaexporter, I cannot do this.
I cannot implement my own withMetricsMarshalers() either because FactoryOption cannot be instantiated as the fields in that type are also private.

Am I missing something? Are components not supposed to be extended? Or is it possible to export this fields?

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

For context, this was un-exported on purpose in #27583, as a part of #26304. The API rules were introduced and defined in #25855.

Can you expand on your goal and use case? Do you have a specific data format you're wanting to add support for?

[mmynk]

My use case is to do some data massaging before publishing the message, maybe add some diagnostic fields and such.
Other than that, the receiver that we have collects, say, x number of metrics every n seconds and forwards. I wanted to send each of these x metrics as a separate message on Kafka.
I could do this unfolding if I were able to write my own marshaler.
I could also write arbitrary massaging logic, say I want to publish custom structs rather than ResourceMetrics. But would this be a more suitable candidate for a custom processor rather than extending an exporter?

Hope that makes sense.

[crobert-1]

I'm pretty new to kafka, so please excuse my ignorance here: What's the value of sending each metric as a separate message? Would the config option producer.flush_max_messages work for you here?

For logic specific to exporting to Kafka we'd want it to be in the exporter, but if you're doing more general data transformation we'd want to use processors. If you share more information about what kind of data you'd want to publish with metrics I can point you towards some possible existing processors that would work.

[mmynk]

I'm sorry, let me clarify my use case. It's not that I want to send each message individually (although that's what I wrote earlier, my bad), I want to really unwrap the metrics and send my own custom proto / json on the bus.
For example, we have resourceMetrics -> scopeMetrics -> metrics, I'd wish to unfold these and parse them into my own custom struct and publish it.

I understand this is out of scope for OpenTelemetry but if those methods were exported, that'd make building such custom logic really easy. Additionally, as I mentioned I could add other types of enhancements as well.

[crobert-1]

From my understanding that's not a supported purpose of the of the marshaller. Marshallers are meant to handle data formatting conversions with official data schemas, like OTLP, Zipkin, Jaegar, etc.

We do have existing processors that can do these kinds of transformations though, so it would be worth checking those out. The transform processor probably has the most functionality with changing metric formatting. Using the metric context or even the datapoint context gets you access to most metric fields. From there you can modify and move around the fields to try to make the format you're looking for.

If processors don't work for your use-case, you're welcome to create your own collector repository and implement a marshaller that works for you.

Let me know though if I've misunderstood your request here!

[mmynk]

That makes sense, I wrote a custom exporter to handle this. Thanks!