[Feature Request] OpenLLMetry Integration (OpenTelemetry)

[amitjoy]

Describe the Feature
OpenTelemetry is a de facto standard for reporting metrices and OpenLLMetry uses OpenTelemetry under the hood. It would be better to have an integration to send such metrices for RAG evaluation to OpenTelemetry collectors as well.

Why is the feature important for you?
Since OpenTelemetry is a common standard which would be used easily in any existing systems.

Additional context
NA

[jjmachan]

hey @amitjoy thanks for the feat request

I have been keeping an eye on OpenLLMetry for a while and with

• feat(langchain): use callbacks by tibor-reiss · Pull Request #1170 · traceloop/openllmetry
• fix(langchain): improve callbacks by tibor-reiss · Pull Request #1317 · traceloop/openllmetry

there should be some progress and it should be easier. I'll take a closer look at this and see what would the easiest 🙂

in the mean time do you use OpenLLMetry for your LLM application tracing also? do you use any other tools to visualize the traces?

[amitjoy]

Hi @jjmachan,

Thank you for your prompt response and the detailed information regarding the integration with ragas. Yes, I do use OpenLLMetry and publish the metrics and spans to Sentry, as it can also function as an OpenTelemetry Collector. Since I already utilize Sentry for monitoring unexpected runtime issues, it seemed logical to extend its use to OpenTelemetry traces as well.