Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs volume stopped showing log level after switching to native OTLP ingestion endpoint #13711

Open
jmichalek132 opened this issue Jul 30, 2024 · 4 comments

Comments

@jmichalek132
Copy link
Contributor

jmichalek132 commented Jul 30, 2024

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

  1. Started Loki (SHA or version) 3.1
  2. Started Java app with the otel java agent (1.25.1) (sending logs to otel collector via OTLP) and otel collector contrib 0.98.0
  3. Switching from using the loki exporter to using the native OTLP ingest endpoint
  4. Query: {service_name="<service_name>"}

Expected behavior
I expected the log volume to keep showing the log severity but now it's just showing unknown.

After reading https://grafana.com/docs/loki/latest/send-data/otel/native_otlp_vs_loki_exporter/#formatting-of-logs I assume it's due to with the previous configuration severity would be an indexed label.
I wonder if there is a way to restore the behavior with native OTLP log ingestion. To be clear the log level is still present it's just under different name (detected_level) and in the structured metatada instead of the index as far I can tell.

Environment:

  • Infrastructure: AKS Kubernetes
  • Deployment tool: helm 6.7.3

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

image

@jmichalek132
Copy link
Contributor Author

Maybe related to #13688

@lieberlois
Copy link

Any solution for this?

@Pamalosebi
Copy link

I am confused. No one at Grafana thinks this is a big problem and seems to ignore it. This is really annoying.

@lieberlois
Copy link

@Pamalosebi It was "solved" by upgrading to Grafana 11 - however i still think having the possiblity to manually index the "level" field (in OTLP this is indexed as severity_text) would be valuable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants