You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
After querying Loki datasource with Grafana (6.2.5) with a large term query, all ingestor pods die with a panic message. Also the querier component (only one of them) used by Grafana gets a panic but does not die.
To Reproduce
Steps to reproduce the behavior:
Started loki (consul 3 pods, ingester 5 pods, querier 3 pods, distributor 3 pods) (sha256@ 4540e21bed15d2a4fa4e7dadb3eb94da64d2236c5e49f1c80871c7e41b09d4ae, git commit da6a133)
Started fluentd plugin from loki repository (v0.1.0) as daemonset, writing rate 3,4TB a week.
Query: {env="dev3"}
Expected behavior
Grafana would show some part of the result (1000) lines. Instead it fails with a msg "Unknown error during query transaction. Please check JS console logs." and all ingester pods are restarting.
Environment:
Infrastructure: Kubenetes v1.12.10
Deployment tool: Helm v2.12.3
Screenshots, promtail config, or terminal output
Error message as shown by kubetail (all ingestors):
Describe the bug
After querying Loki datasource with Grafana (6.2.5) with a large term query, all ingestor pods die with a panic message. Also the querier component (only one of them) used by Grafana gets a panic but does not die.
To Reproduce
Steps to reproduce the behavior:
{env="dev3"}
Expected behavior
Grafana would show some part of the result (1000) lines. Instead it fails with a msg "Unknown error during query transaction. Please check JS console logs." and all ingester pods are restarting.
Environment:
Screenshots, promtail config, or terminal output
Error message as shown by kubetail (all ingestors):
Loki (ingestor pods) are started with command:
loki.yaml as seen inside the pods:
Ingester resources in K8s:
Querier component panic (kubectl logs):
The text was updated successfully, but these errors were encountered: