Missing Data in Loki Query Results Despite Normal Dashboard Metrics

[luizbafilho]

Describe the bug
Data appears to be missing from the index when executing queries, despite Loki dashboards indicating that data has been continuously ingested, no errors on the write, read, or backend components. Interestingly, the data started reappearing spontaneously without any manual intervention. It's worth noting that no recent changes have been made to the Loki or Promtail configurations. I am uncertain how to proceed with debugging this issue, I have no idea where to look since everything looks like is running as supposed, seems that something bad happened to the index, but I don't know what or why.

Environment:

• K8s v1.24.16-eks-2d98532
• Loki 2.9 using the helm chart at v5.20.0
• Promtail 2.8.2 using the helm chart at 6.11.3

Screenshots, Promtail config, or terminal output
Here is the injection for the past 2 days:

Querying all apps for the past 24h

Data seems to be in S3

[luizbafilho]

This PR fixed the issue. #10585