Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

end timestamp must not be before or equal to start time (Loki) #3554

Closed
loganmc10 opened this issue Mar 18, 2021 · 6 comments
Closed

end timestamp must not be before or equal to start time (Loki) #3554

loganmc10 opened this issue Mar 18, 2021 · 6 comments

Comments

@loganmc10
Copy link

What happened:
Sometimes (not always) my time series graph will fail to load with the error shown in the screenshot.

What you expected to happen:
Graph should show data

How to reproduce it (as minimally and precisely as possible):
Not exactly sure, this seems to be new.

Anything else we need to know?:
This is using the new time series panel

image

Environment:

  • Grafana version: 7.4.5
  • Data source type & version: Loki, Grafana Cloud
  • OS Grafana is installed on: Grafana Cloud
  • User OS & Browser: Windows, Chrome 89
  • Grafana plugins: N/A
  • Others:
@loganmc10
Copy link
Author

When it works (if I refresh a few times), it looks like this:
image

@ivanahuckova ivanahuckova self-assigned this Mar 29, 2021
@ivanahuckova
Copy link
Member

Hello @loganmc10m, what Loki version are you using? Could be related to #3286. Also I remember that @dannykopping had the same issue. Have you found a reason for these errors?

@loganmc10
Copy link
Author

I'm using Grafana Cloud so I'm not exactly sure what version of Loki it is, some recent version though I assume.

I haven't figured out a reliable way to reproduce it, but it's still happening. Refreshing the dashboard a few times makes it go away

@ivanahuckova
Copy link
Member

ivanahuckova commented Mar 29, 2021

@cyriltovena I am passing this to Loki repo as I think this is not related to Grafana. It is possible duplicate of #3286, but I would rather leave that to you to determine.

Also, couple weeks ago we were debugging the same issue with Danny. We were getting error for queries that went past 7 days. What was interesting - the metrics queries for that time range failed, but log queries for the exactly same time range worked.

Also, if you think this is not Loki issue, feel free to transfer back and I'll try to look at it again.

@ivanahuckova ivanahuckova transferred this issue from grafana/grafana Mar 29, 2021
@ivanahuckova ivanahuckova removed their assignment Mar 29, 2021
@darioef
Copy link

darioef commented Mar 30, 2021

I'm getting the same error. It's not related to which panel you're using. It happened to me with Stats, Diagram, Table, etc.

I found that it doesn't happen if you avoid Query Frontend and do the query directly to the Querier but I can't find which setting is the resposible of this behaviour.

"Max Interval" in Grafana and "split_queries_by_interval" in Loki have something to do with this but I can't find how.

Loki version: 2.2.0 running on Docker Swarm (own deployment).

@cyriltovena
Copy link
Contributor

This has been fixed by #3540 thanks for the report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants