Prometheus query_range 'start' time

[fxmiii]

What happened: Bar chart where each bar is intended to be a daily total are offset by my timezone. I'm EST and so it all my bars say 20:00. The query inspector shows that it is indeed sending a timestamp for UTC midnight.

What you expected to happen: value sent in query_range for 'start (and 'end') to be determined by the dashboard 'From' time

How to reproduce it (as minimally and precisely as possible): set "Min Step" to 24h and run a query like "increase(number_of_counts[24h])"

Anything else we need to know?: I don't know exactly how grafana determines what to choose for those 'start' and 'end' timestamps. It does make sense that maybe they should be smarter than just the 'From' and 'To' times of the dashboard but it seems clear to me that this isn't quite right. It seems like those 'start' and 'end' are set to be multiples of the step time (24h)? And because the 'start' time is sent to prometheus in UTC I get stuck with my timezone offset in the query?

Environment:

• Grafana version: 6.2.0
• Data source type & version: prometheus 2.9
• OS Grafana is installed on: docker on ubuntu
• User OS & Browser: Firefox on Win10
• Grafana plugins:
• Others:

[torkelo]

What is your dashboard timezone setting ?

[fxmiii]

I am EST. America\New York (-4).

The timezone settings have always been left to 'default'. Selecting "Local Browser time" and 'default' seem to be the same.

Selecting a dashboard timezone 'UTC' basically doesn't change the query sent out to prometheus. And the visualization now shows my datapoints at midnight (00:00).

[torkelo]

Think this is an issue for Prometheus not Grafana. When I specify a time range of "Today" the time range sent to Prometheus is correct with a from (02:00 UTC+2), if I specify an absolute time range with From 00:00 it is also correct as it sends a 02:00 UTC+2 .

The "Today" and Yesterday ranges do not change when swithcing to UTC due to this bug, #10324 but that seems unrelated.

The query inspector shows that it is indeed sending a timestamp for UTC midnight.

You mean the time range or the response? The time range is always sent in UTC second epocs, but when converted to your local time should match what you expect.

It seems like those 'start' and 'end' are set to be multiples of the step time (24h)? And because the 'start' time is sent to prometheus in UTC I get stuck with my timezone offset in the query?

Yes, start & end needs to be multiples of the step to have Prometheus return stable results.

I can replicate your issue and I think it's the 24h interval in Prometheus that is not timezone aware. Grafana sends the correct time range, but Prometheus 24h will be relative to that I think

[fxmiii]

I think I am not clear enough. I'll try some examples.

When running a query using the quick range "previous month" I get this query:
"...&start=1554076800&end=1556668800&step=86400" I get this both when selected "Local browser time" or "UTC".

When "Local browser time" is selected, I believe it should it be "start=1554091200" to send my local time zone midnight (I am -4).

What does change is just the way grafana displays the data. It correctly shows:
In local timezone, On the graph it shows each data point at 20:00 for each day
In UTC is shows each data point at 00:00.

Example using absolute time in my local timezone. [05May2019 00:00:00 to 12May2019 00:00:00] gives the query:
"...&start=1557014400&end=1557619200&step=86400"
That start timestamp is 04May2019 20:00:00 (GMT: 04May2019 00:00:00). Again, shouldn't it be "start=1557028800"?

What also seems wrong is the behavior as I shift that windows by hours:
If I try an absolute window of [05May2019 01:00:00 to 12May2019 01:00:00] the query does not change from what is shown above. Its the same with '02:00:00', '03:00:00'... through [05May2019 19:00:00 to 12May2019 19:00:00]. But at hour 20 [05May2019 20:00:00 to 12May2019 20:00:00] the query finally changes to "...&start=1557100800&end=1557705600&step=86400". My local hour 20 is UTC midnight, remember.

[fxmiii]

You mean the time range or the response? The time range is always sent in UTC second epocs, but when converted to your local time should match what you expect.

I am aware the query is sent in UTC. And grafana seems to always correctly display the results I would expect from the query it sent.

Yes, start & end needs to be multiples of the step to have Prometheus return stable results.

If this is true I guess that would explain everything. It really seems like a huge bug on prometheus's side if it can't "return stable results" for a query like:
query_range?query=increase(somecounts[24h])&start=1556683200&end=1557460800&step86400
Note: the 'start' timestamp is 01May2019 00:00:00 GMT-04.

But it does seem like it can, and should. Though as I showed with the absolute ranges above I can't really test it from grafana.

[torkelo]

Strange unable to replicate.

if I specify a dashboard time range that starts may 26 01:00:00 Grafana sends this as the start param
1558825200

Which is

new Date(1558825200*1000);
Sun May 26 2019 01:00:00 GMT+0200 (Central European Summer Time)

So it correctly sends the start time for me.

But I can replicate your problem with 24h min step now. Must be something with the step alignment
https://github.com/grafana/grafana/blob/master/public/app/plugins/datasource/prometheus/datasource.ts#L571

[fxmiii]

That time stamp 1558825200 is with a 24h min step? Is the the same if you take that query back in time to like 01May?

[torkelo]

Problem is that the step alignment is done on the UTC epoch seconds

[fxmiii]

Right. Does that need to be so?

[fxmiii]

like I was trying to express before

I don't know exactly how grafana determines what to choose for those 'start' and 'end' timestamps. It does make sense that maybe they should be smarter than just the 'From' and 'To' times of the dashboard but it seems clear to me that this isn't quite right.

I can see that maybe sliding the "step alignment" along with dashboard 'To' and 'From' times may not be optimal but I really should be able to slide it exactly 4 hours to return data aligned with my local timezone 24hr day. Right?

[fxmiii]

I am really not a programmer especially not one qualified to work on something like grafana. So feel free to dismiss all this but here is my attempt:

I think the issue is from this code on line 571 in public/app/plugins/datasource/prometheus/datasource.ts

/**
 * Align query range to step.
 * Rounds start and end down to a multiple of step.
 * @param start Timestamp marking the beginning of the range.
 * @param end Timestamp marking the end of the range.
 * @param step Interval to align start and end with.
 */
export function alignRange(start: number, end: number, step: number): { end: number; start: number } {
  const alignedEnd = Math.floor(end / step) * step;
  const alignedStart = Math.floor(start / step) * step;
  return {
    end: alignedEnd,
    start: alignedStart,
  };
}

The comment where this function is called (line 253) is

// Align query interval with step to allow query caching and to ensure
// that about-same-time query results look the same.

I admit I do not know what ill effects comes from sending a query with a non-exact number of steps between 'start' and 'end' but my first thought was maybe the fucntion should be

export function alignRange(start: number, end: number, step: number): { end: number; start: number } {  
  const alignedEnd = Math.floor((end - start) / step) * step + start;
  const alignedStart = start;
  return {
    end: alignedEnd,
    start: alignedStart,
  };
}

That seems right in cases of an absolute time range but seems wrong with at least some relative ranges. Like say 'last 24 hours'. Maybe in cases like that its the 'end' time that should be fixed and used to determine the 'alignedStart'

export function alignRange(start: number, end: number, step: number): { end: number; start: number } {  
  const alignedEnd = end;
  const alignedStart = end - Math.floor((end - start) / step) * step;
  return {
    end: alignedEnd,
    start: alignedStart,
  };
}

------------------------------------------------

Ironically, I think both of these break a different grafana use case I have where I want alignment based on the clock. For example, much the same as above, I have barcharts where I show total counts over time intervals (in this case 1h intervals). So my 'step time' is set to '1h' and the textarea is

increase(somecounts[1h])

Right now, I can run this with a relative time like "last 6 hours" and this gives me a bar for each hour on the hour (a 3 o'clock bar, 4 o'clock bar etc.).

Then I want something more like this for relative ranges:

export function alignRange(start: number, end: number, step: number): { end: number; start: number } {
  if(step > 3600){	//one hour
	const alignedEnd = Math.floor(end / 3600) * 3600;
  } else {
	const alignedEnd = Math.floor(end / step) * step;
  }
  const alignedStart = alignedEnd - Math.floor((end - start) / step) * step;
   
  return {
    end: alignedEnd,
    start: alignedStart,
  };
}

But this seems like a hack. So I don't know what's best. Thanks for your time.

[fxmiii]

The more I think about this the more I think there should be the ability for more user control over this alignment.

I think what would be best is to have a new field on the query view maybe called “Step Alignment”. It could take arguments of ‘auto’ (default), ‘start’, ‘end’, and a time quantity.

• ‘auto’ would yield ‘start’ in an absolute time window. ‘end’ with a relative time.
• ‘start’ would align the steps to the start of the time range.
• ‘end’ would align the steps to the end of the time range.
• Entering time quantity would shift the step alignment by that amount of time relative to selected timezone.

I have applications where I might enter ‘7h’ with a step size of ‘12h’. This would be used places where I want data by work shift. Day-shift starts at 0700 and night-shift takes over at 21:00.

Or another facility runs three shifts. My “Step Alignment” might be ‘5h’ and step-size is ‘8h’. First shift is 5-13, 2nd shift is 13-21, and 3rd shift is 21-5.

Yet another case is where I want a single data point per week. Some people want the week to start on Sunday, some on Monday. I could enter ‘2d’ into “Step Alignment” to make the necessary shift.

Code might be something like:

const alignedEnd = Math.floor((end - stepAlignment + timeZoneOffset*3600) / step) * step - timeZoneOffset*3600 + stepAlignment;
const alignedStart = Math.floor((start - stepAlignment + timeZoneOffset*3600) / step) * step - timeZoneOffset*3600 + stepAlignment;