Error querying Prometheus: bad_data: cannot parse "" to a valid timestamp

[kayrosik]

This is part of my function QueryRange:

	start := time.Now()
	end := now.Add(time.Duration(-config.Prometheus.MetricsTimeRange) * time.Minute)
        resolution := math.Max(math.Floor(end.Sub(start).Seconds()/250), 1)
	step := time.Duration(resolution * 1e9)

	v1api := v1.NewAPI(client)
	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
	defer cancel()
	r := v1.Range{
		Start: start,
		End:   end,
		Step:  step,
	}
	result, warnings, err := v1api.QueryRange(ctx, metric, r)

I use trickster + prometheus(Thanos).
When I try to call the function, I get:
Error querying Prometheus: bad_data: cannot parse "" to a valid timestamp

And I can't understand why it can be (

I change DoGetFallback function (always HTTP-Get):

	resp, body, warnings, err := h.Do(ctx, req)
	u.RawQuery = args.Encode()
	req, err = http.NewRequest(http.MethodGet, u.String(), nil)
	if err != nil {
		return nil, nil, warnings, err
	}
	return h.Do(ctx, req)

And it works! The problem is POST-request.

[beorn7]

It's is actually recommended to use POST for the query and query-range endpoints because it avoids complications with length restrictions of the URL. (PromQL queries can become quite long.)

Maybe Trickster or Thanos don't support POST requests for those endpoints?

But even then, the HTTP client should transparently fallback to GET. Perhaps something doesn't work as intended with the fallback.

My understanding is limited here. This was implemented back in #557 by @jacksontj . Or perhaps @lilic @bwplotka or @joe-elliott would have an idea?

[joe-elliott]

DoGetFallback relies on the server returning a 405. @kayrosik can you confirm what status code the POST is returning? My guess is that Error querying Prometheus: bad_data: cannot parse "" to a valid timestamp is a 400.

Also guessing that error is returned by Trickster based on the text, but it would be nice to confirm.

[kayrosik]

Status-code is 400

[beorn7]

OK, that explains it then. Thanks @joe-elliott for the catch.

@kayrosik I recommend to file an issue with Trickster to either support POST (as it is recommended anyway) or return a 405 so that the client will properly retry.