Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Correctly encode step when translating proto to http internally #13171

Merged
merged 1 commit into from
Jun 7, 2024

Conversation

benclive
Copy link
Contributor

@benclive benclive commented Jun 7, 2024

What this PR does / why we need it:
Fixes the encoding from a proto to a http request for QueryPatterns & DetectedFields & DetectedLabels

  • The step parameter is represented internally as a int64 number of milliseconds but the HTTP API expects either a duration string or a int number of seconds.
  • When using HTTP between internal components, we were adding step to a query string as-is so the step parameter in milliseconds was being treated as seconds, hence the API eventually returned one big aggregated datapoint instead of many smaller ones.
  • This only shows up when not using frontend: protobuf so it was tough to reproduce locally. With protobuf encoding, everything worked as expected.
  • The same bug was affecting DetectedFields too but we just didn't notice it yet.

Fixes #13155

@benclive benclive requested a review from a team as a code owner June 7, 2024 11:12
}

if request.Step != 0 {
params["step"] = []string{fmt.Sprintf("%f", float64(request.Step)/float64(1e3))}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative is to format the step as a duration string so 3000 becomes 3000ms instead of 3.000 seconds.

@@ -202,6 +202,57 @@ func Test_codec_EncodeDecodeRequest(t *testing.T) {
Step: 30 * 1e3, // step is expected in ms; default is 0 or no step
AggregateBy: "series",
}, false},
{"detected_fields", func() (*http.Request, error) {
return DefaultCodec.EncodeRequest(ctx, &DetectedFieldsRequest{
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused as to why we use a wrapped queryrange.DetectedFieldsRequest and not the raw logproto.DetectedFieldsRequest here - is this the right thing to do?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the query-frontend is messy for sure I think it's fine.

return DefaultCodec.EncodeRequest(ctx, &DetectedFieldsRequest{
logproto.DetectedFieldsRequest{
Query: `{foo="bar"}`,
Start: start,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other proto api objects use unix nanos for time, but these 3 (DetectedFields/DetectedLabels/QueryPatterns) use google.protobuf.Timestamp instead. Is there a reason for that? Should we be consistent across all our APIs and use one or the other?

Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM nice catch

@cyriltovena cyriltovena merged commit 740551b into main Jun 7, 2024
60 checks passed
@cyriltovena cyriltovena deleted the fix-step-bug-in-http-mode branch June 7, 2024 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

patterns API: always returns 1-2 data points no matter the time range
2 participants