Queries with stats don't work

[ykrasik]

When running any query from Grafana with a stats pipe I get got unexpected response status code: 400. The same query without the stats pipe works and returns data.

Smallest query which reproduces this error: test | stats count()

I see this error for the above query in the VM logs:

{"ts":"2024-06-22T21:14:51.057Z","level":"warn","caller":"VictoriaMetrics/app/vlselect/logsql/logsql.go:291","msg":"remoteAddr: \"10.1.11.41:52548\"; requestURI: /select/logsql/query?limit=1000&query=_time%3A%5B2024-06-22T20%3A59%3A51.041Z%2C+2024-06-22T21%3A14%3A51.041Z%5D+AND+%28test+%7C+stats+count%28%29%29; cannot parse query [_time:[2024-06-22T20:59:51.041Z, 2024-06-22T21:14:51.041Z] AND (test | stats count())]: unexpected token \"|\" instead of ')'; context: [:59:51.041Z, 2024-06-22T21:14:51.041Z] AND (test |]"}

So it seems that in the presence of a stats pipe in the query, the timestamp (which I guess the plugin adds by itself) is added through an AND which makes the whole query invalid.

[dmitryk-dk]

Hi @ykrasik ! This PR should improve the behavior.
#29

Also please take a look this issue #23

[ykrasik]

Ack, thanks for the quick reply!

[dmitryk-dk]

Hi @ykrasik ! Please check the latest release it should fix your issue

[vakabayashy]

Hi @dmitryk-dk ! I have the same problem with query _time:10m _stream:{fields.app_id="test"} | status count_uniq(_msg) msgs. I updated the plugin to 0.2.2, the problem remains.

[dmitryk-dk]

Hi @dmitryk-dk ! I have the same problem with query _time:10m _stream:{fields.app_id="test"} | status count_uniq(_msg) msgs. I updated the plugin to 0.2.2, the problem remains.

What version of the victorialogs do you use?

[dmitryk-dk]

@vakabayashy i have checked the request test | stats count() and there are no any issue with it

[dmitryk-dk]

Bu i checked the request like
_time:[2024-06-22T20:59:51.041Z, 2024-06-22T21:14:51.041Z] AND (test | stats count())
and
_time:10m _stream:{fields.app_id="test"} | status count_uniq(_msg) msgs
and as far as i see we have a bug

[dmitryk-dk]

Hi @dmitryk-dk ! I have the same problem with query _time:10m _stream:{fields.app_id="test"} | status count_uniq(_msg) msgs. I updated the plugin to 0.2.2, the problem remains.

Could you please update your query to
_time:10m _stream:{fields.app_id="test"} | stats count_uniq(_msg) msgs

Please pay attention that i changed status to stats

[dmitryk-dk]

Hi @ykrasik ! You can remove _time and use only test | stats count(), and you will get the result. Time will be set from the grafana time picker. Please check this PR

Also, you can use the next expression, _time:X test | stats count(), and it should work well.

In the previous version of the victorialogs there was a bug with timestamp parsing, but in the newest version, it should be fixed
https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v0.23.0-victorialogs

[vakabayashy]

Hi @dmitryk-dk ! I have updated to the latest version of Victorialogs v0.23.0, the errors remain.
test | stats count(), same error.
Grafana: 10.4.2
In Victorialogs no errors.
Error in Grafana:
logger=context userId=1 orgId=1 uname=admin t=2024-06-26T10:59:16.516203546Z level=error msg="Internal server error" error="[plugin.downstreamError] client: failed to query data: Failed to query data: rpc error: code = Unknown desc = frame has different field lengths, field 0 is len 0 but field 2 is len 1" remote_addr=192.168.1.11 traceID= logger=context userId=1 orgId=1 uname=admin t=2024-06-26T10:59:16.516297281Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.1.11 time_ms=23 duration=23.184144ms size=116 referer="http://192.168.1.27:3000/explore?orgId=1&panes=%7B%22u7f%22%3A%7B%22datasource%22%3A%22PD775F2863313E6C7%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22datasource%22%3A%7B%22type%22%3A%22victorialogs-datasource%22%2C%22uid%22%3A%22PD775F2863313E6C7%22%7D%2C%22expr%22%3A%22_time%3A100m+_stream%3A%7Bfields.app_id%3D%5C%22test%5C%22%7D+%7C+stats+count%28%29%22%2C%22queryType%22%3A%22range%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-15m%22%2C%22to%22%3A%22now%22%7D%7D%7D&schemaVersion=1" handler=/api/ds/query status_source=downstream

[dmitryk-dk]

test | stats count()

Hi @vakabayashy ! I have tested it on my local environment, and it works as expected.
vmlogs is v0.23.0

Also, we used our data source in the production. and I have checked it there

Both data sources work as expected.

The error that you got was before the fix. Try to upgrade the data source again and check the issue.

Try to restart your Grafana server. It should update the backend part of the Grafana

[Loori-R]

@vakabayashy Make sure you have the latest version of the datasource and try in incognito mode (it could be a cache issue).

[vakabayashy]

I fixed it. With the plugins installed in the directory ./provisioning/plugins, it did not work for me. Perhaps there were no write permissions ./provisioning/plugins:/var/lib/grafana/plugins, where the plugins of the new versions were located. After I disabled the mounting of the plugin volume and entered a environment variable GF_INSTALL_PLUGINS=https://github.com/VictoriaMetrics/victorialogs-datasource/releases/download/v0.2.2/victorialogs-datasource-v0.2.2.zip;victorialogs-datasource,https://github.com/VictoriaMetrics/grafana-datasource/releases/download/v0.8.2/victoriametrics-datasource-v0.8.2.zip;victoriametrics-datasource, after that stats worked correctly

[dmitryk-dk]

Hi @ykrasik , @vakabayashy ! Can we close the issue?

[Ridys]

Fixed in 0.2.2, thanks!

[ykrasik]

Still doesn't work with VL 0.24 and grafana datasource 0.2.2 for me. I'm checking some of the comments in this thread to see if I'm doing something wrong.

VL no longer shows an error for queries with stats, but I see this error in Grafana logs (redacted ip and url):

logger=context userId=2 orgId=1 uname=ykrasik t=2024-06-28T10:25:27.981320516Z level=error msg="Internal server error" error="[plugin.downstreamError] client: failed to query data: Failed to query data: rpc error: code │
│  = Unknown desc = frame has different field lengths, field 0 is len 41 but field 1 is len 0" remote_addr=xxx traceID=                                                                                              │
│ logger=context userId=2 orgId=1 uname=ykrasik t=2024-06-28T10:25:27.981871404Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=xxx time_ms=74 duration=74.114339ms size= │
│ 116 referer="https://grafana.xxx/d/xxx/logs?editPanel=2&from=now-3h&orgId=1&to=now&var-app=All&var-free_text=&var-lead_id=&var-level=info&var-level=warn&var-level=error&var- │
│ level=critical" handler=/api/ds/query

The above happens for any query with | stats that tries to return any data. I confirmed in the vm-ui that the same query works and returns expected records. If I select a filter that does not return any data and pipe it into stats, I get an empty result back without an error.

[ykrasik]

I see - it's #34

Just updated to 0.2.3 and now everything works. Thanks!