error decode response: invalid character 'x' in string escape code

[Proxymiity]

Hi,

When using the latest version of VictoriaLogs (0.18.0 as of today, after upgrading from 0.7.0), the datasource stops working.
A simple _time:5m or host:test-server returns a decoding error in the panel: error decode response: invalid character 'x' in string escape code

No specific error logged in the VictoriaLogs server itself, but the Grafana log has some errors:

GRAFANA  | logger=plugin.victorialogs-datasource t=2024-06-11T12:50:04.627607298Z level=error msg="error decode response: invalid character 'x' in string escape code"
GRAFANA  | logger=context userId=2 orgId=1 uname=*** t=2024-06-11T12:50:04.627862706Z level=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=*** time_ms=105 duration=105.622792ms size=139 referer="http://***/d/bdoghm58zwdmof/test-panel?editPanel=2&orgId=1" handler=/api/ds/query status_source=downstream

The query itself works fine in the vmui. Datasource is on 0.2.1, and when added replies with "This datasource is working".
Rolling back to our known working vlogs version (0.7.0) works well, however we lose new features of the query language.

[rkaracic]

Hi,

I'm having same issue as you.
Latest working version for us is 0.15, DS is on latest version (0.2.1).

Maybe this additional info will help someone:
E.g. query "instance:hostname" will not work, but if I query it like "hostname AND instance:hostname" this will work. It's ugly workaround, but maybe someone will find it useful.

[hagen1778]

@dmitryk-dk is this related to the backend of the plugin?

[dmitryk-dk]

Hi @Proxymiity ! Can you share what version of the VictoriaLogs you have? Can you share an example of your response from the VMUI you have?
This information can help us to detect the problem

[Proxymiity]

Hi,

Sure. The version that does not work is 0.18.0, but the same thing happens with 0.23.0. Here's a screenshot of the VMUI with the request:

And here's what's in the Grafana panel request:

The Grafana variable is defined as follows:

(replacing the Grafana $host variable with the actual host:xxxxxx-test does not change a thing)

[Proxymiity]

FYI, I have a working panel with that version. Some are returning "no data" because there is no data in the timerange (which is expected), one is returning data correctly, and some are erroring with the same error you'd find within the first issue comment

[dmitryk-dk]

FYI, I have a working panel with that version. Some are returning "no data" because there is no data in the timerange (which is expected), one is returning data correctly, and some are erroring with the same error you'd find within the first issue comment

It would be helpful if you could share the response in JSON and remove sensitive data when you get that error.
I will try to reproduce the issue. There are used standard JSON decoders, and I think the problem is with parsing response from the victoria logs.
But it is only my assumption

[Proxymiity]

It would be helpful if you could share the response in JSON and remove sensitive data when you get that error. I will try to reproduce the issue. There are used standard JSON decoders, and I think the problem is with parsing response from the victoria logs. But it is only my assumption

By response, do you mean intercept the response from VLogs back to Grafana?

[dmitryk-dk]

It would be helpful if you could share the response in JSON and remove sensitive data when you get that error. I will try to reproduce the issue. There are used standard JSON decoders, and I think the problem is with parsing response from the victoria logs. But it is only my assumption

By response, do you mean intercept the response from VLogs back to Grafana?

Yes, or response from vmlogs to vmui. They should be identical.

[Proxymiity]

Here are the responses from both VictoriaLogs versions (mind the dot at the end of the url):
0.7.0: https://m.iya.at/hLfWw.
0.21.0: https://m.iya.at/hHZ76.

[dmitryk-dk]

Here are the responses from both VictoriaLogs versions (mind the dot at the end of the url): 0.7.0: https://m.iya.at/hLfWw. 0.21.0: https://m.iya.at/hHZ76.

Thank you for providing examples. I will check it soon

[dmitryk-dk]

Hi @Proxymiity , I think I found the problem and made a draft fix; if you can build a source from my PR and test it, that would be great. I will check the fix on my own

[Proxymiity]

Hi @Proxymiity , I think I found the problem and made a draft fix; if you can build a source from my PR and test it, that would be great. I will check the fix on my own

Hi, thanks for the quick draft. I will look into building the plugin and will keep you informed.

[Proxymiity]

Hi, just compiled and tested your fix. It does not error out anymore, but instead the color has vanished.

Here's vlogs 0.27.0 and your patch:

Here's vlogs 0.7.0 and your patch:

[dmitryk-dk]

@Proxymiity, the logs sent from vlogs only proxied to the UI part of the data source, I will think about how it is better to handle those values with colors. I think it should be handle on the UI part of the plugin @Loori-R , @hagen1778 wdyt?

@Proxymiity, can you also share the Grafana version that you are using?

[valyala]

This issue must be fixed in VictoriaLogs v0.27.1. The bugfix is at VictoriaMetrics/VictoriaMetrics@c0caa69 .

@Proxymiity , @rkaracic , could you upgrade VictoriaLogs to v0.27.1 and verify whether the issue is fixed there?

[clrxbl]

This issue must be fixed in VictoriaLogs v0.27.1. The bugfix is at VictoriaMetrics/VictoriaMetrics@c0caa69 .

@Proxymiity , @rkaracic , could you upgrade VictoriaLogs to v0.27.1 and verify whether the issue is fixed there?

I've had this issue aswell but after upgrading to 0.27.1, it went away.

[rkaracic]

I can confirm that issue is gone in 0.27.1

[hagen1778]

The problem was fixed on the backend side by VictoriaLogs.
Additionally, plugin now understands unicode and ANSI chars thanks to #38.

[dmitryk-dk]

Hi! The issue was fixed by two separate improvement for VictoriaLogs itself and in the datasource.

Please check the release.
Close issue as resolved.

[Proxymiity]

Hi! Apologies for the response delay.

I can confirm it is working as expected as of the latest version of vlogs and the patch you made.

Thank you!