Loki: Return an __error_details__ label for any line which incurs a __error__ while being processed

[slim-bean]

What this PR does / why we need it:

Currently when Loki process a line, there are a numbe of reasons the processing won't go as expected, when this happens Loki sets an __error__ label on each line that had issues, the value of this __error__ label is one of several values e.g. JSONParserErr or TemplateFormatErr

Keeping the value of __error__ to a simple set of values gives users granularity over removing only certain log lines with errors from a result, e.g. __error__ != "TemplateFormatErr"

However currently it's often difficult to understand why there was an error, rather than modify any existing behavior this PR introduces an additiona label __error_details__ which contains more verbose error information with each line as to why the __error__ label was set.

For example, perhaps you were trying to use a template function in your query line_format "{{replace \"test\" .foo }}"

This would set the "__error__": "TemplateFormatErr" in the response but doesn't give you a lot of understanding as to what went wrong.

Now in addition to that existing __error__ label, Loki would also set this label:

"__error_details__": "template: label:1:2: executing \"label\" at <replace>: wrong number of args for replace: want 3 got 2"

Checklist

• Documentation added
• Tests updated
• Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[MichelHollands]

LGTM

[slim-bean]

I think this is probably my biggest question... Is this the right name for this label??

[owen-d]

LGTM

[owen-d]

LGTM

[chaudum]

I think the function func (b *LabelsBuilder) Reset() also needs to reset the errDetails.

loki/pkg/logql/log/labels.go

Lines 132 to 137 in 942d659

	// Reset clears all current state for the builder.
	func (b *LabelsBuilder) Reset() {
	b.del = b.del[:0]
	b.add = b.add[:0]
	b.err = ""
	}

Otherwise all subsequent lines after an error have this label set.

[slim-bean]

I think the function func (b *LabelsBuilder) Reset() also needs to reset the errDetails.

loki/pkg/logql/log/labels.go

Lines 132 to 137 in 942d659

	// Reset clears all current state for the builder.
	func (b *LabelsBuilder) Reset() {
	b.del = b.del[:0]
	b.add = b.add[:0]
	b.err = ""
	}

Otherwise all subsequent lines after an error have this label set.

Great catch, fixed.

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
-        distributor	-0.3%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

[grafanabot]

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%