runtime error: invalid memory address or nil pointer dereference

[wkshare]

Describe the bug
runtime error: invalid memory address or nil pointer dereference

To Reproduce
Steps to reproduce the behavior:

1. Tempo version 1.4.1 (branch: HEAD, revision: d3880a9)

Expected behavior

Environment:

• OS: Oracle Linux 7.8

Additional Context
Error Message:

Jul 26 14:49:03 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:03.933194864Z caller=querier.go:229 msg="failed to query 1 blocks" blockErrs="error retrieving bloom (single-tenant, 1c1cef4a-05f1-4c88-a787-5d36ad78a924): does not exist"
Jul 26 14:49:03 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:03.934937195Z caller=querier.go:229 msg="failed to query 1 blocks" blockErrs="error retrieving bloom (single-tenant, 483dc61f-e29c-46c1-8b08-16fb4d47eaf2): does not exist"
Jul 26 14:49:03 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:03.943929119Z caller=querier.go:229 msg="failed to query 1 blocks" blockErrs="error retrieving bloom (single-tenant, 819ba88b-465e-4e98-b73e-b4b93f84214b): does not exist"
Jul 26 14:49:03 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:03.952991786Z caller=querier.go:229 msg="failed to query 1 blocks" blockErrs="error retrieving bloom (single-tenant, 89856e7b-43a9-43fc-8a86-31a2b9f12ba7): does not exist"
Jul 26 14:49:03 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:03.958947491Z caller=querier.go:229 msg="failed to query 3 blocks" blockErrs="error retrieving bloom (single-tenant, be65cdc1-06f4-4034-8c22-84243834926b): does not exist; error retrieving bloom (single-tenant, bf0ca578-b181-4935-b5c6-e3f6de07eec1): does not exist; error retrieving bloom (single-tenant, b9dd5e22-9dd2-4fca-bce8-b8344018937c): does not exist"
Jul 26 14:49:03 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:03.95947957Z caller=querier.go:229 msg="failed to query 1 blocks" blockErrs="error retrieving bloom (single-tenant, a163e5f5-adc6-486d-a908-6f32c5d4ad89): does not exist"
Jul 26 14:49:03 ech-10-157-130-81 grafana_tempo[3705691]: http: panic serving 10.157.157.252:47044: runtime error: invalid memory address or nil pointer dereference
                                                          goroutine 10834 [running]:
                                                          net/http.(*conn).serve.func1()
                                                                  net/http/server.go:1802 +0xb9
                                                          panic({0x1ae7820, 0x32632c0})
                                                                  runtime/panic.go:1047 +0x266
                                                          github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1()
                                                                  github.com/opentracing-contrib/go-stdlib@v1.0.0/nethttp/server.go:150 +0x139
                                                          panic({0x1ae7820, 0x32632c0})
                                                                  runtime/panic.go:1038 +0x215
                                                          github.com/grafana/tempo/modules/frontend.newTraceByIDMiddleware.func1.1(0xc000181200)
                                                                  github.com/grafana/tempo/modules/frontend/frontend.go:169 +0x8f0
                                                          github.com/grafana/tempo/modules/frontend.RoundTripperFunc.RoundTrip(0x1dce4fd, 0x1dc20f5)
                                                                  github.com/grafana/tempo/modules/frontend/middleware.go:13 +0x1f
                                                          github.com/grafana/tempo/modules/frontend.(*handler).ServeHTTP(0xc0006f9560, {0x2259ef8, 0xc000493ab0}, 0xc000181200)
                                                                  github.com/grafana/tempo/modules/frontend/handler.go:70 +0x24e
                                                          github.com/klauspost/compress/gzhttp.NewWrapper.func1.1({0x225ef08, 0xc00066b380}, 0x2214101)
                                                                  github.com/klauspost/compress@v1.14.4/gzhttp/gzip.go:384 +0x2fc
                                                          net/http.HandlerFunc.ServeHTTP(0x2277be8, {0x225ef08, 0xc00066b380}, 0x22141e8)
                                                                  net/http/server.go:2047 +0x2f
                                                          github.com/grafana/tempo/cmd/tempo/app.glob..func1.1({0x225ef08, 0xc00066b380}, 0xc000180f00)
                                                                  github.com/grafana/tempo/cmd/tempo/app/fake_auth.go:16 +0x294
                                                          net/http.HandlerFunc.ServeHTTP(0xc000180d00, {0x225ef08, 0xc00066b380}, 0x2253e18)
                                                                  net/http/server.go:2047 +0x2f
                                                          github.com/gorilla/mux.(*Router).ServeHTTP(0xc000302fc0, {0x225ef08, 0xc00066b380}, 0xc000180c00)
                                                                  github.com/gorilla/mux@v1.8.0/mux.go:210 +0x1cf
                                                          github.com/weaveworks/common/middleware.Instrument.Wrap.func1.2({0x225ef08, 0xc00066b380})
                                                                  github.com/weaveworks/common@v0.0.0-20210913144402-035033b78a78/middleware/instrument.go:70 +0x39
                                                          github.com/felixge/httpsnoop.CaptureMetricsFn({0x225ab88, 0xc00086ad70}, 0xc000c436f0)
                                                                  github.com/felixge/httpsnoop@v1.0.2/capture_metrics.go:76 +0x212
                                                          github.com/weaveworks/common/middleware.Instrument.Wrap.func1({0x225ab88, 0xc00086ad70}, 0xc000180c00)
                                                                  github.com/weaveworks/common@v0.0.0-20210913144402-035033b78a78/middleware/instrument.go:69 +0x35b
                                                          net/http.HandlerFunc.ServeHTTP(0xc000180c00, {0x225ab88, 0xc00086ad70}, 0xc00066b290)
                                                                  net/http/server.go:2047 +0x2f
                                                          github.com/weaveworks/common/middleware.Log.Wrap.func1({0x225f328, 0xc00086ad20}, 0xc000180c00)
                                                                  github.com/weaveworks/common@v0.0.0-20210913144402-035033b78a78/middleware/logging.go:53 +0x27a
                                                          net/http.HandlerFunc.ServeHTTP(0x40d567, {0x225f328, 0xc00086ad20}, 0xc0006d0001)
                                                                  net/http/server.go:2047 +0x2f
                                                          github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5({0x225d8e8, 0xc0001ec7e0}, 0xc000180b00)
                                                                  github.com/opentracing-contrib/go-stdlib@v1.0.0/nethttp/server.go:154 +0x62d
                                                          net/http.HandlerFunc.ServeHTTP(0x0, {0x225d8e8, 0xc0001ec7e0}, 0x46942e)
                                                                  net/http/server.go:2047 +0x2f
                                                          net/http.serverHandler.ServeHTTP({0xc00066aff0}, {0x225d8e8, 0xc0001ec7e0}, 0xc000180b00)
                                                                  net/http/server.go:2879 +0x43b
                                                          net/http.(*conn).serve(0xc0001dc000, {0x2277be8, 0xc00066b260})
                                                                  net/http/server.go:1930 +0xb08
                                                          created by net/http.(*Server).Serve
                                                                  net/http/server.go:3034 +0x4e8
Jul 26 14:49:10 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:10.751771756Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Jul 26 14:49:10 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:10.758443734Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Jul 26 14:49:10 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:10.76555087Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"
Jul 26 14:49:10 ech-10-157-130-81 grafana_tempo[3705691]: level=warn ts=2022-07-26T06:49:10.77680074Z caller=compactor.go:81 msg="unable to find meta during compaction.  trying again on this block list" err="does not exist"

[annanay25]

Hi @wkshare; thanks for reporting this.

The crash you are seeing is from this line in release v1.4.1
https://github.com/grafana/tempo/blob/v1.4.1/modules/frontend/frontend.go#L169

Since then, the file has been patched and setting of the header has been moved inside the loop where resp != nil.

tempo/modules/frontend/frontend.go

Lines 165 to 167 in 7791a36

	if resp.Header != nil {
	resp.Header.Set(api.HeaderContentType, marshallingFormat)
	}

We expect to cut 1.5 release in the next couple of weeks which will have this change (or you could use the latest image from main branch if that works for you https://hub.docker.com/r/grafana/tempo/tags)

[annanay25]

Closing this issue, feel free to reopen if you have further questions. Thanks!

[wkshare]

@annanay25 Is this problem fixed in version 1.5.0?

[wkshare]

@annanay25 ok, is saw it. #1441. thanks