Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

panic in master #1793

Closed
Dieterbe opened this issue Apr 21, 2020 · 11 comments
Closed

panic in master #1793

Dieterbe opened this issue Apr 21, 2020 · 11 comments
Assignees
@robert-milan
Labels
bug
Milestone
sprint-12

Comments

@Dieterbe
Copy link
Contributor

Dieterbe commented Apr 21, 2020 •
edited by robert-milan
Loading

version : v0.13.1-655-gd950ad98
looking at our internal metrictank-render-request-proxying dashboard for worldping with timerange "last 30days" I get this:

runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:513 (0x42c0b8)
/usr/local/go/src/runtime/panic.go:82 (0x42b20d)
/usr/local/go/src/runtime/signal_unix.go:390 (0x442031)
/go/src/github.com/grafana/metrictank/consolidation/consolidate.go:60 (0xa0140f)
/go/src/github.com/grafana/metrictank/expr/normalize.go:92 (0xb6de42)
/go/src/github.com/grafana/metrictank/expr/normalize.go:30 (0xb6d6fb)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:60 (0xb490d9)
/go/src/github.com/grafana/metrictank/expr/func_alias.go:28 (0xb4a9f3)
/go/src/github.com/grafana/metrictank/expr/plan.go:309 (0xb732c8)
/go/src/github.com/grafana/metrictank/api/graphite.go:837 (0xbe3677)
/go/src/github.com/grafana/metrictank/api/graphite.go:270 (0xbddb56)
/go/src/github.com/grafana/metrictank/api/routes.go:65 (0xbfd894)
/usr/local/go/src/runtime/asm_amd64.s:525 (0x45aa74)
/usr/local/go/src/reflect/value.go:447 (0x4b7b53)
/usr/local/go/src/reflect/value.go:308 (0x4b75e3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xa73553)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xa72eb9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:100 (0xbccdab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8f020)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xa9f50a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xa92343)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/api/middleware/logger.go:45 (0xaf52bf)
/usr/local/go/src/runtime/asm_amd64.s:522 (0x45a8aa) /usr/local/go/src/reflect/value.go:447
(0x4b7b53) /usr/local/go/src/reflect/value.go:308 (0x4b75e3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xa73553)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xa72eb9)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/api/middleware/tracer.go:70 (0xaf6b15)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8f020)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/api/middleware/stats.go:76 (0xaf6268)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8f020)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f09e)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xa9f50a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xa92343)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa73240)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72fab)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f180)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:187 (0xaa0717)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:294 (0xa9aa76)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:220 (0xa93661)
/usr/local/go/src/net/http/server.go:2741 (0x6f49ca)
/usr/local/go/src/net/http/server.go:1847 (0x6f0d65)
/usr/local/go/src/runtime/asm_amd64.s:1333 (0x45c440)
message:""
message:"Query error: 500 "
@Dieterbe Dieterbe added the bug label Apr 21, 2020
@Dieterbe
Copy link
Contributor Author

cleaned up the significant part:

/go/src/github.com/grafana/metrictank/consolidation/consolidate.go:60 (0xa0140f)
/go/src/github.com/grafana/metrictank/expr/normalize.go:92 (0xb6de42)
/go/src/github.com/grafana/metrictank/expr/normalize.go:30 (0xb6d6fb)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:60 (0xb490d9)

@Dieterbe Dieterbe modified the milestones: sprint-10, sprint-11 Apr 21, 2020
@robert-milan robert-milan modified the milestones: sprint-11, sprint-12 May 18, 2020
@robert-milan
Copy link
Contributor

I cannot reproduce this

@shanson7
Copy link
Collaborator

Following #1813 normalize calls will always call Consolidate with a "clean" fit. So it will not get to the problem line via this code path any more.

@robert-milan
Copy link
Contributor

Thanks @shanson7

Fixed by: #1813

@Dieterbe
Copy link
Contributor Author

Dieterbe commented May 21, 2020 •
edited by robert-milan
Loading

I still have this issue on ops, ie running v0.13.1-742-gb494854d
if you don't see it, i suggest you try a couple more times. it seems to happen intermittently. I also see it for short timeranges like 1h.
If I spend one minute refreshing and trying 1h / 30d and 90d I will get it at least once.
The below stackdump happened when i queried for 90d:

PANIC: runtime error: invalid memory address or nil pointer dereference
/usr/local/go/src/runtime/panic.go:513 (0x42c0b8)
/usr/local/go/src/runtime/panic.go:82 (0x42b20d)
/usr/local/go/src/runtime/signal_unix.go:390 (0x442031)
/go/src/github.com/grafana/metrictank/consolidation/consolidate.go:48 (0xa0123e)
/go/src/github.com/grafana/metrictank/expr/normalize.go:93 (0xb6fa17)
/go/src/github.com/grafana/metrictank/expr/normalize.go:30 (0xb6f2cb)
/go/src/github.com/grafana/metrictank/expr/func_aggregate.go:60 (0xb49689)
/go/src/github.com/grafana/metrictank/expr/func_alias.go:28 (0xb4afa3)
\/go/src/github.com/grafana/metrictank/expr/plan.go:309 (0xb74eae)
/go/src/github.com/grafana/metrictank/api/graphite.go:851 (0xbe5a9c)
/go/src/github.com/grafana/metrictank/api/graphite.go:270 (0xbdf0c3)
/go/src/github.com/grafana/metrictank/api/routes.go:65 (0xbfe834)
/usr/local/go/src/runtime/asm_amd64.s:525 (0x45aa74)
/usr/local/go/src/reflect/value.go:447 (0x4b7b53)
/usr/local/go/src/reflect/value.go:308 (0x4b75e3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xa734d3) /go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xa72e39) /go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f100)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f01e)
/go/src/github.com/grafana/metrictank/vendor/github.com/raintank/gziper/gzip.go:100 (0xbce33b)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8efa0)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa731c0)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72f2b)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f100)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f01e)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xa9f48a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xa922c3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa731c0)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72f2b)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f100)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f01e)
/go/src/github.com/grafana/metrictank/api/middleware/logger.go:45 (0xaf523f)
/usr/local/go/src/runtime/asm_amd64.s:522 (0x45a8aa)
/usr/local/go/src/reflect/value.go:447 (0x4b7b53)
/usr/local/go/src/reflect/value.go:308 (0x4b75e3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:177 (0xa734d3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:137 (0xa72e39)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f100)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f01e)
/go/src/github.com/grafana/metrictank/api/middleware/tracer.go:70 (0xaf6a95)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8efa0)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa731c0)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72f2b)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f100)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f01e)
/go/src/github.com/grafana/metrictank/api/middleware/stats.go:76 (0xaf61e8)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:79 (0xa8efa0)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa731c0)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72f2b)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f100)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:112 (0xa8f01e)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/recovery.go:161 (0xa9f48a)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/logger.go:40 (0xa922c3)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:157 (0xa731c0)
/go/src/github.com/grafana/metrictank/vendor/github.com/go-macaron/inject/inject.go:135 (0xa72f2b)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/context.go:121 (0xa8f100)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:187 (0xaa0697)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/router.go:294 (0xa9a9f6)
/go/src/github.com/grafana/metrictank/vendor/gopkg.in/macaron.v1/macaron.go:220 (0xa935e1)
/usr/local/go/src/net/http/server.go:2741 (0x6f49ca)
/usr/local/go/src/net/http/server.go:1847 (0x6f0d65)
/usr/local/go/src/runtime/asm_amd64.s:1333 (0x45c440)

@Dieterbe Dieterbe reopened this May 21, 2020
@shanson7
Copy link
Collaborator

That panic is similar but now I'm the "clean" fit path. Do you know what the query is that causes it?

@Dieterbe
Copy link
Contributor Author 

A: alias(sumSeries(perSecond(metrictank.stats.$env.*.api.request.render.proxied.counter32)), 'proxied-reqs')
C: alias(sumSeries(metrictank.stats.$env.*.api.request.render.values.rate32), 'total-reqs') (disabled through Grafana)
B: alias(diffSeries(#C,#A), "handled-reqs")

robert is on call and can look into it this week. if not i will next week.

@fkaleo
Copy link
Contributor

fkaleo commented Jun 2, 2020

Guessing that this is the same as #1814

@shanson7
Copy link
Collaborator

shanson7 commented Jun 2, 2020

Guessing that this is the same as #1814

Almost certainly as perSecond also sets the consolidator to None.

@fkaleo
Copy link
Contributor

fkaleo commented Jun 6, 2020

Fixed by #1843

@fkaleo fkaleo closed this as completed Jun 6, 2020
@Dieterbe
Copy link
Contributor Author

Dieterbe commented Jun 8, 2020

confirmed! splendid!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@robert-milan robert-milan

Labels
bug
Projects
None yet
Milestone
sprint-12
Development

No branches or pull requests
4 participants
@Dieterbe @shanson7 @fkaleo @robert-milan