Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

receive-distributor: SIGSEGV and workbuf is not empty #7174

Open
vCra opened this issue Feb 29, 2024 · 7 comments
Open

receive-distributor: SIGSEGV and workbuf is not empty #7174

vCra opened this issue Feb 29, 2024 · 7 comments

Comments

@vCra
Copy link

vCra commented Feb 29, 2024

Thanos, Prometheus and Golang version used: Thanos 32.4.

What happened: receive-distributor sometimes throws SIGSEGV

What you expected to happen: it doesn't

How to reproduce it (as minimally and precisely as possible): We're struggling with this - it seems sporadic. We've noticed seeing these today

Arguments:

  • receive
  • --log.level=info
  • --log.format=logfmt
  • --grpc-address=0.0.0.0:10901
  • --http-address=0.0.0.0:10902
  • --remote-write.address=0.0.0.0:19291
  • --label=replica="$(NAME)"
  • --label=receive="true"
  • --receive.hashrings-file=/var/lib/thanos-receive/hashrings.json
  • --receive.replication-factor=3
  • --remote-write.client-server-name=thanos-receive
  • --remote-write.server-tls-client-ca=/etc/tls/ca.crt
  • --remote-write.client-tls-ca=/etc/tls/ca.crt
  • --remote-write.server-tls-cert=/etc/tls/server.crt
  • --remote-write.server-tls-key=/etc/tls/server.key
  • --receive.hashrings-algorithm=ketama

Full logs to relevant components:
See comments
@vCra
Copy link
Author

vCra commented Feb 29, 2024

Logs 1 

2024-02-29 05:57:46.720Z ts=2024-02-29T05:57:46.720041559Z caller=receive.go:134 level=info component=receive mode=RouterOnly msg="running receive"
2024-02-29 05:57:46.720Z ts=2024-02-29T05:57:46.720109529Z caller=options.go:30 level=info component=receive protocol=HTTP msg="enabling server side TLS"
2024-02-29 05:57:46.721Z ts=2024-02-29T05:57:46.72132915Z caller=options.go:66 level=info component=receive protocol=HTTP msg="server TLS client verification enabled"
2024-02-29 05:57:46.721Z ts=2024-02-29T05:57:46.721554189Z caller=receive.go:741 level=info component=receive msg="no existing storage found, no data migration attempted"
2024-02-29 05:57:46.722Z ts=2024-02-29T05:57:46.72242732Z caller=options.go:26 level=info component=receive protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
2024-02-29 05:57:46.723Z ts=2024-02-29T05:57:46.723137025Z caller=receive.go:444 level=info component=receive msg="starting receiver"
2024-02-29 05:57:46.723Z ts=2024-02-29T05:57:46.72330287Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-29 05:57:46.723Z ts=2024-02-29T05:57:46.723357591Z caller=http.go:73 level=info component=receive service=http/server component=receive msg="listening for requests and metrics" address=0.0.0.0:10902
2024-02-29 05:57:46.723Z ts=2024-02-29T05:57:46.72371589Z caller=tls_config.go:274 level=info component=receive service=http/server component=receive msg="Listening on" address=[::]:10902
2024-02-29 05:57:46.723Z ts=2024-02-29T05:57:46.723759772Z caller=tls_config.go:277 level=info component=receive service=http/server component=receive msg="TLS is disabled." http2=false address=[::]:10902
2024-02-29 05:57:46.723Z ts=2024-02-29T05:57:46.723906689Z caller=receive.go:371 level=info component=receive msg="listening for StoreAPI and WritableStoreAPI gRPC" address=0.0.0.0:10901
2024-02-29 05:57:46.724Z ts=2024-02-29T05:57:46.723932484Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-29 05:57:46.724Z ts=2024-02-29T05:57:46.724041481Z caller=grpc.go:131 level=info component=receive service=gRPC/server component=receive msg="listening for serving gRPC" address=0.0.0.0:10901
2024-02-29 05:57:46.725Z ts=2024-02-29T05:57:46.725651885Z caller=handler.go:327 level=info component=receive component=receive-handler msg="Start listening for connections" address=0.0.0.0:19291
2024-02-29 05:57:46.725Z ts=2024-02-29T05:57:46.725750145Z caller=handler.go:349 level=info component=receive component=receive-handler msg="Serving HTTPS" address=0.0.0.0:19291
2024-02-29 05:57:46.729Z ts=2024-02-29T05:57:46.72891255Z caller=receive.go:535 level=info component=receive msg="Set up hashring for the given hashring config."
2024-02-29 05:57:46.729Z ts=2024-02-29T05:57:46.728953336Z caller=intrumentation.go:56 level=info component=receive msg="changing probe status" status=ready
2024-02-29 21:17:45.608Z fatal error: workbuf is not empty
2024-02-29 21:17:45.616Z 
2024-02-29 21:17:45.616Z runtime stack:
2024-02-29 21:17:45.616Z runtime.throw({0x279472e?, 0x100000004?})
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc00008be50 sp=0xc00008be20 pc=0x43907d
2024-02-29 21:17:45.616Z runtime.(*workbuf).checkempty(0x7fda79d61248?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcwork.go:344 +0x36 fp=0xc00008be70 sp=0xc00008be50 pc=0x427ef6
2024-02-29 21:17:45.616Z runtime.getempty()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcwork.go:357 +0x1e8 fp=0xc00008bed0 sp=0xc00008be70 pc=0x428108
2024-02-29 21:17:45.616Z runtime.(*gcWork).init(0xc000067738)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcwork.go:101 +0x1c fp=0xc00008bee0 sp=0xc00008bed0 pc=0x42785c
2024-02-29 21:17:45.616Z runtime.(*gcWork).tryGet(0xc000067738)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcwork.go:207 +0x2a fp=0xc00008bf10 sp=0xc00008bee0 pc=0x427bea
2024-02-29 21:17:45.616Z runtime.gcDrainN(0xc000067738, 0x10000)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcmark.go:1170 +0x105 fp=0xc00008bf40 sp=0xc00008bf10 pc=0x421105
2024-02-29 21:17:45.616Z runtime.gcAssistAlloc1(0xc0165fa340, 0xc000ea5fd0?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcmark.go:567 +0x127 fp=0xc00008bfa8 sp=0xc00008bf40 pc=0x41fc07
2024-02-29 21:17:45.616Z runtime.gcAssistAlloc.func1()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcmark.go:475 +0x25 fp=0xc00008bfc8 sp=0xc00008bfa8 pc=0x41fac5
2024-02-29 21:17:45.616Z runtime.systemstack()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0xc00008bfd0 sp=0xc00008bfc8 pc=0x46e0c9
2024-02-29 21:17:45.616Z 
2024-02-29 21:17:45.616Z goroutine 26384791 [GC assist marking]:
2024-02-29 21:17:45.616Z runtime.systemstack_switch()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/asm_amd64.s:463 fp=0xc002c2be30 sp=0xc002c2be28 pc=0x46e060
2024-02-29 21:17:45.616Z runtime.gcAssistAlloc(0xc0165fa340)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcmark.go:474 +0x1a5 fp=0xc002c2be90 sp=0xc002c2be30 pc=0x41f965
2024-02-29 21:17:45.616Z runtime.deductAssistCredit(0xc0021a800b?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/malloc.go:1217 +0x5a fp=0xc002c2beb8 sp=0xc002c2be90 pc=0x40e0fa
2024-02-29 21:17:45.616Z runtime.mallocgc(0x18, 0x2289100, 0x1)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/malloc.go:932 +0xd0 fp=0xc002c2bf20 sp=0xc002c2beb8 pc=0x40d910
2024-02-29 21:17:45.616Z runtime.convTslice({0xc0010e4000, 0x10000, 0x10006})
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/iface.go:403 +0x4a fp=0xc002c2bf48 sp=0xc002c2bf20 pc=0x40bb6a
2024-02-29 21:17:45.616Z github.com/klauspost/compress/s2.(*Writer).write.func1()
2024-02-29 21:17:45.616Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:541 +0x327 fp=0xc002c2bfe0 sp=0xc002c2bf48 pc=0x13cdf87
2024-02-29 21:17:45.616Z runtime.goexit()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc002c2bfe8 sp=0xc002c2bfe0 pc=0x470141
2024-02-29 21:17:45.616Z created by github.com/klauspost/compress/s2.(*Writer).write
2024-02-29 21:17:45.616Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:505 +0xc9
2024-02-29 21:17:45.616Z 
2024-02-29 21:17:45.616Z goroutine 1 [chan receive, 919 minutes]:
2024-02-29 21:17:45.616Z runtime.gopark(0xc0008cfb98?, 0x40de4a?, 0x80?, 0x2d?, 0xc0008cfbb0?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0009a5b30 sp=0xc0009a5b10 pc=0x43bdd6
2024-02-29 21:17:45.616Z runtime.chanrecv(0xc0001d8540, 0xc0008cfc48, 0x1)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0009a5bc0 sp=0xc0009a5b30 pc=0x406f9d
2024-02-29 21:17:45.616Z runtime.chanrecv1(0xc000160700?, 0x9?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0009a5be8 sp=0xc0009a5bc0 pc=0x406a98
2024-02-29 21:17:45.616Z github.com/oklog/run.(*Group).Run(0xc00012fd70)
2024-02-29 21:17:45.616Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0x16b fp=0xc0009a5c68 sp=0xc0009a5be8 pc=0x62cacb
2024-02-29 21:17:45.616Z main.main()
2024-02-29 21:17:45.616Z        /app/cmd/thanos/main.go:159 +0x1725 fp=0xc0009a5f80 sp=0xc0009a5c68 pc=0x2016905
2024-02-29 21:17:45.616Z runtime.main()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc0009a5fe0 sp=0xc0009a5f80 pc=0x43b9a7
2024-02-29 21:17:45.616Z runtime.goexit()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0009a5fe8 sp=0xc0009a5fe0 pc=0x470141
2024-02-29 21:17:45.616Z 
2024-02-29 21:17:45.616Z goroutine 2 [force gc (idle), 400 minutes]:
2024-02-29 21:17:45.616Z runtime.gopark(0x243effa8e4583?, 0x0?, 0x0?, 0x0?, 0x0?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000076fb0 sp=0xc000076f90 pc=0x43bdd6
2024-02-29 21:17:45.616Z runtime.goparkunlock(...)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:387
2024-02-29 21:17:45.616Z runtime.forcegchelper()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000076fe0 sp=0xc000076fb0 pc=0x43bc10
2024-02-29 21:17:45.616Z runtime.goexit()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000076fe8 sp=0xc000076fe0 pc=0x470141
2024-02-29 21:17:45.616Z created by runtime.init.6
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:293 +0x25
2024-02-29 21:17:45.616Z 
2024-02-29 21:17:45.616Z goroutine 3 [GC sweep wait]:
2024-02-29 21:17:45.616Z runtime.gopark(0x42e1401?, 0x42e1440?, 0xc?, 0x14?, 0x1?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000077780 sp=0xc000077760 pc=0x43bdd6
2024-02-29 21:17:45.616Z runtime.goparkunlock(...)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:387
2024-02-29 21:17:45.616Z runtime.bgsweep(0x0?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000777c8 sp=0xc000077780 pc=0x42603e
2024-02-29 21:17:45.616Z runtime.gcenable.func1()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000777e0 sp=0xc0000777c8 pc=0x41b2a6
2024-02-29 21:17:45.616Z runtime.goexit()
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000777e8 sp=0xc0000777e0 pc=0x470141
2024-02-29 21:17:45.616Z created by runtime.gcenable
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgc.go:178 +0x6b
2024-02-29 21:17:45.616Z 
2024-02-29 21:17:45.616Z goroutine 4 [GC scavenge wait]:
2024-02-29 21:17:45.616Z runtime.gopark(0x259aa0636d68d?, 0xa0f19c?, 0x0?, 0x0?, 0x0?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000077f70 sp=0xc000077f50 pc=0x43bdd6
2024-02-29 21:17:45.616Z runtime.goparkunlock(...)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/proc.go:387
2024-02-29 21:17:45.616Z runtime.(*scavengerState).park(0x42ebcc0)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000077fa0 sp=0xc000077f70 pc=0x423f13
2024-02-29 21:17:45.616Z runtime.bgscavenge(0x0?)
2024-02-29 21:17:45.616Z        /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000077fc8 sp=0xc000077fa0 pc=0x424505
2024-02-29 21:17:45.617Z runtime.gcenable.func2()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000077fe0 sp=0xc000077fc8 pc=0x41b246
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x470141
2024-02-29 21:17:45.617Z created by runtime.gcenable
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:179 +0xaa
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 17 [finalizer wait, 919 minutes]:
2024-02-29 21:17:45.617Z runtime.gopark(0x0?, 0x2875d60?, 0x0?, 0x20?, 0x2000000020?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000076628 sp=0xc000076608 pc=0x43bdd6
2024-02-29 21:17:45.617Z runtime.runfinq()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000767e0 sp=0xc000076628 pc=0x41a2e7
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000767e8 sp=0xc0000767e0 pc=0x470141
2024-02-29 21:17:45.617Z created by runtime.createfing
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mfinal.go:163 +0x45
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 8 [select, 919 minutes]:
2024-02-29 21:17:45.617Z runtime.gopark(0xc000072748?, 0x2?, 0xc0?, 0xe0?, 0xc000072714?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000072590 sp=0xc000072570 pc=0x43bdd6
2024-02-29 21:17:45.617Z runtime.selectgo(0xc000072748, 0xc000072710, 0xc00007e0b8?, 0x0, 0xc0001030c8?, 0x1)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0000726d0 sp=0xc000072590 pc=0x44c4be
2024-02-29 21:17:45.617Z github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
2024-02-29 21:17:45.617Z        /go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:375 +0xb9 fp=0xc0000727e0 sp=0xc0000726d0 pc=0xec8379
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000727e8 sp=0xc0000727e0 pc=0x470141
2024-02-29 21:17:45.617Z created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
2024-02-29 21:17:45.617Z        /go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:368 +0x136
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 9 [GC worker (idle)]:
2024-02-29 21:17:45.617Z runtime.gopark(0x431efe0?, 0x1?, 0x12?, 0xea?, 0x0?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000078750 sp=0xc000078730 pc=0x43bdd6
2024-02-29 21:17:45.617Z runtime.gcBgMarkWorker()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000787e0 sp=0xc000078750 pc=0x41d011
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000787e8 sp=0xc0000787e0 pc=0x470141
2024-02-29 21:17:45.617Z created by runtime.gcBgMarkStartWorkers
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:1199 +0x25
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 10 [GC worker (active)]:
2024-02-29 21:17:45.617Z runtime.systemstack_switch()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:463 fp=0xc000078f50 sp=0xc000078f48 pc=0x46e060
2024-02-29 21:17:45.617Z runtime.gcBgMarkWorker()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:1335 +0x205 fp=0xc000078fe0 sp=0xc000078f50 pc=0x41d125
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x470141
2024-02-29 21:17:45.617Z created by runtime.gcBgMarkStartWorkers
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:1199 +0x25
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 22 [GC worker (idle)]:
2024-02-29 21:17:45.617Z runtime.gopark(0x259a9ffc37395?, 0x3?, 0xcd?, 0x2e?, 0x0?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000072f50 sp=0xc000072f30 pc=0x43bdd6
2024-02-29 21:17:45.617Z runtime.gcBgMarkWorker()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc000072fe0 sp=0xc000072f50 pc=0x41d011
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000072fe8 sp=0xc000072fe0 pc=0x470141
2024-02-29 21:17:45.617Z created by runtime.gcBgMarkStartWorkers
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:1199 +0x25
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 23 [GC worker (idle)]:
2024-02-29 21:17:45.617Z runtime.gopark(0x259aa097cc20e?, 0x3?, 0xa6?, 0xb8?, 0x0?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000073750 sp=0xc000073730 pc=0x43bdd6
2024-02-29 21:17:45.617Z runtime.gcBgMarkWorker()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000737e0 sp=0xc000073750 pc=0x41d011
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000737e8 sp=0xc0000737e0 pc=0x470141
2024-02-29 21:17:45.617Z created by runtime.gcBgMarkStartWorkers
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/mgc.go:1199 +0x25
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 11 [select]:
2024-02-29 21:17:45.617Z runtime.gopark(0xc0006adf88?, 0x3?, 0x3?, 0x0?, 0xc0006adf72?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0006addf8 sp=0xc0006addd8 pc=0x43bdd6
2024-02-29 21:17:45.617Z runtime.selectgo(0xc0006adf88, 0xc0006adf6c, 0x0?, 0x0, 0x3?, 0x1)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0006adf38 sp=0xc0006addf8 pc=0x44c4be
2024-02-29 21:17:45.617Z go.opencensus.io/stats/view.(*worker).start(0xc000154c80)
2024-02-29 21:17:45.617Z        /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xad fp=0xc0006adfc8 sp=0xc0006adf38 pc=0x10d402d
2024-02-29 21:17:45.617Z go.opencensus.io/stats/view.init.0.func1()
2024-02-29 21:17:45.617Z        /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x26 fp=0xc0006adfe0 sp=0xc0006adfc8 pc=0x10d3266
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0006adfe8 sp=0xc0006adfe0 pc=0x470141
2024-02-29 21:17:45.617Z created by go.opencensus.io/stats/view.init.0
2024-02-29 21:17:45.617Z        /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x96
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 15442 [select, 20 minutes]:
2024-02-29 21:17:45.617Z runtime.gopark(0xc000a1bf38?, 0x2?, 0xc0?, 0xee?, 0xc000a1bee4?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000a1bd60 sp=0xc000a1bd40 pc=0x43bdd6
2024-02-29 21:17:45.617Z runtime.selectgo(0xc000a1bf38, 0xc000a1bee0, 0xc00064afc0?, 0x0, 0x0?, 0x1)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000a1bea0 sp=0xc000a1bd60 pc=0x44c4be
2024-02-29 21:17:45.617Z google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00062d5e0)
2024-02-29 21:17:45.617Z        /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:77 +0xa5 fp=0xc000a1bfc8 sp=0xc000a1bea0 pc=0xfda0a5
2024-02-29 21:17:45.617Z google.golang.org/grpc.newCCBalancerWrapper.func1()
2024-02-29 21:17:45.617Z        /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:67 +0x26 fp=0xc000a1bfe0 sp=0xc000a1bfc8 pc=0xfd9fc6
2024-02-29 21:17:45.617Z runtime.goexit()
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000a1bfe8 sp=0xc000a1bfe0 pc=0x470141
2024-02-29 21:17:45.617Z created by google.golang.org/grpc.newCCBalancerWrapper
2024-02-29 21:17:45.617Z        /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:67 +0x245
2024-02-29 21:17:45.617Z 
2024-02-29 21:17:45.617Z goroutine 117 [IO wait, 919 minutes]:
2024-02-29 21:17:45.617Z runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x3?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00058fcc8 sp=0xc00058fca8 pc=0x43bdd6
2024-02-29 21:17:45.617Z runtime.netpollblock(0x485905?, 0x40482f?, 0x0?)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc00058fd00 sp=0xc00058fcc8 pc=0x4345d7
2024-02-29 21:17:45.617Z internal/poll.runtime_pollWait(0x7fda7a617f78, 0x72)
2024-02-29 21:17:45.617Z        /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc00058fd20 sp=0xc00058fd00 pc=0x46a1a9
2024-02-29 21:17:45.617Z internal/poll.(*pollDesc).wait(0xc0002042a0?, 0xc00058fed0?, 0x1)
2024-02-29 21:17:45.617Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00058fd48 sp=0xc00058fd20 pc=0x4e8ad2
2024-02-29 21:17:45.617Z internal/poll.(*pollDesc).waitRead(...)
2024-02-29 21:17:45.617Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2024-02-29 21:17:45.617Z internal/poll.(*FD).Read(0xc0002042a0, {0xc00058fed0, 0x10000, 0x10000})
2024-02-29 21:17:45.617Z        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc00058fde0 sp=0xc00058fd48 pc=0x4e9eb9
2024-02-29 21:17:45.618Z os.(*File).read(...)
2024-02-29 21:17:45.618Z        /usr/local/go/src/os/file_posix.go:31
2024-02-29 21:17:45.618Z os.(*File).Read(0xc00068a1a0, {0xc00058fed0?, 0x0?, 0x0?})
2024-02-29 21:17:45.618Z        /usr/local/go/src/os/file.go:118 +0x5e fp=0xc00058fe38 sp=0xc00058fde0 pc=0x4f5dbe
2024-02-29 21:17:45.618Z github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc00015dae0)
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:356 +0xdf fp=0xc00059ffc8 sp=0xc00058fe38 pc=0x1bcde5f
2024-02-29 21:17:45.618Z github.com/fsnotify/fsnotify.NewWatcher.func1()
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:150 +0x26 fp=0xc00059ffe0 sp=0xc00059ffc8 pc=0x1bcd266
2024-02-29 21:17:45.618Z runtime.goexit()
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00059ffe8 sp=0xc00059ffe0 pc=0x470141
2024-02-29 21:17:45.618Z created by github.com/fsnotify/fsnotify.NewWatcher
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:150 +0x1b0
2024-02-29 21:17:45.618Z 
2024-02-29 21:17:45.618Z goroutine 132 [chan receive, 919 minutes]:
2024-02-29 21:17:45.618Z runtime.gopark(0xc0000746c8?, 0x474585?, 0xc8?, 0x46?, 0x49ffe6?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000746a0 sp=0xc000074680 pc=0x43bdd6
2024-02-29 21:17:45.618Z runtime.chanrecv(0xc0007cc0c0, 0x0, 0x1)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000074730 sp=0xc0000746a0 pc=0x406f9d
2024-02-29 21:17:45.618Z runtime.chanrecv1(0xc00015c410?, 0xc0001d84e0?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc000074758 sp=0xc000074730 pc=0x406a98
2024-02-29 21:17:45.618Z main.main.func2()
2024-02-29 21:17:45.618Z        /app/cmd/thanos/main.go:119 +0x33 fp=0xc000074788 sp=0xc000074758 pc=0x2016ef3
2024-02-29 21:17:45.618Z github.com/oklog/run.(*Group).Run.func1({0xc00012fd88?, 0xc00087c060?})
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc0000747c0 sp=0xc000074788 pc=0x62cc2f
2024-02-29 21:17:45.618Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc0000747e0 sp=0xc0000747c0 pc=0x62cbce
2024-02-29 21:17:45.618Z runtime.goexit()
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000747e8 sp=0xc0000747e0 pc=0x470141
2024-02-29 21:17:45.618Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:17:45.618Z 
2024-02-29 21:17:45.618Z goroutine 133 [select, 919 minutes]:
2024-02-29 21:17:45.618Z runtime.gopark(0xc000074f20?, 0x2?, 0x4a?, 0xde?, 0xc000074eb4?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000074d38 sp=0xc000074d18 pc=0x43bdd6
2024-02-29 21:17:45.618Z runtime.selectgo(0xc000074f20, 0xc000074eb0, 0x0?, 0x0, 0x0?, 0x1)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000074e78 sp=0xc000074d38 pc=0x44c4be
2024-02-29 21:17:45.618Z github.com/thanos-io/thanos/pkg/receive.ConfigFromWatcher({0x2d0f300?, 0xc00015ddb0}, 0xc000204240, 0xc000643080)
2024-02-29 21:17:45.618Z        /app/pkg/receive/config.go:287 +0x1a9 fp=0xc000074f58 sp=0xc000074e78 pc=0x1f80d69
2024-02-29 21:17:45.618Z main.setupHashring.func1()
2024-02-29 21:17:45.618Z        /app/cmd/thanos/receive.go:483 +0x29 fp=0xc000074f88 sp=0xc000074f58 pc=0x202ee89
2024-02-29 21:17:45.618Z github.com/oklog/run.(*Group).Run.func1({0xc000772540?, 0xc00064b830?})
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc000074fc0 sp=0xc000074f88 pc=0x62cc2f
2024-02-29 21:17:45.618Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc000074fe0 sp=0xc000074fc0 pc=0x62cbce
2024-02-29 21:17:45.618Z runtime.goexit()
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x470141
2024-02-29 21:17:45.618Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:17:45.618Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:17:45.618Z 
2024-02-29 21:17:45.618Z goroutine 134 [select, 919 minutes]:
2024-02-29 21:17:45.618Z fatal error: unexpected signal during runtime execution
2024-02-29 21:17:45.618Z panic during panic
2024-02-29 21:17:45.618Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x118 pc=0x45fcfc]
2024-02-29 21:17:45.618Z 
2024-02-29 21:17:45.618Z runtime stack:
2024-02-29 21:17:45.618Z runtime.throw({0x27e0bfb?, 0x3fcaf40?})
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc00008b6d8 sp=0xc00008b6a8 pc=0x43907d
2024-02-29 21:17:45.618Z runtime.sigpanic()
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0xc00008b738 sp=0xc00008b6d8 pc=0x4503a9
2024-02-29 21:17:45.618Z runtime.gentraceback(0x3c7ed98?, 0x3fcaf40?, 0xc00008baf0?, 0xc0007c8820, 0x0, 0x0, 0x64, 0x0, 0xc00008bb10?, 0x0)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/traceback.go:258 +0x8bc fp=0xc00008ba90 sp=0xc00008b738 pc=0x45fcfc
2024-02-29 21:17:45.618Z runtime.traceback1(0xc0007c8820?, 0x43ab00?, 0x3?, 0xc0007c8820, 0x462afb?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/traceback.go:776 +0x1b6 fp=0xc00008bc50 sp=0xc00008ba90 pc=0x461df6
2024-02-29 21:17:45.618Z runtime.traceback(...)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/traceback.go:723
2024-02-29 21:17:45.618Z runtime.tracebackothers.func1(0xc0007c8820)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/traceback.go:992 +0xe5 fp=0xc00008bc90 sp=0xc00008bc50 pc=0x462d25
2024-02-29 21:17:45.618Z runtime.forEachGRace(0xc00008bcf8)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/proc.go:604 +0x4d fp=0xc00008bcc0 sp=0xc00008bc90 pc=0x43c90d
2024-02-29 21:17:45.618Z runtime.tracebackothers(0xc000496d00?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/traceback.go:978 +0xe5 fp=0xc00008bd28 sp=0xc00008bcc0 pc=0x462c05
2024-02-29 21:17:45.618Z runtime.dopanic_m(0xc000496d00, 0x2cdf6e8?, 0x1?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/panic.go:1273 +0x285 fp=0xc00008bda0 sp=0xc00008bd28 pc=0x439a65
2024-02-29 21:17:45.618Z runtime.fatalthrow.func1()
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/panic.go:1127 +0x6e fp=0xc00008bde0 sp=0xc00008bda0 pc=0x43946e
2024-02-29 21:17:45.618Z runtime.fatalthrow(0x8be28?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/panic.go:1120 +0x6c fp=0xc00008be20 sp=0xc00008bde0 pc=0x4393cc
2024-02-29 21:17:45.618Z runtime.throw({0x279472e?, 0x100000004?})
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc00008be50 sp=0xc00008be20 pc=0x43907d
2024-02-29 21:17:45.618Z runtime.(*workbuf).checkempty(0x7fda79d61248?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/mgcwork.go:344 +0x36 fp=0xc00008be70 sp=0xc00008be50 pc=0x427ef6
2024-02-29 21:17:45.618Z runtime.getempty()
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/mgcwork.go:357 +0x1e8 fp=0xc00008bed0 sp=0xc00008be70 pc=0x428108
2024-02-29 21:17:45.618Z runtime.(*gcWork).init(0xc000067738)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/mgcwork.go:101 +0x1c fp=0xc00008bee0 sp=0xc00008bed0 pc=0x42785c
2024-02-29 21:17:45.618Z runtime.(*gcWork).tryGet(0xc000067738)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/mgcwork.go:207 +0x2a fp=0xc00008bf10 sp=0xc00008bee0 pc=0x427bea
2024-02-29 21:17:45.618Z runtime.gcDrainN(0xc000067738, 0x10000)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/mgcmark.go:1170 +0x105 fp=0xc00008bf40 sp=0xc00008bf10 pc=0x421105
2024-02-29 21:17:45.618Z runtime.gcAssistAlloc1(0xc0165fa340, 0xc000ea5fd0?)
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/mgcmark.go:567 +0x127 fp=0xc00008bfa8 sp=0xc00008bf40 pc=0x41fc07
2024-02-29 21:17:45.618Z runtime.gcAssistAlloc.func1()
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/mgcmark.go:475 +0x25 fp=0xc00008bfc8 sp=0xc00008bfa8 pc=0x41fac5
2024-02-29 21:17:45.618Z runtime.systemstack()
2024-02-29 21:17:45.618Z        /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0xc00008bfd0 sp=0xc00008bfc8 pc=0x46e0c9
Stream closed: EOF

@vCra
Copy link
Author

vCra commented Feb 29, 2024

Logs 2

Logs 2 

2024-02-29 00:51:53.165Z ts=2024-02-29T00:51:53.164909311Z caller=receive.go:134 level=info component=receive mode=RouterOnly msg="running receive"
2024-02-29 00:51:53.165Z ts=2024-02-29T00:51:53.165192302Z caller=options.go:30 level=info component=receive protocol=HTTP msg="enabling server side TLS"
2024-02-29 00:51:53.166Z ts=2024-02-29T00:51:53.166278129Z caller=options.go:66 level=info component=receive protocol=HTTP msg="server TLS client verification enabled"
2024-02-29 00:51:53.166Z ts=2024-02-29T00:51:53.16655126Z caller=receive.go:741 level=info component=receive msg="no existing storage found, no data migration attempted"
2024-02-29 00:51:53.167Z ts=2024-02-29T00:51:53.16732527Z caller=options.go:26 level=info component=receive protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
2024-02-29 00:51:53.168Z ts=2024-02-29T00:51:53.168214499Z caller=receive.go:444 level=info component=receive msg="starting receiver"
2024-02-29 00:51:53.168Z ts=2024-02-29T00:51:53.168483118Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-29 00:51:53.168Z ts=2024-02-29T00:51:53.168521851Z caller=http.go:73 level=info component=receive service=http/server component=receive msg="listening for requests and metrics" address=0.0.0.0:10902
2024-02-29 00:51:53.168Z ts=2024-02-29T00:51:53.168545629Z caller=receive.go:371 level=info component=receive msg="listening for StoreAPI and WritableStoreAPI gRPC" address=0.0.0.0:10901
2024-02-29 00:51:53.168Z ts=2024-02-29T00:51:53.16860821Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-29 00:51:53.168Z ts=2024-02-29T00:51:53.168680488Z caller=handler.go:327 level=info component=receive component=receive-handler msg="Start listening for connections" address=0.0.0.0:19291
2024-02-29 00:51:53.168Z ts=2024-02-29T00:51:53.168856958Z caller=tls_config.go:274 level=info component=receive service=http/server component=receive msg="Listening on" address=[::]:10902
2024-02-29 00:51:53.169Z ts=2024-02-29T00:51:53.169033565Z caller=tls_config.go:277 level=info component=receive service=http/server component=receive msg="TLS is disabled." http2=false address=[::]:10902
2024-02-29 00:51:53.169Z ts=2024-02-29T00:51:53.168884283Z caller=grpc.go:131 level=info component=receive service=gRPC/server component=receive msg="listening for serving gRPC" address=0.0.0.0:10901
2024-02-29 00:51:53.169Z ts=2024-02-29T00:51:53.168963767Z caller=handler.go:349 level=info component=receive component=receive-handler msg="Serving HTTPS" address=0.0.0.0:19291
2024-02-29 00:51:53.176Z ts=2024-02-29T00:51:53.175957016Z caller=receive.go:535 level=info component=receive msg="Set up hashring for the given hashring config."
2024-02-29 00:51:53.176Z ts=2024-02-29T00:51:53.176001341Z caller=intrumentation.go:56 level=info component=receive msg="changing probe status" status=ready
2024-02-29 17:47:16.014Z unexpected fault address 0x0
2024-02-29 17:47:16.014Z fatal error: fault
2024-02-29 17:47:16.024Z [signal 0x622f6173 code=0x762d746361746e6f addr=0x756f73190a270a32 pc=0x6b726f775f656372]
2024-02-29 17:47:16.024Z fatal error: unexpected signal during runtime execution
2024-02-29 17:47:16.024Z panic during panic
2024-02-29 17:47:16.024Z [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x43999b]
2024-02-29 17:47:16.024Z 
2024-02-29 17:47:16.024Z runtime stack:
2024-02-29 17:47:16.024Z runtime.throw({0x27e0bfb?, 0x43b45a?})
2024-02-29 17:47:16.024Z        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc00008feb0 sp=0xc00008fe80 pc=0x43907d
2024-02-29 17:47:16.024Z runtime.sigpanic()
2024-02-29 17:47:16.024Z        /usr/local/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0xc00008ff10 sp=0xc00008feb0 pc=0x4503a9
2024-02-29 17:47:16.024Z runtime.dopanic_m(0xc004a23ba0, 0x4390f5?, 0x276f0eb?)
2024-02-29 17:47:16.024Z        /usr/local/go/src/runtime/panic.go:1260 +0x1bb fp=0xc00008ff88 sp=0xc00008ff10 pc=0x43999b
2024-02-29 17:47:16.024Z runtime.fatalthrow.func1()
2024-02-29 17:47:16.024Z        /usr/local/go/src/runtime/panic.go:1127 +0x6e fp=0xc00008ffc8 sp=0xc00008ff88 pc=0x43946e
2024-02-29 17:47:16.024Z runtime.systemstack()
2024-02-29 17:47:16.024Z        /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0xc00008ffd0 sp=0xc00008ffc8 pc=0x46e0c9
2024-02-29 17:47:16.024Z 
2024-02-29 17:47:16.024Z goroutine 7165919078632000028 [???, locked to thread]:
2024-02-29 17:47:16.025Z fatal error: unexpected signal during runtime execution
2024-02-29 17:47:16.025Z stack trace unavailable
Stream closed: EOF

@vCra
Copy link
Author

vCra commented Feb 29, 2024

Logs 3

Logs 3 

2024-02-26 17:20:57.860Z ts=2024-02-26T17:20:57.859953274Z caller=receive.go:134 level=info component=receive mode=RouterOnly msg="running receive"
2024-02-26 17:20:57.860Z ts=2024-02-26T17:20:57.860034052Z caller=options.go:30 level=info component=receive protocol=HTTP msg="enabling server side TLS"
2024-02-26 17:20:57.860Z ts=2024-02-26T17:20:57.86053173Z caller=options.go:66 level=info component=receive protocol=HTTP msg="server TLS client verification enabled"
2024-02-26 17:20:57.860Z ts=2024-02-26T17:20:57.860623753Z caller=receive.go:741 level=info component=receive msg="no existing storage found, no data migration attempted"
2024-02-26 17:20:57.861Z ts=2024-02-26T17:20:57.861136655Z caller=options.go:26 level=info component=receive protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
2024-02-26 17:20:57.861Z ts=2024-02-26T17:20:57.861569618Z caller=receive.go:444 level=info component=receive msg="starting receiver"
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.862058682Z caller=handler.go:327 level=info component=receive component=receive-handler msg="Start listening for connections" address=0.0.0.0:19291
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.86234944Z caller=receive.go:371 level=info component=receive msg="listening for StoreAPI and WritableStoreAPI gRPC" address=0.0.0.0:10901
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.862138244Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.86238647Z caller=http.go:73 level=info component=receive service=http/server component=receive msg="listening for requests and metrics" address=0.0.0.0:10902
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.862431714Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.862612803Z caller=handler.go:349 level=info component=receive component=receive-handler msg="Serving HTTPS" address=0.0.0.0:19291
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.862712228Z caller=tls_config.go:274 level=info component=receive service=http/server component=receive msg="Listening on" address=[::]:10902
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.862734919Z caller=tls_config.go:277 level=info component=receive service=http/server component=receive msg="TLS is disabled." http2=false address=[::]:10902
2024-02-26 17:20:57.862Z ts=2024-02-26T17:20:57.862815364Z caller=grpc.go:131 level=info component=receive service=gRPC/server component=receive msg="listening for serving gRPC" address=0.0.0.0:10901
2024-02-26 17:20:57.866Z ts=2024-02-26T17:20:57.866101083Z caller=receive.go:535 level=info component=receive msg="Set up hashring for the given hashring config."
2024-02-26 17:20:57.866Z ts=2024-02-26T17:20:57.866134377Z caller=intrumentation.go:56 level=info component=receive msg="changing probe status" status=ready
2024-02-29 21:56:27.184Z unexpected fault address 0xc06b04c37d
2024-02-29 21:56:27.184Z fatal error: fault
2024-02-29 21:56:27.192Z [signal SIGSEGV: segmentation violation code=0x1 addr=0xc06b04c37d pc=0x13dbc4a]
2024-02-29 21:56:27.192Z 
2024-02-29 21:56:27.192Z goroutine 128516653 [running]:
2024-02-29 21:56:27.192Z runtime.throw({0x276f0eb?, 0x372d383362313633?})
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc003a99e30 sp=0xc003a99e00 pc=0x43907d
2024-02-29 21:56:27.192Z runtime.sigpanic()
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/signal_unix.go:855 +0x28a fp=0xc003a99e90 sp=0xc003a99e30 pc=0x45024a
2024-02-29 21:56:27.192Z github.com/klauspost/compress/s2.encodeSnappyBetterBlockAsm64K({0xc001b5400b, 0xfffb, 0xfffb}, {0xc001906008, 0xfff8, 0xfffe})
2024-02-29 21:56:27.192Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/encodeblock_amd64.s:15031 +0x12a fp=0xc003ae9eb8 sp=0xc003a99e90 pc=0x13dbc4a
2024-02-29 21:56:27.192Z github.com/klauspost/compress/s2.encodeBlockBetterSnappy({0xc001b5400b?, 0x8?, 0xc002dfeece?}, {0xc001906008?, 0xc002dfef07?, 0x7?})
2024-02-29 21:56:27.192Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/encode_amd64.go:136 +0x145 fp=0xc003ae9f00 sp=0xc003ae9eb8 pc=0x13bcc65
2024-02-29 21:56:27.192Z github.com/klauspost/compress/s2.(*Writer).encodeBlock(0xc001906008?, {0xc001b5400b?, 0x2cf49c0?, 0xc00012c458?}, {0xc001906008?, 0xc000118b90?, 0x47?})
2024-02-29 21:56:27.192Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:445 +0x105 fp=0xc003ae9f48 sp=0xc003ae9f00 pc=0x13cd4e5
2024-02-29 21:56:27.192Z github.com/klauspost/compress/s2.(*Writer).write.func1()
2024-02-29 21:56:27.192Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:514 +0x14b fp=0xc003ae9fe0 sp=0xc003ae9f48 pc=0x13cddab
2024-02-29 21:56:27.192Z runtime.goexit()
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc003ae9fe8 sp=0xc003ae9fe0 pc=0x470141
2024-02-29 21:56:27.192Z created by github.com/klauspost/compress/s2.(*Writer).write
2024-02-29 21:56:27.192Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:505 +0xc9
2024-02-29 21:56:27.192Z 
2024-02-29 21:56:27.192Z goroutine 1 [chan receive, 4595 minutes]:
2024-02-29 21:56:27.192Z runtime.gopark(0xc000afbb98?, 0x40de4a?, 0x50?, 0x10?, 0xc000afbbb0?)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000df1b30 sp=0xc000df1b10 pc=0x43bdd6
2024-02-29 21:56:27.192Z runtime.chanrecv(0xc00027f320, 0xc000afbc48, 0x1)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc000df1bc0 sp=0xc000df1b30 pc=0x406f9d
2024-02-29 21:56:27.192Z runtime.chanrecv1(0xc0005d3b00?, 0x9?)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc000df1be8 sp=0xc000df1bc0 pc=0x406a98
2024-02-29 21:56:27.192Z github.com/oklog/run.(*Group).Run(0xc000117e60)
2024-02-29 21:56:27.192Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:43 +0x16b fp=0xc000df1c68 sp=0xc000df1be8 pc=0x62cacb
2024-02-29 21:56:27.192Z main.main()
2024-02-29 21:56:27.192Z        /app/cmd/thanos/main.go:159 +0x1725 fp=0xc000df1f80 sp=0xc000df1c68 pc=0x2016905
2024-02-29 21:56:27.192Z runtime.main()
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:250 +0x207 fp=0xc000df1fe0 sp=0xc000df1f80 pc=0x43b9a7
2024-02-29 21:56:27.192Z runtime.goexit()
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000df1fe8 sp=0xc000df1fe0 pc=0x470141
2024-02-29 21:56:27.192Z 
2024-02-29 21:56:27.192Z goroutine 2 [force gc (idle), 3310 minutes]:
2024-02-29 21:56:27.192Z runtime.gopark(0x283560bc16b19?, 0x0?, 0x0?, 0x0?, 0x0?)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000076fb0 sp=0xc000076f90 pc=0x43bdd6
2024-02-29 21:56:27.192Z runtime.goparkunlock(...)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:387
2024-02-29 21:56:27.192Z runtime.forcegchelper()
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:305 +0xb0 fp=0xc000076fe0 sp=0xc000076fb0 pc=0x43bc10
2024-02-29 21:56:27.192Z runtime.goexit()
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000076fe8 sp=0xc000076fe0 pc=0x470141
2024-02-29 21:56:27.192Z created by runtime.init.6
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:293 +0x25
2024-02-29 21:56:27.192Z 
2024-02-29 21:56:27.192Z goroutine 3 [GC sweep wait]:
2024-02-29 21:56:27.192Z runtime.gopark(0x42e1401?, 0x42e1440?, 0xc?, 0x14?, 0x1?)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000077780 sp=0xc000077760 pc=0x43bdd6
2024-02-29 21:56:27.192Z runtime.goparkunlock(...)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:387
2024-02-29 21:56:27.192Z runtime.bgsweep(0x0?)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000777c8 sp=0xc000077780 pc=0x42603e
2024-02-29 21:56:27.192Z runtime.gcenable.func1()
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000777e0 sp=0xc0000777c8 pc=0x41b2a6
2024-02-29 21:56:27.192Z runtime.goexit()
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000777e8 sp=0xc0000777e0 pc=0x470141
2024-02-29 21:56:27.192Z created by runtime.gcenable
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/mgc.go:178 +0x6b
2024-02-29 21:56:27.192Z 
2024-02-29 21:56:27.192Z goroutine 4 [runnable]:
2024-02-29 21:56:27.192Z runtime.gopark(0x42ebcc0?, 0x118c89?, 0x0?, 0x0?, 0x0?)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000077f70 sp=0xc000077f50 pc=0x43bdd6
2024-02-29 21:56:27.192Z runtime.goparkunlock(...)
2024-02-29 21:56:27.192Z        /usr/local/go/src/runtime/proc.go:387
2024-02-29 21:56:27.193Z runtime.(*scavengerState).park(0x42ebcc0)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000077fa0 sp=0xc000077f70 pc=0x423f13
2024-02-29 21:56:27.193Z runtime.bgscavenge(0x0?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000077fc8 sp=0xc000077fa0 pc=0x424505
2024-02-29 21:56:27.193Z runtime.gcenable.func2()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc000077fe0 sp=0xc000077fc8 pc=0x41b246
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000077fe8 sp=0xc000077fe0 pc=0x470141
2024-02-29 21:56:27.193Z created by runtime.gcenable
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:179 +0xaa
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 5 [finalizer wait, 4595 minutes]:
2024-02-29 21:56:27.193Z runtime.gopark(0x0?, 0x2875d60?, 0x40?, 0x62?, 0x2000000020?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000076628 sp=0xc000076608 pc=0x43bdd6
2024-02-29 21:56:27.193Z runtime.runfinq()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mfinal.go:193 +0x107 fp=0xc0000767e0 sp=0xc000076628 pc=0x41a2e7
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000767e8 sp=0xc0000767e0 pc=0x470141
2024-02-29 21:56:27.193Z created by runtime.createfing
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mfinal.go:163 +0x45
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 50 [GC worker (idle)]:
2024-02-29 21:56:27.193Z runtime.gopark(0x337e25cd969d5?, 0x3?, 0x20?, 0xe0?, 0x0?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000078750 sp=0xc000078730 pc=0x43bdd6
2024-02-29 21:56:27.193Z runtime.gcBgMarkWorker()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000787e0 sp=0xc000078750 pc=0x41d011
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000787e8 sp=0xc0000787e0 pc=0x470141
2024-02-29 21:56:27.193Z created by runtime.gcBgMarkStartWorkers
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:1199 +0x25
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 9 [GC worker (idle)]:
2024-02-29 21:56:27.193Z runtime.gopark(0x337e25cd9605f?, 0x3?, 0xc3?, 0x23?, 0x0?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00086df50 sp=0xc00086df30 pc=0x43bdd6
2024-02-29 21:56:27.193Z runtime.gcBgMarkWorker()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc00086dfe0 sp=0xc00086df50 pc=0x41d011
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00086dfe8 sp=0xc00086dfe0 pc=0x470141
2024-02-29 21:56:27.193Z created by runtime.gcBgMarkStartWorkers
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:1199 +0x25
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 20 [GC worker (idle)]:
2024-02-29 21:56:27.193Z runtime.gopark(0x337e25f45af4b?, 0x1?, 0x50?, 0xf1?, 0x0?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000072750 sp=0xc000072730 pc=0x43bdd6
2024-02-29 21:56:27.193Z runtime.gcBgMarkWorker()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000727e0 sp=0xc000072750 pc=0x41d011
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000727e8 sp=0xc0000727e0 pc=0x470141
2024-02-29 21:56:27.193Z created by runtime.gcBgMarkStartWorkers
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:1199 +0x25
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 10 [GC worker (active)]:
2024-02-29 21:56:27.193Z runtime.systemstack_switch()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:463 fp=0xc000079750 sp=0xc000079748 pc=0x46e060
2024-02-29 21:56:27.193Z runtime.gcBgMarkWorker()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:1335 +0x205 fp=0xc0000797e0 sp=0xc000079750 pc=0x41d125
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000797e8 sp=0xc0000797e0 pc=0x470141
2024-02-29 21:56:27.193Z created by runtime.gcBgMarkStartWorkers
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/mgc.go:1199 +0x25
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 11 [select, 4595 minutes]:
2024-02-29 21:56:27.193Z runtime.gopark(0xc000072f48?, 0x2?, 0x0?, 0x0?, 0xc000072f14?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000072d90 sp=0xc000072d70 pc=0x43bdd6
2024-02-29 21:56:27.193Z runtime.selectgo(0xc000072f48, 0xc000072f10, 0x0?, 0x0, 0x0?, 0x1)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000072ed0 sp=0xc000072d90 pc=0x44c4be
2024-02-29 21:56:27.193Z github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1()
2024-02-29 21:56:27.193Z        /go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:375 +0xb9 fp=0xc000072fe0 sp=0xc000072ed0 pc=0xec8379
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000072fe8 sp=0xc000072fe0 pc=0x470141
2024-02-29 21:56:27.193Z created by github.com/baidubce/bce-sdk-go/util/log.NewLogger
2024-02-29 21:56:27.193Z        /go/pkg/mod/github.com/baidubce/bce-sdk-go@v0.9.111/util/log/logger.go:368 +0x136
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 12 [select]:
2024-02-29 21:56:27.193Z runtime.gopark(0xc000073788?, 0x3?, 0x9?, 0x18?, 0xc000073772?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0000735f8 sp=0xc0000735d8 pc=0x43bdd6
2024-02-29 21:56:27.193Z runtime.selectgo(0xc000073788, 0xc00007376c, 0x0?, 0x0, 0x3?, 0x1)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000073738 sp=0xc0000735f8 pc=0x44c4be
2024-02-29 21:56:27.193Z go.opencensus.io/stats/view.(*worker).start(0xc0000d1100)
2024-02-29 21:56:27.193Z        /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0xad fp=0xc0000737c8 sp=0xc000073738 pc=0x10d402d
2024-02-29 21:56:27.193Z go.opencensus.io/stats/view.init.0.func1()
2024-02-29 21:56:27.193Z        /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x26 fp=0xc0000737e0 sp=0xc0000737c8 pc=0x10d3266
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000737e8 sp=0xc0000737e0 pc=0x470141
2024-02-29 21:56:27.193Z created by go.opencensus.io/stats/view.init.0
2024-02-29 21:56:27.193Z        /go/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x96
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 127 [select, 4595 minutes]:
2024-02-29 21:56:27.193Z runtime.gopark(0xc0007cdf20?, 0x2?, 0x4a?, 0xde?, 0xc0007cdeb4?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0007cdd38 sp=0xc0007cdd18 pc=0x43bdd6
2024-02-29 21:56:27.193Z runtime.selectgo(0xc0007cdf20, 0xc0007cdeb0, 0x4?, 0x0, 0x4064f1?, 0x1)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0007cde78 sp=0xc0007cdd38 pc=0x44c4be
2024-02-29 21:56:27.193Z github.com/thanos-io/thanos/pkg/receive.ConfigFromWatcher({0x2d0f300?, 0xc0005af950}, 0xc0003051a0, 0xc0007591e0)
2024-02-29 21:56:27.193Z        /app/pkg/receive/config.go:287 +0x1a9 fp=0xc0007cdf58 sp=0xc0007cde78 pc=0x1f80d69
2024-02-29 21:56:27.193Z main.setupHashring.func1()
2024-02-29 21:56:27.193Z        /app/cmd/thanos/receive.go:483 +0x29 fp=0xc0007cdf88 sp=0xc0007cdf58 pc=0x202ee89
2024-02-29 21:56:27.193Z github.com/oklog/run.(*Group).Run.func1({0xc00022f770?, 0xc0002aa2e0?})
2024-02-29 21:56:27.193Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc0007cdfc0 sp=0xc0007cdf88 pc=0x62cc2f
2024-02-29 21:56:27.193Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.193Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc0007cdfe0 sp=0xc0007cdfc0 pc=0x62cbce
2024-02-29 21:56:27.193Z runtime.goexit()
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0007cdfe8 sp=0xc0007cdfe0 pc=0x470141
2024-02-29 21:56:27.193Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.193Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.193Z 
2024-02-29 21:56:27.193Z goroutine 38 [IO wait]:
2024-02-29 21:56:27.193Z runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000271a78 sp=0xc000271a58 pc=0x43bdd6
2024-02-29 21:56:27.193Z runtime.netpollblock(0x7f39168ac348?, 0x40482f?, 0x0?)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000271ab0 sp=0xc000271a78 pc=0x4345d7
2024-02-29 21:56:27.193Z internal/poll.runtime_pollWait(0x7f3916874eb0, 0x72)
2024-02-29 21:56:27.193Z        /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000271ad0 sp=0xc000271ab0 pc=0x46a1a9
2024-02-29 21:56:27.193Z internal/poll.(*pollDesc).wait(0xc0000d0300?, 0x4?, 0x0)
2024-02-29 21:56:27.193Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000271af8 sp=0xc000271ad0 pc=0x4e8ad2
2024-02-29 21:56:27.193Z internal/poll.(*pollDesc).waitRead(...)
2024-02-29 21:56:27.193Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2024-02-29 21:56:27.193Z internal/poll.(*FD).Accept(0xc0000d0300)
2024-02-29 21:56:27.193Z        /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000271ba0 sp=0xc000271af8 pc=0x4ee3dd
2024-02-29 21:56:27.193Z net.(*netFD).accept(0xc0000d0300)
2024-02-29 21:56:27.193Z        /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000271c58 sp=0xc000271ba0 pc=0x591675
2024-02-29 21:56:27.193Z net.(*TCPListener).accept(0xc0007340a8)
2024-02-29 21:56:27.193Z        /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000271c80 sp=0xc000271c58 pc=0x5ac525
2024-02-29 21:56:27.193Z net.(*TCPListener).Accept(0xc0007340a8)
2024-02-29 21:56:27.193Z        /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000271cb0 sp=0xc000271c80 pc=0x5ab61d
2024-02-29 21:56:27.193Z net/http.(*onceCloseListener).Accept(0xc000b963f0?)
2024-02-29 21:56:27.193Z        <autogenerated>:1 +0x2a fp=0xc000271cc8 sp=0xc000271cb0 pc=0x7e144a
2024-02-29 21:56:27.194Z net/http.(*Server).Serve(0xc0008745a0, {0x2d0c680, 0xc0007340a8})
2024-02-29 21:56:27.194Z        /usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc000271df8 sp=0xc000271cc8 pc=0x7bb4c5
2024-02-29 21:56:27.194Z github.com/prometheus/exporter-toolkit/web.Serve({0x2d0c680, 0xc0007340a8}, 0xc0008745a0, 0xc000640048, {0x2cf1b40?, 0xc0005af9f0})
2024-02-29 21:56:27.194Z        /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:278 +0x35f fp=0xc000271f38 sp=0xc000271df8 pc=0x159ea7f
2024-02-29 21:56:27.194Z github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1()
2024-02-29 21:56:27.194Z        /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:232 +0x31 fp=0xc000271f78 sp=0xc000271f38 pc=0x159e191
2024-02-29 21:56:27.194Z golang.org/x/sync/errgroup.(*Group).Go.func1()
2024-02-29 21:56:27.194Z        /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75 +0x64 fp=0xc000271fe0 sp=0xc000271f78 pc=0xa9b4a4
2024-02-29 21:56:27.194Z runtime.goexit()
2024-02-29 21:56:27.194Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000271fe8 sp=0xc000271fe0 pc=0x470141
2024-02-29 21:56:27.194Z created by golang.org/x/sync/errgroup.(*Group).Go
2024-02-29 21:56:27.194Z        /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:72 +0xa5
2024-02-29 21:56:27.194Z 
2024-02-29 21:56:27.194Z goroutine 111 [IO wait, 4595 minutes]:
2024-02-29 21:56:27.194Z runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x3?)
2024-02-29 21:56:27.194Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00060fcc8 sp=0xc00060fca8 pc=0x43bdd6
2024-02-29 21:56:27.194Z runtime.netpollblock(0x485905?, 0x40482f?, 0x0?)
2024-02-29 21:56:27.194Z        /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc00060fd00 sp=0xc00060fcc8 pc=0x4345d7
2024-02-29 21:56:27.194Z internal/poll.runtime_pollWait(0x7f3916875090, 0x72)
2024-02-29 21:56:27.194Z        /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc00060fd20 sp=0xc00060fd00 pc=0x46a1a9
2024-02-29 21:56:27.194Z internal/poll.(*pollDesc).wait(0xc000305200?, 0xc00060fed0?, 0x1)
2024-02-29 21:56:27.194Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00060fd48 sp=0xc00060fd20 pc=0x4e8ad2
2024-02-29 21:56:27.194Z internal/poll.(*pollDesc).waitRead(...)
2024-02-29 21:56:27.194Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2024-02-29 21:56:27.194Z internal/poll.(*FD).Read(0xc000305200, {0xc00060fed0, 0x10000, 0x10000})
2024-02-29 21:56:27.194Z        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc00060fde0 sp=0xc00060fd48 pc=0x4e9eb9
2024-02-29 21:56:27.194Z os.(*File).read(...)
2024-02-29 21:56:27.194Z        /usr/local/go/src/os/file_posix.go:31
2024-02-29 21:56:27.194Z os.(*File).Read(0xc00012c318, {0xc00060fed0?, 0x0?, 0x0?})
2024-02-29 21:56:27.194Z        /usr/local/go/src/os/file.go:118 +0x5e fp=0xc00060fe38 sp=0xc00060fde0 pc=0x4f5dbe
2024-02-29 21:56:27.194Z github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0005af680)
2024-02-29 21:56:27.194Z        /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:356 +0xdf fp=0xc00061ffc8 sp=0xc00060fe38 pc=0x1bcde5f
2024-02-29 21:56:27.194Z github.com/fsnotify/fsnotify.NewWatcher.func1()
2024-02-29 21:56:27.194Z        /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:150 +0x26 fp=0xc00061ffe0 sp=0xc00061ffc8 pc=0x1bcd266
2024-02-29 21:56:27.194Z runtime.goexit()
2024-02-29 21:56:27.194Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00061ffe8 sp=0xc00061ffe0 pc=0x470141
2024-02-29 21:56:27.194Z created by github.com/fsnotify/fsnotify.NewWatcher
2024-02-29 21:56:27.194Z        /go/pkg/mod/github.com/fsnotify/fsnotify@v1.6.0/backend_inotify.go:150 +0x1b0
2024-02-29 21:56:27.194Z 
2024-02-29 21:56:27.194Z goroutine 126 [chan receive, 4595 minutes]:
2024-02-29 21:56:27.194Z runtime.gopark(0xc0007d06c8?, 0x474585?, 0xc8?, 0x6?, 0x49ffe6?)
2024-02-29 21:56:27.194Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0007d06a0 sp=0xc0007d0680 pc=0x43bdd6
2024-02-29 21:56:27.194Z runtime.chanrecv(0xc0001615c0, 0x0, 0x1)
2024-02-29 21:56:27.194Z        /usr/local/go/src/runtime/chan.go:583 +0x49d fp=0xc0007d0730 sp=0xc0007d06a0 pc=0x406f9d
2024-02-29 21:56:27.195Z runtime.chanrecv1(0xc0005ae5f0?, 0xc00027f2c0?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0007d0758 sp=0xc0007d0730 pc=0x406a98
2024-02-29 21:56:27.195Z main.main.func2()
2024-02-29 21:56:27.195Z        /app/cmd/thanos/main.go:119 +0x33 fp=0xc0007d0788 sp=0xc0007d0758 pc=0x2016ef3
2024-02-29 21:56:27.195Z github.com/oklog/run.(*Group).Run.func1({0xc000117e78?, 0xc0002b48a0?})
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc0007d07c0 sp=0xc0007d0788 pc=0x62cc2f
2024-02-29 21:56:27.195Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc0007d07e0 sp=0xc0007d07c0 pc=0x62cbce
2024-02-29 21:56:27.195Z runtime.goexit()
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0007d07e8 sp=0xc0007d07e0 pc=0x470141
2024-02-29 21:56:27.195Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.195Z 
2024-02-29 21:56:27.195Z goroutine 128 [select, 4595 minutes]:
2024-02-29 21:56:27.195Z runtime.gopark(0xc000274f30?, 0x2?, 0x8?, 0x0?, 0xc000274e8c?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000274ce0 sp=0xc000274cc0 pc=0x43bdd6
2024-02-29 21:56:27.195Z runtime.selectgo(0xc000274f30, 0xc000274e88, 0x1?, 0x0, 0x1?, 0x1)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000274e20 sp=0xc000274ce0 pc=0x44c4be
2024-02-29 21:56:27.195Z main.setupHashring.func5()
2024-02-29 21:56:27.195Z        /app/cmd/thanos/receive.go:520 +0x19f fp=0xc000274f88 sp=0xc000274e20 pc=0x202e7ff
2024-02-29 21:56:27.195Z github.com/oklog/run.(*Group).Run.func1({0xc00057c380?, 0xc0002aa2f0?})
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc000274fc0 sp=0xc000274f88 pc=0x62cc2f
2024-02-29 21:56:27.195Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc000274fe0 sp=0xc000274fc0 pc=0x62cbce
2024-02-29 21:56:27.195Z runtime.goexit()
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000274fe8 sp=0xc000274fe0 pc=0x470141
2024-02-29 21:56:27.195Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.195Z 
2024-02-29 21:56:27.195Z goroutine 129 [semacquire, 4595 minutes]:
2024-02-29 21:56:27.195Z runtime.gopark(0xc00005a020?, 0x2d0f338?, 0x80?, 0xe1?, 0x27711ca?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000647cd0 sp=0xc000647cb0 pc=0x43bdd6
2024-02-29 21:56:27.195Z runtime.goparkunlock(...)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/proc.go:387
2024-02-29 21:56:27.195Z runtime.semacquire1(0xc0002fa150, 0x18?, 0x1, 0x0, 0x51?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/sema.go:160 +0x20f fp=0xc000647d38 sp=0xc000647cd0 pc=0x44d5af
2024-02-29 21:56:27.195Z sync.runtime_Semacquire(0xc0007340d8?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/sema.go:62 +0x27 fp=0xc000647d70 sp=0xc000647d38 pc=0x46bd27
2024-02-29 21:56:27.195Z sync.(*WaitGroup).Wait(0x259b960?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/sync/waitgroup.go:116 +0x4b fp=0xc000647d98 sp=0xc000647d70 pc=0x47c7eb
2024-02-29 21:56:27.195Z golang.org/x/sync/errgroup.(*Group).Wait(0xc0002fa140)
2024-02-29 21:56:27.195Z        /go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:53 +0x27 fp=0xc000647db8 sp=0xc000647d98 pc=0xa9b307
2024-02-29 21:56:27.195Z github.com/prometheus/exporter-toolkit/web.ServeMultiple({0xc000710230, 0x1, 0x7ffdcdaa228e?}, 0xc0008745a0, 0xc000640048, {0x2cf1b40?, 0xc0005af9f0})
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:235 +0x151 fp=0xc000647e08 sp=0xc000647db8 pc=0x159e0f1
2024-02-29 21:56:27.195Z github.com/prometheus/exporter-toolkit/web.ListenAndServe(0x0?, 0xc000640048, {0x2cf1b40, 0xc0005af9f0})
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.10.0/web/tls_config.go:268 +0x488 fp=0xc000647ee0 sp=0xc000647e08 pc=0x159e648
2024-02-29 21:56:27.195Z github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe(0xc00057c540)
2024-02-29 21:56:27.195Z        /app/pkg/server/http/http.go:85 +0x227 fp=0xc000647f68 sp=0xc000647ee0 pc=0x1657007
2024-02-29 21:56:27.195Z main.runReceive.func1()
2024-02-29 21:56:27.195Z        /app/cmd/thanos/receive.go:302 +0x35 fp=0xc000647f88 sp=0xc000647f68 pc=0x202da75
2024-02-29 21:56:27.195Z github.com/oklog/run.(*Group).Run.func1({0xc0000634c0?, 0xc0000634e0?})
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc000647fc0 sp=0xc000647f88 pc=0x62cc2f
2024-02-29 21:56:27.195Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc000647fe0 sp=0xc000647fc0 pc=0x62cbce
2024-02-29 21:56:27.195Z runtime.goexit()
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000647fe8 sp=0xc000647fe0 pc=0x470141
2024-02-29 21:56:27.195Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.195Z 
2024-02-29 21:56:27.195Z goroutine 130 [IO wait, 4595 minutes]:
2024-02-29 21:56:27.195Z runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000648b08 sp=0xc000648ae8 pc=0x43bdd6
2024-02-29 21:56:27.195Z runtime.netpollblock(0x0?, 0x40482f?, 0x0?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000648b40 sp=0xc000648b08 pc=0x4345d7
2024-02-29 21:56:27.195Z internal/poll.runtime_pollWait(0x7f3916874dc0, 0x72)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000648b60 sp=0xc000648b40 pc=0x46a1a9
2024-02-29 21:56:27.195Z internal/poll.(*pollDesc).wait(0xc0000d0400?, 0x416250?, 0x0)
2024-02-29 21:56:27.195Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000648b88 sp=0xc000648b60 pc=0x4e8ad2
2024-02-29 21:56:27.195Z internal/poll.(*pollDesc).waitRead(...)
2024-02-29 21:56:27.195Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2024-02-29 21:56:27.195Z internal/poll.(*FD).Accept(0xc0000d0400)
2024-02-29 21:56:27.195Z        /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000648c30 sp=0xc000648b88 pc=0x4ee3dd
2024-02-29 21:56:27.195Z net.(*netFD).accept(0xc0000d0400)
2024-02-29 21:56:27.195Z        /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000648ce8 sp=0xc000648c30 pc=0x591675
2024-02-29 21:56:27.195Z net.(*TCPListener).accept(0xc0007340f0)
2024-02-29 21:56:27.195Z        /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000648d10 sp=0xc000648ce8 pc=0x5ac525
2024-02-29 21:56:27.195Z net.(*TCPListener).Accept(0xc0007340f0)
2024-02-29 21:56:27.195Z        /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000648d40 sp=0xc000648d10 pc=0x5ab61d
2024-02-29 21:56:27.195Z google.golang.org/grpc.(*Server).Serve(0xc000787340, {0x2d0c680, 0xc0007340f0})
2024-02-29 21:56:27.195Z        /go/pkg/mod/google.golang.org/grpc@v1.45.0/server.go:779 +0x362 fp=0xc000648e58 sp=0xc000648d40 pc=0xfee1a2
2024-02-29 21:56:27.195Z github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe(0xc00024d540)
2024-02-29 21:56:27.195Z        /app/pkg/server/grpc/grpc.go:132 +0x19e fp=0xc000648ef8 sp=0xc000648e58 pc=0x1ef60fe
2024-02-29 21:56:27.195Z main.runReceive.func5()
2024-02-29 21:56:27.195Z        /app/cmd/thanos/receive.go:373 +0x165 fp=0xc000648f88 sp=0xc000648ef8 pc=0x202d285
2024-02-29 21:56:27.195Z github.com/oklog/run.(*Group).Run.func1({0xc0002866c0?, 0xc00012b100?})
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc000648fc0 sp=0xc000648f88 pc=0x62cc2f
2024-02-29 21:56:27.195Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc000648fe0 sp=0xc000648fc0 pc=0x62cbce
2024-02-29 21:56:27.195Z runtime.goexit()
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000648fe8 sp=0xc000648fe0 pc=0x470141
2024-02-29 21:56:27.195Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.195Z 
2024-02-29 21:56:27.195Z goroutine 131 [IO wait, 3306 minutes]:
2024-02-29 21:56:27.195Z runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000925850 sp=0xc000925830 pc=0x43bdd6
2024-02-29 21:56:27.195Z runtime.netpollblock(0xc0009258b0?, 0x40482f?, 0x0?)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000925888 sp=0xc000925850 pc=0x4345d7
2024-02-29 21:56:27.195Z internal/poll.runtime_pollWait(0x7f3916874fa0, 0x72)
2024-02-29 21:56:27.195Z        /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc0009258a8 sp=0xc000925888 pc=0x46a1a9
2024-02-29 21:56:27.195Z internal/poll.(*pollDesc).wait(0xc000720000?, 0x0?, 0x0)
2024-02-29 21:56:27.195Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0009258d0 sp=0xc0009258a8 pc=0x4e8ad2
2024-02-29 21:56:27.195Z internal/poll.(*pollDesc).waitRead(...)
2024-02-29 21:56:27.195Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2024-02-29 21:56:27.195Z internal/poll.(*FD).Accept(0xc000720000)
2024-02-29 21:56:27.195Z        /usr/local/go/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000925978 sp=0xc0009258d0 pc=0x4ee3dd
2024-02-29 21:56:27.195Z net.(*netFD).accept(0xc000720000)
2024-02-29 21:56:27.195Z        /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc000925a30 sp=0xc000925978 pc=0x591675
2024-02-29 21:56:27.195Z net.(*TCPListener).accept(0xc000734030)
2024-02-29 21:56:27.195Z        /usr/local/go/src/net/tcpsock_posix.go:148 +0x25 fp=0xc000925a58 sp=0xc000925a30 pc=0x5ac525
2024-02-29 21:56:27.195Z net.(*TCPListener).Accept(0xc000734030)
2024-02-29 21:56:27.195Z        /usr/local/go/src/net/tcpsock.go:297 +0x3d fp=0xc000925a88 sp=0xc000925a58 pc=0x5ab61d
2024-02-29 21:56:27.195Z github.com/mwitkow/go-conntrack.(*connTrackListener).Accept(0xc000734078)
2024-02-29 21:56:27.195Z        /go/pkg/mod/github.com/mwitkow/go-conntrack@v0.0.0-20190716064945-2f068394615f/listener_wrapper.go:100 +0x6c fp=0xc000925af0 sp=0xc000925a88 pc=0xac166c
2024-02-29 21:56:27.195Z crypto/tls.(*listener).Accept(0xc000734090)
2024-02-29 21:56:27.195Z        /usr/local/go/src/crypto/tls/tls.go:66 +0x2d fp=0xc000925b30 sp=0xc000925af0 pc=0x7361ad
2024-02-29 21:56:27.195Z net/http.(*onceCloseListener).Accept(0xc000b96630?)
2024-02-29 21:56:27.195Z        <autogenerated>:1 +0x2a fp=0xc000925b48 sp=0xc000925b30 pc=0x7e144a
2024-02-29 21:56:27.195Z net/http.(*Server).Serve(0xc000744000, {0x2d04100, 0xc000734090})
2024-02-29 21:56:27.196Z        /usr/local/go/src/net/http/server.go:3059 +0x385 fp=0xc000925c78 sp=0xc000925b48 pc=0x7bb4c5
2024-02-29 21:56:27.196Z net/http.(*Server).ServeTLS(0xc000744000, {0x2d06a10?, 0xc000734078}, {0x0, 0x0}, {0x0, 0x0})
2024-02-29 21:56:27.196Z        /usr/local/go/src/net/http/server.go:3129 +0x4bd fp=0xc000925e58 sp=0xc000925c78 pc=0x7bbe5d
2024-02-29 21:56:27.196Z github.com/thanos-io/thanos/pkg/receive.(*Handler).Run(0xc0002008c0)
2024-02-29 21:56:27.196Z        /app/pkg/receive/handler.go:351 +0x526 fp=0xc000925f58 sp=0xc000925e58 pc=0x1f836c6
2024-02-29 21:56:27.196Z main.runReceive.func7()
2024-02-29 21:56:27.196Z        /app/cmd/thanos/receive.go:388 +0x1d fp=0xc000925f88 sp=0xc000925f58 pc=0x202cf7d
2024-02-29 21:56:27.196Z github.com/oklog/run.(*Group).Run.func1({0xc00002ea70?, 0xc00002ea80?})
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc000925fc0 sp=0xc000925f88 pc=0x62cc2f
2024-02-29 21:56:27.196Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc000925fe0 sp=0xc000925fc0 pc=0x62cbce
2024-02-29 21:56:27.196Z runtime.goexit()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000925fe8 sp=0xc000925fe0 pc=0x470141
2024-02-29 21:56:27.196Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.196Z 
2024-02-29 21:56:27.196Z goroutine 132 [select, 35 minutes]:
2024-02-29 21:56:27.196Z runtime.gopark(0xc000646ed0?, 0x2?, 0x20?, 0x1d?, 0xc000646ea4?)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000646d28 sp=0xc000646d08 pc=0x43bdd6
2024-02-29 21:56:27.196Z runtime.selectgo(0xc000646ed0, 0xc000646ea0, 0xc000075ef8?, 0x0, 0xc0002a6428?, 0x1)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000646e68 sp=0xc000646d28 pc=0x44c4be
2024-02-29 21:56:27.196Z github.com/thanos-io/thanos/pkg/runutil.Repeat(0xc0002a6410?, 0xc0008180c0, 0xc000646f48)
2024-02-29 21:56:27.196Z        /app/pkg/runutil/runutil.go:77 +0xd9 fp=0xc000646f08 sp=0xc000646e68 pc=0x16a2459
2024-02-29 21:56:27.196Z main.runReceive.func11()
2024-02-29 21:56:27.196Z        /app/cmd/thanos/receive.go:417 +0xb2 fp=0xc000646f88 sp=0xc000646f08 pc=0x202ca72
2024-02-29 21:56:27.196Z github.com/oklog/run.(*Group).Run.func1({0xc000280f60?, 0xc00002eaa0?})
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc000646fc0 sp=0xc000646f88 pc=0x62cc2f
2024-02-29 21:56:27.196Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc000646fe0 sp=0xc000646fc0 pc=0x62cbce
2024-02-29 21:56:27.196Z runtime.goexit()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000646fe8 sp=0xc000646fe0 pc=0x470141
2024-02-29 21:56:27.196Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.196Z 
2024-02-29 21:56:27.196Z goroutine 133 [select, 4595 minutes]:
2024-02-29 21:56:27.196Z runtime.gopark(0xc000079ef0?, 0x2?, 0xa0?, 0x14?, 0xc000079ebc?)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000079d38 sp=0xc000079d18 pc=0x43bdd6
2024-02-29 21:56:27.196Z runtime.selectgo(0xc000079ef0, 0xc000079eb8, 0x2?, 0x0, 0x0?, 0x1)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc000079e78 sp=0xc000079d38 pc=0x44c4be
2024-02-29 21:56:27.196Z main.interrupt({0x2cf1be0, 0xc00032d140}, 0xc0001613e0)
2024-02-29 21:56:27.196Z        /app/cmd/thanos/main.go:170 +0x10a fp=0xc000079f60 sp=0xc000079e78 pc=0x20173ca
2024-02-29 21:56:27.196Z main.main.func4()
2024-02-29 21:56:27.196Z        /app/cmd/thanos/main.go:143 +0x25 fp=0xc000079f88 sp=0xc000079f60 pc=0x2016d05
2024-02-29 21:56:27.196Z github.com/oklog/run.(*Group).Run.func1({0xc00012b180?, 0xc00002eac0?})
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc000079fc0 sp=0xc000079f88 pc=0x62cc2f
2024-02-29 21:56:27.196Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc000079fe0 sp=0xc000079fc0 pc=0x62cbce
2024-02-29 21:56:27.196Z runtime.goexit()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000079fe8 sp=0xc000079fe0 pc=0x470141
2024-02-29 21:56:27.196Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.196Z 
2024-02-29 21:56:27.196Z goroutine 134 [select, 4595 minutes]:
2024-02-29 21:56:27.196Z runtime.gopark(0xc0007ccf08?, 0x2?, 0x4a?, 0xc8?, 0xc0007cceb4?)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0007ccd28 sp=0xc0007ccd08 pc=0x43bdd6
2024-02-29 21:56:27.196Z runtime.selectgo(0xc0007ccf08, 0xc0007cceb0, 0x1?, 0x0, 0x0?, 0x1)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0007cce68 sp=0xc0007ccd28 pc=0x44c4be
2024-02-29 21:56:27.196Z main.reload({0x2cf1be0, 0xc00032d140}, 0xc000161440, 0x0?)
2024-02-29 21:56:27.196Z        /app/cmd/thanos/main.go:183 +0xfe fp=0xc0007ccf58 sp=0xc0007cce68 pc=0x201769e
2024-02-29 21:56:27.196Z main.main.func6()
2024-02-29 21:56:27.196Z        /app/cmd/thanos/main.go:153 +0x29 fp=0xc0007ccf88 sp=0xc0007ccf58 pc=0x2016c89
2024-02-29 21:56:27.196Z github.com/oklog/run.(*Group).Run.func1({0xc000280fc0?, 0xc00002ead0?})
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:38 +0x2f fp=0xc0007ccfc0 sp=0xc0007ccf88 pc=0x62cc2f
2024-02-29 21:56:27.196Z github.com/oklog/run.(*Group).Run.func2()
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:39 +0x2e fp=0xc0007ccfe0 sp=0xc0007ccfc0 pc=0x62cbce
2024-02-29 21:56:27.196Z runtime.goexit()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0007ccfe8 sp=0xc0007ccfe0 pc=0x470141
2024-02-29 21:56:27.196Z created by github.com/oklog/run.(*Group).Run
2024-02-29 21:56:27.196Z        /go/pkg/mod/github.com/oklog/run@v1.1.0/group.go:37 +0x69
2024-02-29 21:56:27.196Z 
2024-02-29 21:56:27.196Z goroutine 135 [select, 4595 minutes, locked to thread]:
2024-02-29 21:56:27.196Z runtime.gopark(0xc0007cd7a8?, 0x2?, 0x90?, 0x56?, 0xc0007cd7a4?)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc0007cd618 sp=0xc0007cd5f8 pc=0x43bdd6
2024-02-29 21:56:27.196Z runtime.selectgo(0xc0007cd7a8, 0xc0007cd7a0, 0x0?, 0x0, 0x0?, 0x1)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc0007cd758 sp=0xc0007cd618 pc=0x44c4be
2024-02-29 21:56:27.196Z runtime.ensureSigM.func1()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/signal_unix.go:1004 +0x1b0 fp=0xc0007cd7e0 sp=0xc0007cd758 pc=0x466970
2024-02-29 21:56:27.196Z runtime.goexit()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0007cd7e8 sp=0xc0007cd7e0 pc=0x470141
2024-02-29 21:56:27.196Z created by runtime.ensureSigM
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/signal_unix.go:987 +0xbd
2024-02-29 21:56:27.196Z 
2024-02-29 21:56:27.196Z goroutine 136 [syscall, 4595 minutes]:
2024-02-29 21:56:27.196Z runtime.notetsleepg(0x0?, 0x0?)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc0007d1fa0 sp=0xc0007d1f68 pc=0x40c934
2024-02-29 21:56:27.196Z os/signal.signal_recv()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc0007d1fc0 sp=0xc0007d1fa0 pc=0x46c54f
2024-02-29 21:56:27.196Z os/signal.loop()
2024-02-29 21:56:27.196Z        /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc0007d1fe0 sp=0xc0007d1fc0 pc=0x62c799
2024-02-29 21:56:27.196Z runtime.goexit()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0007d1fe8 sp=0xc0007d1fe0 pc=0x470141
2024-02-29 21:56:27.196Z created by os/signal.Notify.func1.1
2024-02-29 21:56:27.196Z        /usr/local/go/src/os/signal/signal.go:151 +0x2a
2024-02-29 21:56:27.196Z 
2024-02-29 21:56:27.196Z goroutine 137 [select]:
2024-02-29 21:56:27.196Z runtime.gopark(0xc000923f58?, 0x4?, 0x4?, 0x0?, 0xc000923ed0?)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00065ad40 sp=0xc00065ad20 pc=0x43bdd6
2024-02-29 21:56:27.196Z runtime.selectgo(0xc00065af58, 0xc000923ec8, 0xc0005af950?, 0x0, 0x4?, 0x1)
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00065ae80 sp=0xc00065ad40 pc=0x44c4be
2024-02-29 21:56:27.196Z github.com/thanos-io/thanos/pkg/receive.(*ConfigWatcher).Run(0xc0007591e0, {0x2d0f300, 0xc0005af950})
2024-02-29 21:56:27.196Z        /app/pkg/receive/config.go:171 +0x19f fp=0xc00065afb8 sp=0xc00065ae80 pc=0x1f7fbff
2024-02-29 21:56:27.196Z github.com/thanos-io/thanos/pkg/receive.ConfigFromWatcher.func2()
2024-02-29 21:56:27.196Z        /app/pkg/receive/config.go:284 +0x2e fp=0xc00065afe0 sp=0xc00065afb8 pc=0x1f80f0e
2024-02-29 21:56:27.196Z runtime.goexit()
2024-02-29 21:56:27.196Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00065afe8 sp=0xc00065afe0 pc=0x470141
2024-02-29 21:56:27.196Z created by github.com/thanos-io/thanos/pkg/receive.ConfigFromWatcher
2024-02-29 21:56:27.196Z        /app/pkg/receive/config.go:284 +0x105
2024-02-29 21:56:27.197Z 
2024-02-29 21:56:27.197Z goroutine 161 [IO wait]:
2024-02-29 21:56:27.197Z runtime.gopark(0x28b?, 0xb?, 0x0?, 0x0?, 0x8?)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc000949900 sp=0xc0009498e0 pc=0x43bdd6
2024-02-29 21:56:27.197Z runtime.netpollblock(0x485905?, 0x40482f?, 0x0?)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/netpoll.go:527 +0xf7 fp=0xc000949938 sp=0xc000949900 pc=0x4345d7
2024-02-29 21:56:27.197Z internal/poll.runtime_pollWait(0x7f3916874cd0, 0x72)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/netpoll.go:306 +0x89 fp=0xc000949958 sp=0xc000949938 pc=0x46a1a9
2024-02-29 21:56:27.197Z internal/poll.(*pollDesc).wait(0xc0005f8100?, 0xc000c16000?, 0x0)
2024-02-29 21:56:27.197Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000949980 sp=0xc000949958 pc=0x4e8ad2
2024-02-29 21:56:27.197Z internal/poll.(*pollDesc).waitRead(...)
2024-02-29 21:56:27.197Z        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2024-02-29 21:56:27.197Z internal/poll.(*FD).Read(0xc0005f8100, {0xc000c16000, 0x1000, 0x1000})
2024-02-29 21:56:27.197Z        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000949a18 sp=0xc000949980 pc=0x4e9eb9
2024-02-29 21:56:27.197Z net.(*netFD).Read(0xc0005f8100, {0xc000c16000?, 0x4e8fc6?, 0x0?})
2024-02-29 21:56:27.197Z        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000949a60 sp=0xc000949a18 pc=0x58f4e9
2024-02-29 21:56:27.197Z net.(*conn).Read(0xc0005fc010, {0xc000c16000?, 0x0?, 0xc0005fa2a8?})
2024-02-29 21:56:27.197Z        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000949aa8 sp=0xc000949a60 pc=0x5a2f65
2024-02-29 21:56:27.197Z net.(*TCPConn).Read(0xc0005fa2a0?, {0xc000c16000?, 0x0?, 0xc000949ac8?})
2024-02-29 21:56:27.197Z        <autogenerated>:1 +0x29 fp=0xc000949ad8 sp=0xc000949aa8 pc=0x5b5d89
2024-02-29 21:56:27.197Z net/http.(*connReader).Read(0xc0005fa2a0, {0xc000c16000, 0x1000, 0x1000})
2024-02-29 21:56:27.197Z        /usr/local/go/src/net/http/server.go:782 +0x171 fp=0xc000949b28 sp=0xc000949ad8 pc=0x7b0ad1
2024-02-29 21:56:27.197Z bufio.(*Reader).fill(0xc000c14000)
2024-02-29 21:56:27.197Z        /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000949b60 sp=0xc000949b28 pc=0x569f7f
2024-02-29 21:56:27.197Z bufio.(*Reader).Peek(0xc000c14000, 0x4)
2024-02-29 21:56:27.197Z        /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000949b80 sp=0xc000949b60 pc=0x56a0dd
2024-02-29 21:56:27.197Z net/http.(*conn).serve(0xc0005fe240, {0x2d0f3a8, 0xc0002fc450})
2024-02-29 21:56:27.197Z        /usr/local/go/src/net/http/server.go:2030 +0x77c fp=0xc000949fb8 sp=0xc000949b80 pc=0x7b6a9c
2024-02-29 21:56:27.197Z net/http.(*Server).Serve.func3()
2024-02-29 21:56:27.197Z        /usr/local/go/src/net/http/server.go:3089 +0x2e fp=0xc000949fe0 sp=0xc000949fb8 pc=0x7bb8ae
2024-02-29 21:56:27.197Z runtime.goexit()
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000949fe8 sp=0xc000949fe0 pc=0x470141
2024-02-29 21:56:27.197Z created by net/http.(*Server).Serve
2024-02-29 21:56:27.197Z        /usr/local/go/src/net/http/server.go:3089 +0x5ed
2024-02-29 21:56:27.197Z 
2024-02-29 21:56:27.197Z goroutine 128516655 [runnable]:
2024-02-29 21:56:27.197Z github.com/klauspost/compress/s2.(*Writer).write.func1()
2024-02-29 21:56:27.197Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:505 fp=0xc0007cf7e0 sp=0xc0007cf7d8 pc=0x13cdc60
2024-02-29 21:56:27.197Z runtime.goexit()
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0007cf7e8 sp=0xc0007cf7e0 pc=0x470141
2024-02-29 21:56:27.197Z created by github.com/klauspost/compress/s2.(*Writer).write
2024-02-29 21:56:27.197Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:505 +0xc9
2024-02-29 21:56:27.197Z 
2024-02-29 21:56:27.197Z goroutine 128516667 [runnable]:
2024-02-29 21:56:27.197Z github.com/klauspost/compress/s2.(*Writer).write.func1()
2024-02-29 21:56:27.197Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:505 fp=0xc0007cc7e0 sp=0xc0007cc7d8 pc=0x13cdc60
2024-02-29 21:56:27.197Z runtime.goexit()
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0007cc7e8 sp=0xc0007cc7e0 pc=0x470141
2024-02-29 21:56:27.197Z created by github.com/klauspost/compress/s2.(*Writer).write
2024-02-29 21:56:27.197Z        /go/pkg/mod/github.com/klauspost/compress@v1.16.7/s2/writer.go:505 +0xc9
2024-02-29 21:56:27.197Z 
2024-02-29 21:56:27.197Z goroutine 37545 [select, 16 minutes]:
2024-02-29 21:56:27.197Z runtime.gopark(0xc00093bf38?, 0x2?, 0xa0?, 0xf0?, 0xc00093bee4?)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/proc.go:381 +0xd6 fp=0xc00093bd60 sp=0xc00093bd40 pc=0x43bdd6
2024-02-29 21:56:27.197Z runtime.selectgo(0xc00093bf38, 0xc00093bee0, 0xc00029a900?, 0x0, 0x0?, 0x1)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/select.go:327 +0x7be fp=0xc00093bea0 sp=0xc00093bd60 pc=0x44c4be
2024-02-29 21:56:27.197Z google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000296320)
2024-02-29 21:56:27.197Z        /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:77 +0xa5 fp=0xc00093bfc8 sp=0xc00093bea0 pc=0xfda0a5
2024-02-29 21:56:27.197Z google.golang.org/grpc.newCCBalancerWrapper.func1()
2024-02-29 21:56:27.197Z        /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:67 +0x26 fp=0xc00093bfe0 sp=0xc00093bfc8 pc=0xfd9fc6
2024-02-29 21:56:27.197Z runtime.goexit()
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00093bfe8 sp=0xc00093bfe0 pc=0x470141
2024-02-29 21:56:27.197Z created by google.golang.org/grpc.newCCBalancerWrapper
2024-02-29 21:56:27.197Z        /go/pkg/mod/google.golang.org/grpc@v1.45.0/balancer_conn_wrappers.go:67 +0x245
2024-02-29 21:56:27.197Z 
2024-02-29 21:56:27.197Z goroutine 2311734330305421332 [???, locked to thread]:
2024-02-29 21:56:27.197Z fatal error: unexpected signal during runtime execution
2024-02-29 21:56:27.197Z panic during panic
2024-02-29 21:56:27.197Z [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x461da7]
2024-02-29 21:56:27.197Z 
2024-02-29 21:56:27.197Z runtime stack:
2024-02-29 21:56:27.197Z runtime.throw({0x27e0bfb?, 0xc000089c68?})
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0xc000089c18 sp=0xc000089be8 pc=0x43907d
2024-02-29 21:56:27.197Z runtime.sigpanic()
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/signal_unix.go:825 +0x3e9 fp=0xc000089c78 sp=0xc000089c18 pc=0x4503a9
2024-02-29 21:56:27.197Z runtime.traceback1(0xc000c044e0?, 0x276c200?, 0x3?, 0xc000c044e0, 0x0?)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/traceback.go:766 +0x167 fp=0xc000089e38 sp=0xc000089c78 pc=0x461da7
2024-02-29 21:56:27.197Z runtime.traceback(...)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/traceback.go:723
2024-02-29 21:56:27.197Z runtime.tracebackothers.func1(0xc000c044e0)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/traceback.go:992 +0xe5 fp=0xc000089e78 sp=0xc000089e38 pc=0x462d25
2024-02-29 21:56:27.197Z runtime.forEachGRace(0xc000089ee0)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/proc.go:604 +0x4d fp=0xc000089ea8 sp=0xc000089e78 pc=0x43c90d
2024-02-29 21:56:27.197Z runtime.tracebackothers(0xc00097d1e0?)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/traceback.go:978 +0xe5 fp=0xc000089f10 sp=0xc000089ea8 pc=0x462c05
2024-02-29 21:56:27.197Z runtime.dopanic_m(0xc00097d1e0, 0x4390f5?, 0x276f0eb?)
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/panic.go:1273 +0x285 fp=0xc000089f88 sp=0xc000089f10 pc=0x439a65
2024-02-29 21:56:27.197Z runtime.fatalthrow.func1()
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/panic.go:1127 +0x6e fp=0xc000089fc8 sp=0xc000089f88 pc=0x43946e
2024-02-29 21:56:27.197Z runtime.systemstack()
2024-02-29 21:56:27.197Z        /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0xc000089fd0 sp=0xc000089fc8 pc=0x46e0c9
Stream closed: EOF

@vCra
Copy link
Author

vCra commented Feb 29, 2024

Logs 4 

2024-02-29 13:28:41.620Z ts=2024-02-29T13:28:41.619848492Z caller=receive.go:134 level=info component=receive mode=RouterOnly msg="running receive"
2024-02-29 13:28:41.620Z ts=2024-02-29T13:28:41.619899225Z caller=options.go:30 level=info component=receive protocol=HTTP msg="enabling server side TLS"
2024-02-29 13:28:41.620Z ts=2024-02-29T13:28:41.620373924Z caller=options.go:66 level=info component=receive protocol=HTTP msg="server TLS client verification enabled"
2024-02-29 13:28:41.620Z ts=2024-02-29T13:28:41.620543639Z caller=receive.go:741 level=info component=receive msg="no existing storage found, no data migration attempted"
2024-02-29 13:28:41.620Z ts=2024-02-29T13:28:41.620930975Z caller=options.go:26 level=info component=receive protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
2024-02-29 13:28:41.621Z ts=2024-02-29T13:28:41.621349256Z caller=receive.go:444 level=info component=receive msg="starting receiver"
2024-02-29 13:28:41.621Z ts=2024-02-29T13:28:41.621478404Z caller=receive.go:371 level=info component=receive msg="listening for StoreAPI and WritableStoreAPI gRPC" address=0.0.0.0:10901
2024-02-29 13:28:41.621Z ts=2024-02-29T13:28:41.621520001Z caller=handler.go:327 level=info component=receive component=receive-handler msg="Start listening for connections" address=0.0.0.0:19291
2024-02-29 13:28:41.621Z ts=2024-02-29T13:28:41.621756663Z caller=handler.go:349 level=info component=receive component=receive-handler msg="Serving HTTPS" address=0.0.0.0:19291
2024-02-29 13:28:41.621Z ts=2024-02-29T13:28:41.621883803Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-29 13:28:41.622Z ts=2024-02-29T13:28:41.621994917Z caller=http.go:73 level=info component=receive service=http/server component=receive msg="listening for requests and metrics" address=0.0.0.0:10902
2024-02-29 13:28:41.622Z ts=2024-02-29T13:28:41.622723592Z caller=tls_config.go:274 level=info component=receive service=http/server component=receive msg="Listening on" address=[::]:10902
2024-02-29 13:28:41.622Z ts=2024-02-29T13:28:41.622851449Z caller=tls_config.go:277 level=info component=receive service=http/server component=receive msg="TLS is disabled." http2=false address=[::]:10902
2024-02-29 13:28:41.622Z ts=2024-02-29T13:28:41.622925351Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-29 13:28:41.623Z ts=2024-02-29T13:28:41.62303853Z caller=grpc.go:131 level=info component=receive service=gRPC/server component=receive msg="listening for serving gRPC" address=0.0.0.0:10901
2024-02-29 13:28:41.626Z ts=2024-02-29T13:28:41.626037872Z caller=receive.go:535 level=info component=receive msg="Set up hashring for the given hashring config."
2024-02-29 13:28:41.626Z ts=2024-02-29T13:28:41.626074827Z caller=intrumentation.go:56 level=info component=receive msg="changing probe status" status=ready
2024-02-29 21:16:28.205Z 2024/02/29 21:16:28 ERROR: [core] subconn returned from pick is type <nil>, not *acBalancerWrapper
2024-02-29 21:16:28.205Z unexpected fault address 0x0
2024-02-29 21:16:28.205Z fatal error: fault
2024-02-29 21:16:28.211Z fatal error: unexpected signal during runtime execution
2024-02-29 21:16:28.211Z panic during panic
2024-02-29 21:16:28.211Z [signal SIGSEGV: segmentation violation code=0x2 addr=0xc0001cae40 pc=0xc0001cae40]
2024-02-29 21:16:28.211Z 
2024-02-29 21:16:28.211Z runtime stack:
2024-02-29 21:16:28.211Z runtime.throw({0x27e0bfb?, fatal: bad g in signal handler
Stream closed: EOF

@vCra
Copy link
Author

vCra commented Feb 29, 2024

Logs 5

Logs 5 

2024-02-29 11:25:53.537Z ts=2024-02-29T11:25:53.537422377Z caller=receive.go:134 level=info component=receive mode=RouterOnly msg="running receive"
2024-02-29 11:25:53.537Z ts=2024-02-29T11:25:53.537462647Z caller=options.go:30 level=info component=receive protocol=HTTP msg="enabling server side TLS"
2024-02-29 11:25:53.537Z ts=2024-02-29T11:25:53.537903251Z caller=options.go:66 level=info component=receive protocol=HTTP msg="server TLS client verification enabled"
2024-02-29 11:25:53.538Z ts=2024-02-29T11:25:53.538020306Z caller=receive.go:741 level=info component=receive msg="no existing storage found, no data migration attempted"
2024-02-29 11:25:53.538Z ts=2024-02-29T11:25:53.538417979Z caller=options.go:26 level=info component=receive protocol=gRPC msg="disabled TLS, key and cert must be set to enable"
2024-02-29 11:25:53.538Z ts=2024-02-29T11:25:53.538734898Z caller=receive.go:444 level=info component=receive msg="starting receiver"
2024-02-29 11:25:53.539Z ts=2024-02-29T11:25:53.538969573Z caller=handler.go:327 level=info component=receive component=receive-handler msg="Start listening for connections" address=0.0.0.0:19291
2024-02-29 11:25:53.539Z ts=2024-02-29T11:25:53.539067971Z caller=receive.go:371 level=info component=receive msg="listening for StoreAPI and WritableStoreAPI gRPC" address=0.0.0.0:10901
2024-02-29 11:25:53.539Z ts=2024-02-29T11:25:53.539101928Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-29 11:25:53.539Z ts=2024-02-29T11:25:53.539396268Z caller=handler.go:349 level=info component=receive component=receive-handler msg="Serving HTTPS" address=0.0.0.0:19291
2024-02-29 11:25:53.539Z ts=2024-02-29T11:25:53.539479349Z caller=grpc.go:131 level=info component=receive service=gRPC/server component=receive msg="listening for serving gRPC" address=0.0.0.0:10901
2024-02-29 11:25:53.540Z ts=2024-02-29T11:25:53.539994298Z caller=intrumentation.go:75 level=info component=receive msg="changing probe status" status=healthy
2024-02-29 11:25:53.540Z ts=2024-02-29T11:25:53.540107344Z caller=http.go:73 level=info component=receive service=http/server component=receive msg="listening for requests and metrics" address=0.0.0.0:10902
2024-02-29 11:25:53.540Z ts=2024-02-29T11:25:53.540255089Z caller=tls_config.go:274 level=info component=receive service=http/server component=receive msg="Listening on" address=[::]:10902
2024-02-29 11:25:53.540Z ts=2024-02-29T11:25:53.540342081Z caller=tls_config.go:277 level=info component=receive service=http/server component=receive msg="TLS is disabled." http2=false address=[::]:10902
2024-02-29 11:25:53.543Z ts=2024-02-29T11:25:53.54310801Z caller=receive.go:535 level=info component=receive msg="Set up hashring for the given hashring config."
2024-02-29 11:25:53.543Z ts=2024-02-29T11:25:53.543147142Z caller=intrumentation.go:56 level=info component=receive msg="changing probe status" status=ready
2024-02-29 18:43:21.406Z runtime: g 4522227: unexpected return pc for runtime.gopark called from 0x0
2024-02-29 18:43:21.406Z stack: frame={sp:0xc003391e70, fp:0xc003391e90} stack=[0xc003352000,0xc003392000)
2024-02-29 18:43:21.406Z 0x000000c003391d70:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391d80:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391d90:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391da0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391db0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391dc0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391dd0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391de0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391df0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e00:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e10:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e20:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e30:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e40:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e50:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e60:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e70: <0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e80:  0x0000000000000000 !0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391e90: >0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391ea0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391eb0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391ec0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391ed0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391ee0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391ef0:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f00:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f10:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f20:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f30:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f40:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f50:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f60:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f70:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z 0x000000c003391f80:  0x0000000000000000  0x0000000000000000
2024-02-29 18:43:21.406Z fatal error: unknown caller pc
2024-02-29 18:43:21.408Z fatal: bad g in signal handler

@vCra
Copy link
Author

vCra commented Mar 1, 2024

looks similar to #6942

@anarcher
Copy link

I have the same symptoms (v0.32.1). Has this been resolved in v0.33.0?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@anarcher @vCra