Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Problems running gRPC server, misteriously led to failure attaching event_exit_acct_process on latest version #2263

Closed
inliquid opened this issue Mar 27, 2024 · 5 comments
Closed

Problems running gRPC server, misteriously led to failure attaching event_exit_acct_process on latest version #2263

inliquid opened this issue Mar 27, 2024 · 5 comments
Labels
kind/bug Something isn't working

Comments

@inliquid
Copy link
Contributor

inliquid commented Mar 27, 2024 •
edited
Loading

What happened?

This issue stated as a discussion with several participants who had mostly the same problem with Tetragon v1.0.0-v1.0.2.

Symptoms are:

  1. Tetragon can't run due to liveness probe failure, which is based on tetra being connected to Tetragon over gRPC
  2. When liveness probe is disabled Tetragon starts, but gRPC server not run despite logs say that it should:
time="2024-03-23T15:19:59Z" level=info msg="Starting gRPC server" address="localhost:54321" protocol=tcp

When this line is present there should be goroutine started with gRPC server instance.
However, what we observed is that for some mysterious reason it's not run, and stack trace shows no such goroutine - try search for "grpc" in the stack trace output:

# gops stack 1
goroutine 14 [running]:
runtime/pprof.writeGoroutineStacks({0x2e63600, 0xc00086a310})
        /usr/local/go/src/runtime/pprof/pprof.go:703 +0x6a
runtime/pprof.writeGoroutine({0x2e63600?, 0xc00086a310?}, 0xc0000a9400?)
        /usr/local/go/src/runtime/pprof/pprof.go:692 +0x25
runtime/pprof.(*Profile).WriteTo(0x2a6f74e?, {0x2e63600?, 0xc00086a310?}, 0x231f?)
        /usr/local/go/src/runtime/pprof/pprof.go:329 +0x146
github.com/google/gops/agent.handle({0x7c842f4de5c8?, 0xc00086a310}, {0xc0007a6000?, 0x1?, 0x1?})
        /go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:200 +0xd2
github.com/google/gops/agent.listen({0x2e83de8, 0xc0002b9520})
        /go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:144 +0x1a5
created by github.com/google/gops/agent.Listen in goroutine 1
        /go/src/github.com/cilium/tetragon/vendor/github.com/google/gops/agent/agent.go:122 +0x379

goroutine 1 [chan send]:
github.com/cilium/tetragon/pkg/sensors.(*Manager).RemoveAllSensors(...)
        /go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:278
github.com/cilium/tetragon/pkg/observer.RemoveSensors({0x2e98248?, 0xc002d7e910})
        /go/src/github.com/cilium/tetragon/pkg/observer/observer.go:420 +0xc7
main.tetragonExecute.func5()
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:303 +0x1e
main.tetragonExecute()
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:440 +0x1c65
main.execute.func1(0xc000670700?, {0x2a63272?, 0x4?, 0x2a63276?})
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:733 +0xcb
github.com/spf13/cobra.(*Command).execute(0xc00056b200, {0xc000072050, 0x1, 0x1})
        /go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:944 +0x863
github.com/spf13/cobra.(*Command).ExecuteC(0xc00056b200)
        /go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:1068 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/cilium/tetragon/vendor/github.com/spf13/cobra/command.go:992
main.execute()
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:746 +0x16f
main.main()
        /go/src/github.com/cilium/tetragon/cmd/tetragon/tetragon.go:11 +0x13

goroutine 138 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 134
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 16 [syscall, 1 minutes]:
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/local/go/src/os/signal/signal.go:151 +0x1f

goroutine 49 [chan receive, 1 minutes]:
main.tetragonExecute.func3(0x0?)
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:255 +0x50
created by main.tetragonExecute in goroutine 1
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:253 +0xed7

goroutine 50 [chan receive, 1 minutes]:
github.com/cilium/tetragon/pkg/sensors.startSensorManager.func1()
        /go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:65 +0x65
created by github.com/cilium/tetragon/pkg/sensors.startSensorManager in goroutine 1
        /go/src/github.com/cilium/tetragon/pkg/sensors/manager.go:60 +0xc5

goroutine 51 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7c842fb4cea8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc002b18000?, 0xc002b13c50?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc002b18000)
        /usr/local/go/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc002b18000)
        /usr/local/go/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc002b16080)
        /usr/local/go/src/net/tcpsock_posix.go:152 +0x1e
net.(*TCPListener).Accept(0xc002b16080)
        /usr/local/go/src/net/tcpsock.go:315 +0x30
net/http.(*Server).Serve(0xc002d9c000, {0x2e83de8, 0xc002b16080})
        /usr/local/go/src/net/http/server.go:3056 +0x364
net/http.(*Server).ListenAndServe(0xc002d9c000)
        /usr/local/go/src/net/http/server.go:2985 +0x71
net/http.ListenAndServe(...)
        /usr/local/go/src/net/http/server.go:3239
github.com/cilium/tetragon/pkg/metrics.EnableMetrics({0xc000589144, 0x5})
        /go/src/github.com/cilium/tetragon/pkg/metrics/metrics.go:25 +0x1cf
created by main.tetragonExecute in goroutine 1
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:323 +0x109e

goroutine 81 [select]:
k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc002b98060)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:276 +0x305
created by k8s.io/client-go/util/workqueue.newDelayingQueue in goroutine 31
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/delaying_queue.go:113 +0x21f

goroutine 31 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0038ecb50, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000b808a0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/util/workqueue.(*Type).Get(0xc000b808a0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/util/workqueue/queue.go:200 +0x99
github.com/cilium/tetragon/pkg/metrics.StartPodDeleteHandler()
        /go/src/github.com/cilium/tetragon/pkg/metrics/metricwithpod.go:126 +0x44
created by main.tetragonExecute in goroutine 1
        /go/src/github.com/cilium/tetragon/cmd/tetragon/main.go:325 +0x10b6

goroutine 143 [select]:
github.com/cilium/tetragon/pkg/process.(*Cache).cacheGarbageCollector.func1()
        /go/src/github.com/cilium/tetragon/pkg/process/cache.go:47 +0xf1
created by github.com/cilium/tetragon/pkg/process.(*Cache).cacheGarbageCollector in goroutine 1
        /go/src/github.com/cilium/tetragon/pkg/process/cache.go:43 +0xe7

goroutine 137 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e190, 0xc001215dd0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 134
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 96 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002b90f38?, {0x2e66c80, 0xc002bf9170}, 0x1, 0xc002ab0e40)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0xc0?, 0xc002b90f88?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002586ab0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 111
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 86 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc002c2c028, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000943be0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc002c2c000, 0xc002bcf9f0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc002c2a140)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc002bf9080}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002bfb6f0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc002c2a140, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e210, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 1
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205

goroutine 139 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc00250c500}, {0x7c842f8c6a20, 0xc00370e4d0}, {0x2ec3af0?, 0x2a1c760}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002ba4540, {0x0?, 0x0?}, 0xc0000d6420, 0x61bf?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002ba4540, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002514420?, {0x2e66ca0, 0xc002d7e640}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002ba4540, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 134
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 104 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000948f38?, {0x2e66c80, 0xc003056090}, 0x1, 0xc00305a000)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x30?, 0xc0023da8a0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002507950)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 105 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002507950)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 120 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0031902e8, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc002528be0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0031902c0, 0xc002585f30)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc001762e60)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc00297b8f0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002994f90?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc001762e60, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc003190210, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 118
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205

goroutine 115 [IO wait]:
internal/poll.runtime_pollWait(0x7c842fb4cdb0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:343 +0x85
internal/poll.(*pollDesc).wait(0xc002cddd00?, 0xc002562000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc002cddd00, {0xc002562000, 0xa000, 0xa000})
        /usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
net.(*netFD).Read(0xc002cddd00, {0xc002562000?, 0xc00256603a?, 0x1a5?})
        /usr/local/go/src/net/fd_posix.go:55 +0x25
net.(*conn).Read(0xc0004d6008, {0xc002562000?, 0x6f33b7?, 0xc0031421b8?})
        /usr/local/go/src/net/net.go:179 +0x45
crypto/tls.(*atLeastReader).Read(0xc0005e9698, {0xc002562000?, 0xc0005e9698?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:805 +0x3b
bytes.(*Buffer).ReadFrom(0xc0031422a8, {0x2e65a40, 0xc0005e9698})
        /usr/local/go/src/bytes/buffer.go:211 +0x98
crypto/tls.(*Conn).readFromUntil(0xc003142000, {0x2e63520?, 0xc0004d6008}, 0x5fcb?)
        /usr/local/go/src/crypto/tls/conn.go:827 +0xde
crypto/tls.(*Conn).readRecordOrCCS(0xc003142000, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:625 +0x250
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc003142000, {0xc003195000, 0x1000, 0x98cbc9?})
        /usr/local/go/src/crypto/tls/conn.go:1369 +0x158
bufio.(*Reader).Read(0xc003196000, {0xc003180200, 0x9, 0x25aecc0?})
        /usr/local/go/src/bufio/bufio.go:244 +0x197
io.ReadAtLeast({0x2e63320, 0xc003196000}, {0xc003180200, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:335 +0x90
io.ReadFull(...)
        /usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc003180200, 0x9, 0xa8400?}, {0x2e63320?, 0xc003196000?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/frame.go:237 +0x65
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0031801c0)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/frame.go:498 +0x85
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc00315ff98)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2275 +0x11f
golang.org/x/net/http2.(*ClientConn).readLoop(0xc003192000)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2170 +0x65
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 114
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:821 +0xcbe

goroutine 84 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002ba4540, 0xc0000d6420, 0xc002ab0840, 0x209?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 139
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 44 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc00370e448, 0x19)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc0036f92c0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc00370e420, 0xc0025047b0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc0031741e0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc002508ae0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0025021d0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc0031741e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e0b0, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:150 +0x5c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 1
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:148 +0x205

goroutine 85 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc002c01680, 0xc002aa7f00)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0xc0025024e0?, 0xc002522fb8?)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 139
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308

goroutine 109 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028ac780, 0xc0027f4700)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0xc0025021d0?, 0xc002529fb8?)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 129
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308

goroutine 47 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e140, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 44
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 48 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 44
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 129 [select]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc00250c300}, {0x7c842f8c6a20, 0xc00370e420}, {0x2ec3af0?, 0x2a1b8e0}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002ba40e0, {0x0?, 0x0?}, 0xc0000d6420, 0xc003c4fd80?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002ba40e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002514090?, {0x2e66ca0, 0xc002d7e2d0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002ba40e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 44
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 108 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002ba40e0, 0xc0000d6420, 0xc00305a8a0, 0xf?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 129
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 106 [chan receive]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc003c53738?, {0x2e66c80, 0xc0030560c0}, 0x1, 0xc00305a060)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc003c53788?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0025079e0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 107 [select]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc0025079e0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 47
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 133 [sync.Cond.Wait]:
sync.runtime_notifyListWait(0xc0028ac7c8, 0x18)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x2e79fe0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028ac7b0, {0xc00216a001, 0x7dff, 0x7dff})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0x2a62177?}, {0xc00216a001?, 0x2a73ede?, 0xa?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc001717e00)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc001717e00)
        /usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc001717e00, {0x25ff1c0, 0xc0005e9dd0})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc002509560, {0xc002192000, 0x8000, 0xa000})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc002d7e550, 0xc002516840?, {0x2e7a968, 0xc0020f1dc0})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002b16840)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00250c300)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 129
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125

goroutine 134 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc00370e4f8, 0x1)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc002a9ed40?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc00370e4d0, 0xc002504d90)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc003174500)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc0025096b0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0025024e0?, 0x3b9aca00, 0x0, 0x0?, 0xdf8475800?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc003174500, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc00370e160, 0xc002524f20?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
k8s.io/client-go/informers.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:150 +0x5c
created by k8s.io/client-go/informers.(*sharedInformerFactory).Start in goroutine 1
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/informers/factory.go:148 +0x205

goroutine 142 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc002c016c8, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x58?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc002c016b0, {0xc003cea600, 0x200, 0x200})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc002b94d48?}, {0xc003cea600?, 0x2?, 0x100000009?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002548140)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002548140)
        /usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002548140, {0x25ff1c0, 0xc002c84cc0})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc002c827e0, {0xc00253c800, 0x400, 0x400})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc002d7e780, 0x8?, {0x2e7a968, 0xc00250c540})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002b16d80)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc00250c500)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 139
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125

goroutine 94 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:969 +0x4b
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000944f38?, {0x2e66c80, 0xc002bf9140}, 0x1, 0xc002ab0de0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002bf90b0?, 0x3b9aca00, 0x0, 0x20?, 0xc000944f88?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*processorListener).run(0xc002586b40)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:968 +0x69
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 122
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 89 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc002d7e1e0, 0xc0006847a8?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 86
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 90 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 86
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 91 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7df60, 0xc002c282a0}, {0x7c842f8c6a20, 0xc002c2c000}, {0x2ec3af0?, 0x29ef460}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc003a1cb60, {0x0?, 0x0?}, 0xc0000d6420, 0xc000800000?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc003a1cb60, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc002ab0c60?, {0x2e66ca0, 0xc002d0c320}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc003a1cb60, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 86
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 93 [select]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc003a1cb60, 0xc0000d6420, 0xc002ab0d20, 0xc002540f10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 91
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 144 [select]:
github.com/cilium/tetragon/pkg/eventcache.(*Cache).loop(0xc00250c5c0)
        /go/src/github.com/cilium/tetragon/pkg/eventcache/eventcache.go:169 +0x117
created by github.com/cilium/tetragon/pkg/eventcache.NewWithTimer in goroutine 1
        /go/src/github.com/cilium/tetragon/pkg/eventcache/eventcache.go:238 +0x178

goroutine 119 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028d6028, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc000948be0?)
        /usr/local/go/src/sync/cond.go:70 +0x85
k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc0028d6000, 0xc002879bd0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:575 +0x236
k8s.io/client-go/tools/cache.(*controller).processLoop(0xc00286d9a0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:188 +0x30
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x4466c0?, {0x2e66c80, 0xc0028d41e0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc002895b10?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:161
k8s.io/client-go/tools/cache.(*controller).Run(0xc00286d9a0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:159 +0x393
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run(0xc003190160, 0x22a17f4?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:504 +0x2c8
github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:119 +0x5c
created by github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions.(*sharedInformerFactory).Start in goroutine 118
        /go/src/github.com/cilium/tetragon/vendor/github.com/cilium/tetragon/pkg/k8s/client/informers/externalversions/factory.go:117 +0x205

goroutine 122 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc003148280, 0x2ea2f70?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 120
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 123 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 120
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 124 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc002899280}, {0x7c842f8c6a20, 0xc0031902c0}, {0x2ec3af0?, 0x2a01200}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc002a300e0, {0x0?, 0x0?}, 0xc0000d6420, 0xc003c4ff80?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc002a300e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00298cf00?, {0x2e66ca0, 0xc0031482d0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc002a300e0, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 120
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 180 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc002a300e0, 0xc0000d6420, 0xc00305ae40, 0x25cbf40?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 124
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 111 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*sharedProcessor).run(0xc003148230, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:803 +0x47
k8s.io/client-go/tools/cache.(*sharedIndexInformer).Run.(*Group).StartWithChannel.func4()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 119
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 112 [chan receive, 1 minutes]:
k8s.io/client-go/tools/cache.(*controller).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:132 +0x25
created by k8s.io/client-go/tools/cache.(*controller).Run in goroutine 119
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0xab

goroutine 95 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002586b40)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 122
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 161 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*processorListener).pop(0xc002586ab0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/shared_informer.go:939 +0x10d
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 111
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 177 [select, 1 minutes]:
k8s.io/client-go/tools/cache.watchHandler({0x0?, 0x0?, 0x4704d60?}, {0x2e7ae40, 0xc002a1cec0}, {0x7c842f8c6a20, 0xc0028d6000}, {0x2ec3af0?, 0x29fc4a0}, 0x0, ...)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:714 +0x17b
k8s.io/client-go/tools/cache.(*Reflector).watch(0xc0028d8000, {0x0?, 0x0?}, 0xc0000d6420, 0x0?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:433 +0x545
k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch(0xc0028d8000, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:358 +0x36c
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:291 +0x25
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00305ac00?, {0x2e66ca0, 0xc0029fe0a0}, 0x1, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0028d8000, 0xc0000d6420)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:290 +0x1cd
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1e
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4f
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 119
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

goroutine 182 [select, 1 minutes]:
k8s.io/client-go/tools/cache.(*Reflector).startResync(0xc0028d8000, 0xc0000d6420, 0xc00305b140, 0xc002525fb8?)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:370 +0x10f
created by k8s.io/client-go/tools/cache.(*Reflector).ListAndWatch in goroutine 177
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/tools/cache/reflector.go:357 +0x348

goroutine 181 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028aca80, 0xc0027f4d00)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0xc003c4ff10?)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 124
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308

goroutine 127 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028acc48, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0x58?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028acc30, {0xc00319cc00, 0x200, 0x200})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc000947d48?}, {0xc00319cc00?, 0x2?, 0x10000000f?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002a223c0)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002a223c0)
        /usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002a223c0, {0x25ff1c0, 0xc0029beaf8})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc00297bdd0, {0xc002a2dc00, 0x400, 0x400})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0031484b0, 0x8?, {0x2e7a968, 0xc002a1cf00})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc002597b20)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc002a1cec0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 177
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125

goroutine 183 [select, 1 minutes]:
golang.org/x/net/http2.(*clientStream).writeRequest(0xc0028acc00, 0xc0027f5000)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1464 +0xac7
golang.org/x/net/http2.(*clientStream).doRequest(0x0?, 0xc003c4df10?)
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1326 +0x18
created by golang.org/x/net/http2.(*ClientConn).RoundTrip in goroutine 177
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:1232 +0x308

goroutine 184 [sync.Cond.Wait, 1 minutes]:
sync.runtime_notifyListWait(0xc0028acac8, 0x0)
        /usr/local/go/src/runtime/sema.go:527 +0x159
sync.(*Cond).Wait(0xc0028acab8?)
        /usr/local/go/src/sync/cond.go:70 +0x85
golang.org/x/net/http2.(*pipe).Read(0xc0028acab0, {0xc0039aa800, 0x200, 0x200})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/pipe.go:76 +0xdf
golang.org/x/net/http2.transportResponseBody.Read({0xc003c4dd48?}, {0xc0039aa800?, 0xc003c4dd10?, 0xc003c4dcac?})
        /go/src/github.com/cilium/tetragon/vendor/golang.org/x/net/http2/transport.go:2558 +0x65
encoding/json.(*Decoder).refill(0xc002889900)
        /usr/local/go/src/encoding/json/stream.go:165 +0x188
encoding/json.(*Decoder).readValue(0xc002889900)
        /usr/local/go/src/encoding/json/stream.go:140 +0x9d
encoding/json.(*Decoder).Decode(0xc002889900, {0x25ff1c0, 0xc0028e2210})
        /usr/local/go/src/encoding/json/stream.go:63 +0x75
k8s.io/apimachinery/pkg/util/framer.(*jsonFrameReader).Read(0xc0028d5ef0, {0xc0028cf400, 0x400, 0x400})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/util/framer/framer.go:152 +0x19c
k8s.io/apimachinery/pkg/runtime/serializer/streaming.(*decoder).Decode(0xc0029fe5f0, 0xc003192050?, {0x2e7a968, 0xc0028992c0})
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/runtime/serializer/streaming/streaming.go:77 +0xa3
k8s.io/client-go/rest/watch.(*Decoder).Decode(0xc0028f65c0)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/client-go/rest/watch/decoder.go:49 +0x4b
k8s.io/apimachinery/pkg/watch.(*StreamWatcher).receive(0xc002899280)
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:105 +0xdb
created by k8s.io/apimachinery/pkg/watch.NewStreamWatcher in goroutine 124
        /go/src/github.com/cilium/tetragon/vendor/k8s.io/apimachinery/pkg/watch/streamwatcher.go:76 +0x125

Here is gops 1 output in our case, it's clear that gRPC server is not run:

# gops 1
threads:        16
memory usage:   0.540%
cpu usage:      1.152%
username:       root
cmd+args:       /usr/bin/tetragon --config-dir=/etc/tetragon/tetragon.conf.d/
elapsed time:   01:21
local/remote:   127.0.0.1:8118 <-> 0.0.0.0:0 (LISTEN)
local/remote:   0.0.0.0:2112 <-> 0.0.0.0:0 (LISTEN)
local/remote:   10.80.122.156:42272 <-> 10.43.0.1:443 (ESTABLISHED)

After upgrade to latest CI image with help of these command:

helm install --namespace kube-system \    
        --set tetragonOperator.image.override=quay.io/cilium/tetragon-operator-ci:latest \
        --set tetragon.image.override=quay.io/cilium/tetragon-ci:latest  \
        tetragon cilium/tetragon

Tetragon start failing to start with this message:

time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_execve_joined_info_map_stats path=/sys/fs/bpf/tetragon/tg_execve_joined_info_map_stats sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=execve_calls path=/sys/fs/bpf/tetragon/execve_calls sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tcpmon_map path=/sys/fs/bpf/tetragon/tcpmon_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_conf_map path=/sys/fs/bpf/tetragon/tg_conf_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="tetragon, map loaded." map=tg_stats_map path=/sys/fs/bpf/tetragon/tg_stats_map sensor=__base__
time="2024-03-26T12:26:53Z" level=info msg="Loading BPF program" Attach=acct_process Program=/var/lib/tetragon/bpf_exit.o Type=kprobe
time="2024-03-26T12:26:53Z" level=info msg="BPF events statistics: 0 received, 0% events loss"
time="2024-03-26T12:26:53Z" level=info msg="Observer events statistics" errors=0 filterDrop=0 filterPass=0 lost=0 received=0
time="2024-03-26T12:26:53Z" level=info msg="sensor controller starts"
time="2024-03-26T12:26:53Z" level=fatal msg="Failed to start tetragon" error="failed prog /var/lib/tetragon/bpf_exit.o kern_version 328721 loadInstance: attaching 'event_exit_acct_process' failed: creating perf_kprobe PMU (arch-specific fallback for \"acct_process\"): token acct_process: opening perf event: operation not permitted"

OS is Oracle Linux 8.5 with CONFIG_BSD_PROCESS_ACCT enabled

# grep CONFIG_BSD_PROCESS_ACCT  /boot/config-$(uname -r)
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y

Also:

# grep acct_process < /proc/kallsyms
ffffffffb7d5fc20 t do_acct_process
ffffffffb7d60810 T acct_process

Tetragon Version

v1.0.0, v1.0.2, latest CI

Kernel Version

5.4.17-2136.304.4.1.el8uek.x86_64

Kubernetes Version

No response

Bugtool

No response

Relevant log output

No response

Anything else?

No response
@inliquid inliquid added the kind/bug Something isn't working label Mar 27, 2024
@kkourt
Copy link
Contributor

kkourt commented Mar 28, 2024 •
edited
Loading

Thanks @inliquid,

Is Oracle Linux using lockdown by any chance? What's the output of:

 sudo cat /sys/kernel/security/lockdown

@inliquid
Copy link
Contributor Author

Yes it seems so:

# cat /sys/kernel/security/lockdown
none integrity [confidentiality]

@kkourt
Copy link
Contributor

kkourt commented Mar 28, 2024

This mode prohibits loading BPF programs, which means that Tetragon cannot really work. We suggest to switch to "integrity" instead. See iovisor/bcc#2565 for more discussion on this.

There is a PR from @tixxdz to issue an early warning about this: #2149.

@tixxdz tixxdz mentioned this issue Mar 28, 2024
Open
@inliquid
Copy link
Contributor Author

@kkourt @tixxdz thank you, makes sense!

I'll try with "integrity" mode and get back with result.

@mtardy
Copy link
Member

mtardy commented Apr 18, 2024

Given that we now have an issue to add the documentation for lockdown-related issues:

I'll close this issue, feel free to reopen this if you still have issues with this and this will not be addressed by #2265 being resolved.

@mtardy mtardy closed this as completed Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kkourt @inliquid @mtardy