Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

subnet: handle unknown state in delete event #1000

Merged
merged 1 commit into from
Jul 9, 2018

Conversation

ggaaooppeenngg
Copy link
Contributor

@ggaaooppeenngg ggaaooppeenngg commented May 31, 2018

To fix panic (see log below), add a handler for
cache.DeletedFinalStateUnknown.

/usr/local/go/src/runtime/asm_amd64.s:2197
panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node

goroutine 19 [running]:
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x1782360, 0xc420945e80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x1782360, 0xc420945e80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/flannel/subnet/kube.(*kubeSubnetManager).handleAddLeaseEvent(0xc420014f00, 0x1, 0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:176 +0x2d3
github.com/coreos/flannel/subnet/kube.newKubeSubnetManager.func4(0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:165 +0x47
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(0xc4202ba6e0, 0xc4202ba6f0, 0xc4202ba700, 0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:206 +0x49
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnDelete(0xc42018c780, 0x17e38a0, 0xc4204f61c0)
	<autogenerated>:56 +0x73
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.NewIndexerInformer.func1(0x1792520, 0xc4204f6680, 0x1792520, 0xc4204f6680)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:342 +0x519
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc4203120c0, 0xc4203aede0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 +0x27e
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop(0xc420067180)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:147 +0x40
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x2a
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4205fcf70)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420865f70, 0x3b9aca00, 0x0, 0x166ac01, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xad
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4205fcf70, 0x3b9aca00, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).Run(0xc420067180, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x237
github.com/coreos/flannel/subnet/kube.(*kubeSubnetManager).Run(0xc420014f00, 0x7f340ee4d3b0, 0xc4200161a8)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:284 +0x8a
created by github.com/coreos/flannel/subnet/kube.NewSubnetManager
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:128 +0x592

Signed-off-by: Peng Gao peng.gao.dut@gmail.com

Description

Todos

  • Tests
  • Documentation
  • Release note

Release Note

None required

To fix panic (see log below), add a handler for
cache.DeletedFinalStateUnknown.

```
/usr/local/go/src/runtime/asm_amd64.s:2197
panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node [recovered]
	panic: interface conversion: interface {} is cache.DeletedFinalStateUnknown, not *v1.Node

goroutine 19 [running]:
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x1782360, 0xc420945e80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x126
panic(0x1782360, 0xc420945e80)
	/usr/local/go/src/runtime/panic.go:489 +0x2cf
github.com/coreos/flannel/subnet/kube.(*kubeSubnetManager).handleAddLeaseEvent(0xc420014f00, 0x1, 0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:176 +0x2d3
github.com/coreos/flannel/subnet/kube.newKubeSubnetManager.func4(0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:165 +0x47
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnDelete(0xc4202ba6e0, 0xc4202ba6f0, 0xc4202ba700, 0x17e38a0, 0xc4204f61c0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:206 +0x49
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnDelete(0xc42018c780, 0x17e38a0, 0xc4204f61c0)
	<autogenerated>:56 +0x73
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.NewIndexerInformer.func1(0x1792520, 0xc4204f6680, 0x1792520, 0xc4204f6680)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:342 +0x519
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc4203120c0, 0xc4203aede0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 +0x27e
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop(0xc420067180)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:147 +0x40
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).(github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x2a
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4205fcf70)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x5e
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc420865f70, 0x3b9aca00, 0x0, 0x166ac01, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xad
github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4205fcf70, 0x3b9aca00, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x4d
github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache.(*controller).Run(0xc420067180, 0x0)
	/go/src/github.com/coreos/flannel/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x237
github.com/coreos/flannel/subnet/kube.(*kubeSubnetManager).Run(0xc420014f00, 0x7f340ee4d3b0, 0xc4200161a8)
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:284 +0x8a
created by github.com/coreos/flannel/subnet/kube.NewSubnetManager
	/go/src/github.com/coreos/flannel/subnet/kube/kube.go:128 +0x592
```

Signed-off-by: Peng Gao <peng.gao.dut@gmail.com>
Copy link
Contributor

@tomdee tomdee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tomdee tomdee merged commit c7ee1fc into flannel-io:master Jul 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants