Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send the lh update worker into its own routine instead of taking over the reload routine #935

Merged
merged 2 commits into from
Jul 27, 2023

Conversation

nbrownus
Copy link
Collaborator

@nbrownus nbrownus commented Jul 25, 2023

@johnmaguire came across this issue today.

If the lighthouse.interval config is changed and nebula reloads it then the reload routine is taken over. This will cause nebula to be unable to shutdown cleanly and block future reloads from occurring.

goroutine 11 [select]:
runtime.gopark(0xc0001f56b0?, 0x2?, 0x0?, 0x0?, 0xc0001f567c?)
        runtime/proc.go:381 +0xd6 fp=0xc0001f54f8 sp=0xc0001f54d8 pc=0x43b9b6
runtime.selectgo(0xc0001f56b0, 0xc0001f5678, 0xc000191200?, 0x0, 0x0?, 0x1)
        runtime/select.go:327 +0x7be fp=0xc0001f5638 sp=0xc0001f54f8 pc=0x44bc5e
github.com/slackhq/nebula.(*LightHouse).LhUpdateWorker(0xc000184a50, {0xd3e608?, 0xc000036fa0}, {0xd3e8e0?, 0xc000191200})
        github.com/slackhq/nebula@v1.7.2/lighthouse.go:774 +0x20c fp=0xc0001f56e8 sp=0xc0001f5638 pc=0xa70dac
github.com/slackhq/nebula.(*LightHouse).reload(0xc000184a50, 0x1171f00?, 0x0)
        github.com/slackhq/nebula@v1.7.2/lighthouse.go:220 +0x775 fp=0xc0001f5b18 sp=0xc0001f56e8 pc=0xa6c315
github.com/slackhq/nebula.NewLightHouseFromConfig.func1(0xc000034cc0?)
        github.com/slackhq/nebula@v1.7.2/lighthouse.go:134 +0x32 fp=0xc0001f5ba0 sp=0xc0001f5b18 pc=0xa6b9d2
github.com/slackhq/nebula/config.(*C).ReloadConfig(0xc000034cc0)
        github.com/slackhq/nebula@v1.7.2/config/config.go:158 +0x3a3 fp=0xc0001f5e18 sp=0xc0001f5ba0 pc=0x896383

@nbrownus nbrownus force-pushed the lh-reload-bug branch 3 times, most recently from 758c939 to 27007af Compare July 25, 2023 04:27
brad-defined
brad-defined previously approved these changes Jul 26, 2023
lighthouse.go Outdated Show resolved Hide resolved
@wadey wadey added this to the v1.8.0 milestone Jul 27, 2023
@nbrownus nbrownus merged commit 14d0106 into master Jul 27, 2023
@nbrownus nbrownus deleted the lh-reload-bug branch July 27, 2023 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants