health: detect missing task checks #7366

notnoop · 2020-03-17T16:24:00Z

Fixes a bug where an allocation is considered healthy if some of the
tasks are being restarted and as such, their checks aren't tracked by
consul agent client.

The underlying problem is that allocation registration in consul
agent/client code is mutable: tasks get removed as services from consul,
prior to stopping/restarting to allow for graceful removal from LBs.
The downside is that the health tracker may consider the allocation as
healthy if one of the task is down.

This uses the simplest approach to patch the problem by detecting the
number of expected checks against the registered checks.

I don't anticipate disrepency of counters. sreg.Checks should only
contain checks that nomad agent explicitly registered and filter out
unexpected or unrelated checks:

nomad/command/agent/consul/client.go

Lines 1138 to 1147 in 0ecda99

    
           for _, treg := range reg.Tasks { 
        
           	for serviceID, sreg := range treg.Services { 
        
           		sreg.Service = services[serviceID] 
        
           		for checkID := range sreg.checkIDs { 
        
           			if check, ok := checks[checkID]; ok { 
        
           				sreg.Checks = append(sreg.Checks, check) 
        
           			} 
        
           		} 
        
           	} 
        
           }

.

A better approach would have been to strictly compare the found check
IDs against an immutable list of expected IDs. This sadly requires
significant code changes both to task runner service hooks and consul
hooks, that I'm not comfortable so close to cutting a new release.

Fixes #7320

Add tests to check for failing or missing service checks in consul update.

Fixes a bug where an allocation is considered healthy if some of the tasks are being restarted and as such, their checks aren't tracked by consul agent client. The underlying problem is that allocation registration in consul agent/client code is mutable: tasks get removed as services from consul, prior to stopping/restarting to allow for graceful removal from LBs. The downside is that the health tracker may consider the allocation as healthy if one of the task is down. This uses the simplest approach to patch the problem by detecting the number of expected checks against the registered checks. I don't anticipate disrepency of counters. `sreg.Checks` should only contain checks that nomad agent explicitly registered and filter out unexpected or unrelated checks: https://github.com/hashicorp/nomad/blob/0ecda992317d3300e1c1da05170f8bba18410357/command/agent/consul/client.go#L1138-L1147 . A better approach would have been to strictly compare the found check IDs against an immutable list of expected IDs. This sadly requires significant code changes both to task runner service hooks and consul hooks, that I'm not comfortable so close to cutting a new release.

notnoop · 2020-03-17T16:25:30Z

client/allocrunner/health_hook_test.go

+	select {
+	case <-time.After(5 * time.Second):
+		// great no healthy status
+	case health := <-hs.healthCh:
+		require.False(health.healthy)
+
+		// Unhealthy allocs shouldn't emit task events
+		ev := health.taskEvents[task.Name]
+		require.NotNilf(ev, "%#v", health.taskEvents)
+	}


We seem to lack tests for health check logic for failure and it's not obvious to me how to effectively test this quickly without changing many interfaces :(. Waiting 5 seconds is pretty long. Suggestions?

Setting HealthyDeadline=2 * time.Second might be safe and tighten things up a little bit. Not sure how else to test this without extensive changes. t.Parallel should help us out a lot here, but 5s minimum would be unfortunate.

Also testing in the allochealth package may allow peeking at internals in a time-independent way? I'm not sure after a quick peek, but the deadline is only taken into account in this package.

I added a test in allochealth that's not so time sensitive and lowered the tests here to 2s.

schmichael

Seems like we still need to take group checks into account!

nomad/client/allochealth/tracker.go

Lines 241 to 246 in 0047882

    
           //TODO(schmichael) for now skip unknown tasks as 
        
           //they're task group services which don't currently 
        
           //support checks anyway 
        
           if v, ok := t.taskHealth[task]; ok { 
        
           	v.state = state 
        
           }

Out of scope for this PR so I opened this issue: #7375

schmichael · 2020-03-17T21:59:45Z

client/allocrunner/health_hook_test.go

+		// Unhealthy allocs shouldn't emit task events
+		ev := health.taskEvents[task.Name]
+		require.NotNilf(ev, "%#v", health.taskEvents)


The comment says "unhealthy alloc shouldn't emit task events" yet the assertion is non-nil.

I'm also a little confused why both cases (timeout and health=false) seem valid? Is this test non-deterministic?

Sorry - This case case never actually gets exercised - it's dead code, I was trying to get some error messages. will remove it.

schmichael · 2020-03-17T22:01:11Z

client/allocrunner/health_hook_test.go

+	select {
+	case <-time.After(5 * time.Second):
+		// great no healthy status
+	case health := <-hs.healthCh:
+		require.False(health.healthy)
+
+		// Unhealthy allocs shouldn't emit task events
+		ev := health.taskEvents[task.Name]
+		require.NotNilf(ev, "%#v", health.taskEvents)
+	}


Setting HealthyDeadline=2 * time.Second might be safe and tighten things up a little bit. Not sure how else to test this without extensive changes. t.Parallel should help us out a lot here, but 5s minimum would be unfortunate.

schmichael · 2020-03-17T22:34:33Z

client/allocrunner/health_hook_test.go

+	select {
+	case <-time.After(5 * time.Second):
+		// great no healthy status
+	case health := <-hs.healthCh:
+		require.False(health.healthy)
+
+		// Unhealthy allocs shouldn't emit task events
+		ev := health.taskEvents[task.Name]
+		require.NotNilf(ev, "%#v", health.taskEvents)
+	}


Also testing in the allochealth package may allow peeking at internals in a time-independent way? I'm not sure after a quick peek, but the deadline is only taken into account in this package.

dadgar · 2020-03-18T13:40:19Z

client/allochealth/tracker.go

+		if foundChecks < t.consulCheckCount {
+			// didn't see all the checks we expect, task might be slow to start
+			// or restarting tasks that had their service hooks deregistered
+			passed = false


Will this logic work when it is a migration for a batch job:

nomad/client/allocrunner/health_hook.go

Line 253 in c346a47

minHealthyTime = strategy.MinHealthyTime

?

I believe so - the assumption this depends on that checks are unique and that job definition has a 1 to 1 mapping with consul agent check registration, regardless of exact parameter or minHealthyTime. This needs to be tweaked for task group level checks, but will address that in a follow up PR.

Sorry should have added more for my thinking. If it is a batch job, the tasks can finish and then the number of checks that will be registered in Consul will be less than those defined in the job spec.

Ah - excellent point. Going to drawing board a bit to think this through a bit more. Not sure if I fully understand expectations of service health checks in batch - but it's definitely a problem for service jobs with init containers with task lifecycle/dependencies features. Thanks for raising it.

Checked - and looks like batch jobs don't support migrate or update/deployments [1] - do you know when this code is relevant for batch by any chance?

[1] https://github.com/hashicorp/nomad/blob/v0.10.4/nomad/structs/structs.go#L4997-L5021

Ah sorry for the noise! Thought batch supported migrate!

:) You helped found a bug though - addressed it with a different approach in #7383 .

notnoop · 2020-03-23T20:33:46Z

Closing this one as it's superceeded by #7383

github-actions · 2023-01-14T02:13:36Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

Mahmood Ali added 2 commits March 17, 2020 11:43

tests: add a check for failing service checks

b9e3e12

Add tests to check for failing or missing service checks in consul update.

notnoop requested a review from schmichael March 17, 2020 16:24

notnoop self-assigned this Mar 17, 2020

notnoop added this to Triaged in Nomad - Community Issues Triage via automation Mar 17, 2020

notnoop commented Mar 17, 2020

View reviewed changes

schmichael reviewed Mar 17, 2020

View reviewed changes

dadgar reviewed Mar 18, 2020

View reviewed changes

more targetted health tests

a263a4e

notnoop mentioned this pull request Mar 18, 2020

health: detect failing tasks #7383

Merged

notnoop closed this Mar 23, 2020

Nomad - Community Issues Triage automation moved this from Triaged to Done Mar 23, 2020

github-actions bot locked as resolved and limited conversation to collaborators Jan 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

health: detect missing task checks #7366

health: detect missing task checks #7366

notnoop commented Mar 17, 2020

notnoop Mar 17, 2020

schmichael Mar 17, 2020

schmichael Mar 17, 2020

notnoop Mar 18, 2020

schmichael left a comment

schmichael Mar 17, 2020

notnoop Mar 17, 2020

schmichael Mar 17, 2020

schmichael Mar 17, 2020

dadgar Mar 18, 2020

notnoop Mar 18, 2020

dadgar Mar 18, 2020

notnoop Mar 18, 2020

notnoop Mar 18, 2020 •

edited

Loading

dadgar Mar 18, 2020

notnoop Mar 18, 2020

notnoop commented Mar 23, 2020

github-actions bot commented Jan 14, 2023

	for _, treg := range reg.Tasks {
	for serviceID, sreg := range treg.Services {
	sreg.Service = services[serviceID]
	for checkID := range sreg.checkIDs {
	if check, ok := checks[checkID]; ok {
	sreg.Checks = append(sreg.Checks, check)
	}
	}
	}
	}

	//TODO(schmichael) for now skip unknown tasks as
	//they're task group services which don't currently
	//support checks anyway
	if v, ok := t.taskHealth[task]; ok {
	v.state = state
	}

health: detect missing task checks #7366

health: detect missing task checks #7366

Conversation

notnoop commented Mar 17, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schmichael left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notnoop Mar 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

notnoop commented Mar 23, 2020

github-actions bot commented Jan 14, 2023

notnoop Mar 18, 2020 •

edited

Loading