config: make docker health check configurable #1624

adnxn · 2018-10-16T23:15:08Z

Summary

carried over from #1522

Implementation details

Testing

Builds on Linux (make release)
Builds on Windows (go build -out amazon-ecs-agent.exe ./agent)
Unit tests on Linux (make test) pass
Unit tests on Windows (go test -timeout=25s ./agent/...) pass
Integration tests on Linux (make run-integ-tests) pass
Integration tests on Windows (.\scripts\run-integ-tests.ps1) pass
Functional tests on Linux (make run-functional-tests) pass
Functional tests on Windows (.\scripts\run-functional-tests.ps1) pass

New tests cover the changes:

Description for the changelog

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

yumex93 · 2018-10-17T03:54:55Z

agent/tcs/handler/types.go

@@ -50,3 +51,10 @@ func (params *TelemetrySessionParams) time() ttime.Time {
 	})
 	return params._time
 }
+
+func (params *TelemetrySessionParams) isMetricsDisabled() (bool, error) {


The function name makes me a little bit confusing. It is called isMetricsDisabled, but when DisableMetrics flag is true, the function will not return true. Do you want to say MetricsSessionDisabled or something? Maybe change the function name and add some description as comments here.

fair. changed to isContainerHealthMetricsDisabled since checking against both DisabledMetrics && DisableDockerHealthCheck

yumex93 · 2018-10-17T03:55:26Z

agent/tcs/handler/types_test.go

+			result:      true,
+			err:         nil,
+			description: "both telemetry and health metrics were disable should return true",
+		},


Is it in golang, for && operator, if the first one is false, it will not proceed to check the value of second parameter? Since seems you miss a case, when DisableMetrics is false and DisableDockerHealthCheck is true.

fenxiong · 2018-10-17T04:04:49Z

agent/app/agent_capability_test.go

+		gomock.Any()).AnyTimes().Return([]string{}, nil)
+
+	ctx, cancel := context.WithCancel(context.TODO())
+	// Cancel the context to cancel async routines


why is this needed?

without it we end up with flakey tests that have run away go routines. maybe you've seen some errors similar to "goroutine ended after test" kind of thing. please correct me if i'm wrong though.

fenxiong · 2018-10-17T04:07:41Z

agent/app/agent_capability_test.go

+	capabilities, err := agent.capabilities()
+	require.NoError(t, err)
+
+	capMap := make(map[string]bool)


i think we can use assert.NotContains instead of these

oh cool, changed.

fenxiong · 2018-10-17T04:11:33Z

agent/tcs/handler/handler.go

+		return
+	}
+	if ok {
+		seelog.Warnf("Metrics were disabled, not start the telemetry session")


nit: start -> starting

fenxiong · 2018-10-17T04:16:38Z

agent/tcs/handler/handler_test.go

@@ -73,6 +73,19 @@ func (*mockStatsEngine) GetTaskHealthMetrics() (*ecstcs.HealthMetadata, []*ecstc
 	return nil, nil, nil
 }

+// TestDisableMetrics tests the StartMetricsSession will return immediately if


just wondering what is the result of breaking the logic we test here? it seems that in that case we will just not return immediately, but i'm not sure whether the test will fail?

hrm yea this is a weird one, this looks like it needs a StartMetricsSession refactor to be able to have more sensible testing. we can track this as tech debt.

nit: add a TODO for this

fenxiong · 2018-10-17T04:18:18Z

agent/tcs/handler/types_test.go

+			ok, err := tc.param.isMetricsDisabled()
+			if tc.err != nil {
+				assert.Error(t, err)
+			}


else we should assert no error?

haikuoliu · 2018-10-17T04:25:57Z

agent/tcs/handler/types.go

+
+func (params *TelemetrySessionParams) isMetricsDisabled() (bool, error) {
+	if params.Cfg != nil {
+		return params.Cfg.DisableMetrics && params.Cfg.DisableDockerHealthCheck, nil


Just for understanding, is DisableMetrics something that we have introduced previously but never used? Would it be better to add a description in the PR summary for it?

DisableMetrics maps to ECS_DISABLE_METRICS and is already covered in the README. this PR is to add a toggle for the health checks, which relies on the underlying tcs/metrics subsystem.

adnxn · 2018-10-17T16:18:09Z

thanks for taking a pass at this, will make changes and need to update the test for this actually as well.

fenxiong · 2019-01-09T23:14:02Z

agent/tcs/handler/handler_test.go

@@ -73,6 +73,19 @@ func (*mockStatsEngine) GetTaskHealthMetrics() (*ecstcs.HealthMetadata, []*ecstc
 	return nil, nil, nil
 }

+// TestDisableMetrics tests the StartMetricsSession will return immediately if


nit: add a TODO for this

adnxn · 2019-01-10T17:22:13Z

windows ftest failing:

=== RUN   TestTwoTasksSharedLocalVolume
go : panic: test timed out after 30m0s
At C:\Users\Administrator\AppData\Local\Temp\amazon-ecs-agent\go\src\github.com\aws\amazon-ecs-agent\scripts\run-functi
onal-tests.ps1:26 char:3
+   go test -tags functional -timeout=30m -v ../agent/functional_tests/ ...
+   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (panic: test timed out after 30m0s:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

adnxn · 2019-01-10T17:29:21Z

merging this to dev and tracking intermittent TestTwoTasksSharedLocalVolume separately here #1786.

adnxn requested a review from a team October 16, 2018 23:15

adnxn mentioned this pull request Oct 16, 2018

Make docker health check configurable #1522

Closed

8 tasks

yumex93 reviewed Oct 17, 2018

View reviewed changes

fenxiong reviewed Oct 17, 2018

View reviewed changes

haikuoliu reviewed Oct 17, 2018

View reviewed changes

adnxn changed the title ~~config: make docker health check configurable~~ [wip] config: make docker health check configurable Oct 17, 2018

config: make docker health check configurable

877a310

adnxn force-pushed the richardpen/container-health-config branch from 72f1713 to 877a310 Compare January 9, 2019 22:57

adnxn changed the title ~~[wip] config: make docker health check configurable~~ config: make docker health check configurable Jan 9, 2019

fenxiong approved these changes Jan 9, 2019

View reviewed changes

yumex93 approved these changes Jan 10, 2019

View reviewed changes

adnxn mentioned this pull request Jan 10, 2019

testing: TestTwoTasksSharedLocalVolume timing out on windows #1786

Closed

adnxn merged commit 6635069 into aws:dev Jan 10, 2019

adnxn added this to the 1.25.0 milestone Jan 23, 2019

adnxn deleted the richardpen/container-health-config branch February 19, 2019 19:50

adnxn restored the richardpen/container-health-config branch February 19, 2019 19:51

adnxn deleted the richardpen/container-health-config branch October 21, 2019 20:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config: make docker health check configurable #1624

config: make docker health check configurable #1624

adnxn commented Oct 16, 2018

yumex93 Oct 17, 2018

adnxn Jan 9, 2019

yumex93 Oct 17, 2018

fenxiong Oct 17, 2018

adnxn Jan 9, 2019

fenxiong Oct 17, 2018

adnxn Jan 9, 2019

fenxiong Oct 17, 2018

fenxiong Oct 17, 2018

adnxn Jan 9, 2019

fenxiong Jan 9, 2019

fenxiong Oct 17, 2018

haikuoliu Oct 17, 2018

adnxn Jan 9, 2019

adnxn commented Oct 17, 2018

fenxiong Jan 9, 2019

adnxn commented Jan 10, 2019

adnxn commented Jan 10, 2019 •

edited

Loading

config: make docker health check configurable #1624

config: make docker health check configurable #1624

Conversation

adnxn commented Oct 16, 2018

Summary

Implementation details

Testing

Description for the changelog

Licensing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adnxn commented Oct 17, 2018

Choose a reason for hiding this comment

adnxn commented Jan 10, 2019

adnxn commented Jan 10, 2019 • edited Loading

adnxn commented Jan 10, 2019 •

edited

Loading