engine: introduce a new env var to distinct pull image behavior #1348

haikuoliu · 2018-04-17T06:41:22Z

Summary

This PR is to fix issue #413, it introduces a new environment variable ECS_AGENT_PULL_BEHAVIOR to allow users to define different pull image behaviors, please see the design doc in the comments of issue #413, and usage of the ECS_AGENT_PULL_BEHAVIOR in README.md.

Implementation details

Add an enum environment variable to config file.
Add a new variable PullSucceeded in ImageState, it will be set to true when the pull succeeds.
For prefer-cached behavior, check if there is cached image by inspecting image, if inspecting image fails, then pull image, otherwise just update container reference. Make task fail when pull fails.
For never behavior, do nothing but update container reference when pulling container.
For once behavior, check if the image has been pulled by inspecting the IsPullSuccess of the image state, do nothing when the image has been pulled, otherwise pull as usual. If the pull image API call fails, make the task fail.
Abstract the update container reference out in docker_task_engine.go.
Some other refactor/bug fix.

Testing

Builds on Linux (make release)
Builds on Windows (go build -out amazon-ecs-agent.exe ./agent)
Unit tests on Linux (make test) pass
Unit tests on Windows (go test -timeout=25s ./agent/...) pass
Integration tests on Linux (make run-integ-tests) pass
Integration tests on Windows (.\scripts\run-integ-tests.ps1) pass
Functional tests on Linux (make run-functional-tests) pass
Functional tests on Windows (.\scripts\run-functional-tests.ps1) pass

New tests cover the changes: yes, unit tests, integ tests and some manual tests.

Description for the changelog

Feature - Introduce a new environment variable ECS_AGENT_PULL_BEHAVIOR to make agent pull behavior configurable

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sharanyad · 2018-04-18T18:49:26Z

CHANGELOG.md

@@ -1,5 +1,8 @@
 # Changelog

+## 1.18.0


The convention is to use 1.18.0-dev before release

sharanyad · 2018-04-18T19:02:50Z

agent/engine/task_manager.go

+			mtask.cfg.AgentPullBehavior == config.OnceAgentPullBehavior {
+			seelog.Criticalf("Managed task [%s]: Error while pulling container %s, task will fail: %v",
+				mtask.Arn, container.Name, event.Error)
+			mtask.SetDesiredStatus(api.TaskStopped)


Here, the task will be moved to STOPPED even if the container is not essential. Is that what we are implying by using the config variable? If yes, we might want to document that.

Essential or non-essential is only effective for RUNNING containers. It shouldn't have a bearing on Pulled transition failing

The task will stop for both essential or non-essential containers. I will add some comments for that.

Today, when a container fails with desired CREATED , we move the task to STOPPED only if it is an essential container. We should do the same for PULLED too.
If not, we might want to change the logic to stop the task for failures during CREATED, irrespective of the essential status of containers.

aaithal · 2018-04-18T20:58:00Z

@haikuoliu can you please fix failing tests?

haikuoliu · 2018-04-18T21:08:49Z

@aaithal It's the timeout test TestStartContainerTimeout and not related to this PR. The timeout tests have a small change to fail, because the timeout value is too small and some methods don't get to execute.

We talked about this in the wire context PR #1329: to fix it either use a mock context (implemented in the initial PR), or use a cancel context to simulate a timeout context (in one of the revised PR), or don't fix it at all (in final PR), we choose the last method.

aaithal · 2018-04-18T22:03:52Z

or don't fix it at all (in final PR), we choose the last method.

That doesn't seem right. I think you 'fix' it by adding MaxTimes(1) to the mock docker client call. Can you please make that change it it hasn't already been made somewhere else?

aaithal · 2018-04-18T20:58:54Z

CHANGELOG.md

@@ -1,5 +1,8 @@
 # Changelog

+## 1.18.0
+* Enhancement - Introduce a new environment variable `ECS_AGENT_PULL_BEHAVIOR` to make agent pull behavior configurable [#to be added](to be added)


you can actually add the to be added info now. Also, this falls under the Feature bucket, not Enhancement

aaithal · 2018-04-18T20:59:38Z

README.md

@@ -169,6 +169,7 @@ additional details on each available environment variable.
 | `ECS_IMAGE_CLEANUP_INTERVAL` | 30m | The time interval between automated image cleanup cycles. If set to less than 10 minutes, the value is ignored. | 30m | 30m |
 | `ECS_IMAGE_MINIMUM_CLEANUP_AGE` | 30m | The minimum time interval between when an image is pulled and when it can be considered for automated image cleanup. | 1h | 1h |
 | `ECS_NUM_IMAGES_DELETE_PER_CYCLE` | 5 | The maximum number of images to delete in a single automated image cleanup cycle. If set to less than 1, the value is ignored. | 5 | 5 |
+| `ECS_AGENT_PULL_BEHAVIOR` | &lt;default &#124; always &#124; never &#124; once&gt; | Behavior used to customize the pull image process. default: pull image remotely, use cached image if the pull fails; always: only pull image remotely, the task fails when the pull fails; never: only use cached image; once: pull image remotely only if this image has never been pulled before, use cached image otherwise. | default | default |


can you please get @nrdlngr to review this?

OK, will ask review from him.

aaithal · 2018-04-18T21:05:20Z

agent/config/config.go

@@ -86,6 +86,23 @@ const (
 	// performing image cleanup.
 	minimumNumImagesToDeletePerCycle = 1

+	// DefaultAgentPullBehavior specficies the behavior that if an image pull API call fails,
+	// agent tries to start from the Docker image cache anyway, assuming that the image has not changed.
+	DefaultAgentPullBehavior AgentPullBehaviorType = 0


please modify this be to in its own const block and use iota to be more idiomatic: https://golang.org/ref/spec#Iota

aaithal · 2018-04-18T22:04:33Z

agent/config/types.go

@@ -20,6 +20,8 @@ import (
 	cnitypes "github.com/containernetworking/cni/pkg/types"
 )

+type AgentPullBehaviorType int8


this needs a comment

aaithal · 2018-04-18T22:07:57Z

agent/engine/docker_task_engine.go

+	default:
+		// If the agent pull behavior is never, we don't try to pull the image,
+		// try to use local image cache instead.
+		if engine.cfg.AgentPullBehavior == config.NeverAgentPullBehavior {


can you please refactor these into a method? May be something that returns a boolean to indicate if you should return dockerapi.DockerContainerMetadata{Error: nil}?

OK, will abstract it out.

aaithal · 2018-04-18T22:08:35Z

agent/engine/image/types.go

+	Containers      []*api.Container `json:"-"`
+	PulledAt        time.Time
+	LastUsedAt      time.Time
+	IsPulledSuccess bool


PullSucceeded seems like a better name for this. Also, please add documentation for this and all other fields.

aaithal · 2018-04-18T22:09:42Z

agent/engine/task_manager.go

+			mtask.cfg.AgentPullBehavior == config.OnceAgentPullBehavior {
+			seelog.Criticalf("Managed task [%s]: Error while pulling container %s, task will fail: %v",
+				mtask.Arn, container.Name, event.Error)
+			mtask.SetDesiredStatus(api.TaskStopped)


Essential or non-essential is only effective for RUNNING containers. It shouldn't have a bearing on Pulled transition failing

aaithal · 2018-04-18T22:10:50Z

@haikuoliu you also need to update the version and documentation for the agent's state file in

amazon-ecs-agent/agent/statemanager/state_manager.go

Line 65 in 159ae5c

ECSDataVersion = 11

haikuoliu · 2018-04-18T23:16:16Z

@aaithal My bad, thanks for pointing out. I will fix it. Will also update the version and documentation for state_manager.go

haikuoliu · 2018-04-19T01:23:16Z

Update: fix the timeout test, add more comments, update change log, and some other refactor.

Still waiting for review of README.md.

aaithal · 2018-04-19T14:56:08Z

CHANGELOG.md

@@ -1,5 +1,8 @@
 # Changelog

+## 1.18.0-dev
+* Feature - Introduce a new environment variable `ECS_AGENT_PULL_BEHAVIOR` to make agent pull behavior configurable [#1348](https://github.com/aws/amazon-ecs-agent/pull/1348)


Feature - Configurable agent pull behavior [#1348].. seems like a better description for this.

aaithal · 2018-04-19T14:56:49Z

agent/config/config.go

@@ -325,6 +344,7 @@ func environmentConfig() (Config, error) {
 	imageCleanupInterval := parseEnvVariableDuration("ECS_IMAGE_CLEANUP_INTERVAL")
 	numImagesToDeletePerCycleEnvVal := os.Getenv("ECS_NUM_IMAGES_DELETE_PER_CYCLE")
 	numImagesToDeletePerCycle, err := strconv.Atoi(numImagesToDeletePerCycleEnvVal)
+	agentPullBehavior := getAgentPullBehavior()


you don't need this variable. Just set AgentPullBehavior to getAgentPullBehavior().

I wanted to be consistent with other code. I just saw your refactor config commit, will change accordingly.

aaithal · 2018-04-19T14:59:24Z

agent/config/config.go

@@ -455,6 +476,24 @@ func getContainerStartTimeout() time.Duration {
 	return containerStartTimeout
 }

+func getAgentPullBehavior() AgentPullBehaviorType {
+	var agentPullBehavior AgentPullBehaviorType


you can get rid of this variable as well:

func getAgentPullBehavior() AgentPullBehaviorType { switch agentPullBehaviorString { case "always": return AlwaysAgentPullBehavior ... } }

aaithal · 2018-04-19T15:00:37Z

agent/config/config_test.go

@@ -369,6 +371,14 @@ func TestImageCleanupMinimumNumImagesToDeletePerCycle(t *testing.T) {
 	assert.Equal(t, cfg.NumImagesToDeletePerCycle, DefaultNumImagesToDeletePerCycle, "Wrong value for NumImagesToDeletePerCycle")
 }

+func TestInvalidAgentPullBehavior(t *testing.T) {


can you add a test for just getAgentPullBehavior() as well? Make it a table driven test so that you can test that all 4 valid values and an invalid value for ECS_AGENT_PULL_BEHAVIOR return the expected output.

aaithal · 2018-04-19T15:04:27Z

agent/engine/docker_task_engine.go

@@ -741,16 +776,23 @@ func (engine *DockerTaskEngine) pullAndUpdateContainerReference(task *api.Task,
 	if container.IsInternal() {
 		return metadata
 	}
+	PullSucceeded := metadata.Error == nil


why is this PullSucceeded and not pullSucceeded?

Yes, it should be pullSucceeded.

aaithal · 2018-04-19T15:05:56Z

agent/engine/docker_task_engine.go

+		// (the image can be prepopulated with the AMI and never be pulled).
+		imageState := engine.imageManager.GetImageStateFromImageName(container.Image)
+		if imageState != nil && imageState.PullSucceeded {
+			seelog.Infof("Task engine [%s]: image %s has been pulled, use it directly for container %s",


i'd suggest changing this slightly to :image %s has been pulled once, not pulling it again

aaithal · 2018-04-19T15:06:30Z

agent/engine/docker_task_engine.go

+	taskArn string) bool {
+	// If the agent pull behavior is never, we don't try to pull the image,
+	// try to use local image cache instead.
+	if agentPullBehavior == config.NeverAgentPullBehavior {


minor: this'd be more readable with a switch and cases.

aaithal · 2018-04-19T15:07:50Z

agent/engine/docker_task_engine.go

+		// otherwise pull the image as usual, regardless whether the image exists or not
+		// (the image can be prepopulated with the AMI and never be pulled).
+		imageState := engine.imageManager.GetImageStateFromImageName(container.Image)
+		if imageState != nil && imageState.PullSucceeded {


I think all references to PullSucceeded need to be protected with a lock. Please add setters and getters in image state struct for that use those methods everywhere.

aaithal · 2018-04-19T15:10:23Z

agent/engine/task_manager.go

+		// where there is no local image cache will be handled when creating container.
+		if mtask.cfg.AgentPullBehavior == config.AlwaysAgentPullBehavior ||
+			mtask.cfg.AgentPullBehavior == config.OnceAgentPullBehavior {
+			seelog.Criticalf("Managed task [%s]: Error while pulling container %s, task will fail: %v",


you should probably log the container image here as well.

aaithal · 2018-04-19T15:17:45Z

agent/config/config.go

@@ -455,6 +476,24 @@ func getContainerStartTimeout() time.Duration {
 	return containerStartTimeout
 }

+func getAgentPullBehavior() AgentPullBehaviorType {


you should rebase on top my pr #1353 once it's merged so that this gets moved to parse.go.

richardpen · 2018-04-19T17:42:05Z

agent/config/config.go

+const (
+	// DefaultAgentPullBehavior specficies the behavior that if an image pull API call fails,
+	// agent tries to start from the Docker image cache anyway, assuming that the image has not changed.
+	DefaultAgentPullBehavior AgentPullBehaviorType = iota


ImagePullDefault ImagePullAlwasy ImagePullNever ImagePullOnce seems to be better than this, can you change the name to be more readable? Also, can you move this into the types.go?

Yeah I can change the name, but I think they are more of constants like DefaultClusterName and DefaultTaskCleanupWaitDuration, so they should be kept together with other constants defined in config.go.

richardpen · 2018-04-19T17:43:38Z

agent/config/config.go

+func getAgentPullBehavior() AgentPullBehaviorType {
+	var agentPullBehavior AgentPullBehaviorType
+	agentPullBehaviorString := os.Getenv("ECS_AGENT_PULL_BEHAVIOR")
+	switch agentPullBehaviorString {


You may need to format the string a little bit before comparing, like trim space, convert all character to lowercase.

ok, Good idea!

Hi I think this has already been done by func (cfg *Config) trimWhitespace()

richardpen · 2018-04-19T18:20:23Z

agent/engine/docker_image_manager.go

@@ -154,6 +154,9 @@ func (imageManager *dockerImageManager) addContainerReferenceToNewImageState(con
 			Image:      sourceImage,
 			PulledAt:   time.Now(),
 			LastUsedAt: time.Now(),
+			// The PullSucceeded filed is false by default,
+			// one has to explicitly set it to be true when the pull image succeeds.
+			PullSucceeded: false,


Please help me understand: the addContainerReferenceToNewImageState is called after the image is pulled succeed, why PullSucceeded was set to false, what will happen if it was set to true here?

This is because even if the image pull fails, the container reference will still be updated, so we set it to be false here. We check if there is no error with image pull, we then set it to be true.

richardpen · 2018-04-19T18:29:58Z

agent/engine/docker_task_engine.go

+		seelog.Infof("Task engine [%s]: use cached image directly for container %s",
+			taskArn, container.Name)
+		// It's not an internal container, so it's safe to add container reference.
+		engine.updateContainerReference(false, container, taskArn)


In this case, do we even need to track the image state? If it's not pulled by agent, we shouldn't remove it.

@richardpen and I discussed about the problem we may face about image cleaning. If we use never or once agent behavior setup, the image cleanup process could remove the image, and I think for never and once we should keep the image cache, right?

We think what we can do:

Let customer know that there may be conflicts between image cleaning and agent pull behavior setup, and suggest them when using never or once, set NumImagesToDeletePerCycle to be 0 (or some other image cleanup settings that can prevent image from being removed).

Disable image cleanup internally when using never or once, add some documentation on that so that user knows.

@sharanyad @richardpen @aaithal what do you think about this?

richardpen · 2018-04-19T19:57:24Z

agent/engine/task_manager.go

+				mtask.Arn, container.Name, event.Error)
+			// The task should be stopped regardless of whether this container is
+			// essential or non-essential.
+			mtask.SetDesiredStatus(api.TaskStopped)


Why are you emitting the STOPPED event, the task may not be STOPPED at this point.

yeah, don't notify ACS right now is better.

sharanyad · 2018-04-19T20:54:20Z

agent/engine/docker_task_engine.go

+// useLocalImageCache returns true if local image cache should be used, or return false to continue
+// pulling image, by inspecting the agent pull behavior variable defined in config. The caller has
+// to make sure the container passed in is not an internal container.
+func (engine *DockerTaskEngine) useLocalImageCache(agentPullBehavior config.AgentPullBehaviorType,


Renaming this to ShouldUseLocalImage might be more readable.

sharanyad · 2018-04-19T20:55:41Z

agent/engine/docker_task_engine.go

+		// (the image can be prepopulated with the AMI and never be pulled).
+		imageState := engine.imageManager.GetImageStateFromImageName(container.Image)
+		if imageState != nil && imageState.PullSucceeded {
+			seelog.Infof("Task engine [%s]: image %s has been pulled, use it directly for container %s",


Shouldn't we call updateContainerReference in this case?

yeah you are right, we should.

sharanyad · 2018-04-19T21:05:39Z

agent/engine/docker_task_engine.go

@@ -741,16 +776,23 @@ func (engine *DockerTaskEngine) pullAndUpdateContainerReference(task *api.Task,
 	if container.IsInternal() {
 		return metadata
 	}
+	PullSucceeded := metadata.Error == nil
+	engine.updateContainerReference(PullSucceeded, container, task.Arn)


Can this logic be handled here -

amazon-ecs-agent/agent/engine/docker_image_manager.go

Line 112 in 159ae5c

if err != nil {

i.e, depending on InspectImage, set PullSucceeded value?

I think inspect image and pull image are two different things, when you inspect an image, the image can be there because it' reconfigured with the AMI, instead of being pulled from user's ECR, this matters when the setting is once.

sharanyad · 2018-04-19T21:09:13Z

agent/engine/docker_task_engine_test.go

+	assert.Equal(t, dockerapi.DockerContainerMetadata{}, metadata, "expected empty metadata")
+}
+
+func TestPullImageWithOnceAgentPullBehavior(t *testing.T) {


Can you also add a test where config is once and PullSucceeded is false?

haikuoliu · 2018-04-26T20:55:49Z

Update: address the comments. Introduce a new behavior "prefer_cached". Will update README, add comments and tests for the new behavior and remove "never" behavior due to the change of the requirement.

haikuoliu · 2018-04-26T22:00:41Z

Update: Update README, add comments and tests for "prefer_cached" behavior and remove "never" behavior due to the change of the requirement. Also do a rebase.

petderek · 2018-04-26T23:01:33Z

agent/config/config_test.go

+}
+
+func TestParseImagePullBehavior(t *testing.T) {
+	testcases := []struct {


petderek · 2018-04-26T23:03:22Z

agent/config/parse.go

@@ -117,6 +117,22 @@ func parseNumImagesToDeletePerCycle() int {
 	return numImagesToDeletePerCycle
 }

+func parseImagePullBehavior() ImagePullBehaviorType {
+	ImagePullBehaviorString := os.Getenv("ECS_IMAGE_PULL_BEHAVIOR")


should we .ToLowerCase() the string?

I think we use trimWhitespace to format every env var string and we should not do extra work outside of this method:

amazon-ecs-agent/agent/config/config.go

Line 187 in 159ae5c

func (cfg *Config) trimWhitespace() {

petderek · 2018-04-26T23:03:57Z

agent/config/types.go

@@ -20,6 +20,10 @@ import (
 	cnitypes "github.com/containernetworking/cni/pkg/types"
 )

+// ImagePullBehaviorType is an enum variable type corresponding to different agent pull
+// behaviors including default, always, never and once.
+type ImagePullBehaviorType int8


why an int8 over an int?

Because it's just an enum variable, which doesn't require a large range, there is usage of int8 elsewhere in agent:

amazon-ecs-agent/agent/acs/update_handler/updater.go

Line 65 in 159ae5c

type updateStage int8

petderek · 2018-04-26T23:09:34Z

agent/engine/docker_image_manager.go

+	// If the image pull behavior is prefer cached, don't clean up the image,
+	// because the cached image is needed.
+	if imageManager.imagePullBehavior != config.ImagePullPreferCachedBehavior {
+		// passing the cleanup interval as argument which would help during testing


[do this!]
log that you are disabling this.

[nit]
I think I'd prefer positive comparison instead of !=.

if imageManager.imagePullBehavior == config.ImagePullPreferCachedBehavior { seelog.Info("Pull behavior is set to always use cache. Disabling cleanup") return } imageManager.performPeriodicImageCleanup(ctx, imageManager.imageCleanupTimeInterval)

petderek · 2018-04-26T23:11:29Z

agent/engine/docker_task_engine.go

+	case config.ImagePullPreferCachedBehavior:
+		// If the behavior is prefer_cached, don't pull if we found cached image
+		// by inspecting the image.
+		_, err := engine.client.InspectImage(container.Image)


does this need a timeout?

No, the docker client API does not need a timeout for inspecting an image.

aaithal

Can you please remove this from the 1.17.4 milestone? We can sync offline about reasons to do so.

aaithal · 2018-04-27T15:10:41Z

agent/config/parse.go

+		return ImagePullAlwaysBehavior
+	case "once":
+		return ImagePullOnceBehavior
+	case "prefer_cached":


please change this to preferCached.

I found there are Example values: ["awslogs","fluentd","gelf","json-file","journald","splunk"] and Example values: {"custom attribute": "custom_attribute_value"} for a var with multiple words from official docs, but I did not find a camelCase one, I will double check with Joel about this.

Those are examples. These are key words that we're defining. All of our APIs are camlCased and don't have underscroes

For this, I don't think the APIs matter because this is a value rather than a parameter. My opinion is to use "prefer-cached".

aaithal · 2018-04-27T15:12:43Z

agent/engine/docker_task_engine.go

+func (engine *DockerTaskEngine) shouldUseLocalImage(ImagePullBehavior config.ImagePullBehaviorType,
+	container *api.Container,
+	taskArn string) bool {
+	switch ImagePullBehavior {


function param names should never begin with upper case letters. For a second, I got freaked out that you were comparing a const here. Please modify that.

aaithal · 2018-04-27T15:15:02Z

agent/engine/docker_task_engine.go

+		// otherwise pull the image as usual, regardless whether the image exists or not
+		// (the image can be prepopulated with the AMI and never be pulled).
+		imageState := engine.imageManager.GetImageStateFromImageName(container.Image)
+		if imageState != nil && imageState.GetPullSucceeded() {


nil check for existence of something in a map somewhere isn't a good paradigm. Please change the return type of GetImageStateFromImageName to be (ImageState, bool) instead of just *ImageState and use the bool here.

aaithal · 2018-04-27T15:17:53Z

agent/engine/docker_task_engine.go

+// shouldUseLocalImage returns true if local image cache should be used, or return false to continue
+// pulling image, by inspecting the agent pull behavior variable defined in config. The caller has
+// to make sure the container passed in is not an internal container.
+func (engine *DockerTaskEngine) shouldUseLocalImage(ImagePullBehavior config.ImagePullBehaviorType,


The selective action that a caller needs to perform when calling this method is to pull the image if this returns a negative value. Essentially !if shouldUseLocalImage() { // pull.. }. It's much more intuitive if we operated in the reverse manner. Instead of shouldUseLocalImage, please make this imagePullRequired so that you can do if imagePullRequired() { // pull.. }

aaithal · 2018-04-27T15:18:54Z

agent/engine/docker_task_engine.go

 	}
 	imageState := engine.imageManager.GetImageStateFromImageName(container.Image)
+	if imageState != nil && pullSucceeded {


Here too, as per above comment, please use the boolean returned from GetImageStateFromImageName to do this

aaithal · 2018-04-27T15:19:19Z

agent/engine/image/types.go

+	// PullSucceeded defines whether this image has been pulled successfully before,
+	// this should be set to true when one of the pull image call succeeds.
+	PullSucceeded bool
+	updateLock    sync.RWMutex


please rename this to just lock.

aaithal · 2018-04-27T15:20:06Z

agent/config/config.go

+const (
+	// ImagePullDefaultBehavior specifies the behavior that if an image pull API call fails,
+	// agent tries to start from the Docker image cache anyway, assuming that the image has not changed.
+	ImagePullDefaultBehavior ImagePullBehaviorType = iota


This can be just ImagePullDefault

I think ImagePullDefault is more general, it describes the ImagePull but does not specify which aspect it's describing. I wouldn't know if it's ImagePull default timeout, or ImagePull default behavior, etc. So ImagePullDefaultBehavior makes more sense to me. Let me know if you have more comments on this.

ImagePullDefault would be of the type ImagePullBehaviorType which would convey the meaning. Behavior suffix can be removed. Also the type can be just ImagePullType
something like:

ImagePullDefault ImagePullType = iota ImagePullAlways

and the environment variable can be ECS_IMAGE_PULL_TYPE. But I would suggest consulting Joel for this.

yeah my concern is:
If I name it ImagePullDefault, although it's of ImagePullBehaviorType , but if some day in the future, we have something like ImagePullTimeoutType (just for example) as an enum type, and it also has a default value, so it should be named as ImagePullDefault to be consistent, which conflicts what we have now, does this make sense to you?

aaithal · 2018-04-27T15:20:17Z

agent/config/config.go

+
+	// ImagePullAlwaysBehavior specifies the behavior that if an image pull API call fails,
+	// the task fails instead of using cached image.
+	ImagePullAlwaysBehavior


This can be just ImagePullAlways

See comment above.

aaithal · 2018-04-27T15:20:26Z

agent/config/config.go

+	// ImagePullOnceBehavior specifies the behavior that agent will only attempt to pull
+	// the same image once, once an image is pulled, local image cache will be used
+	// for all the containers.
+	ImagePullOnceBehavior


This can be just ImagePullOnce

See comment above.

aaithal · 2018-04-27T15:20:37Z

agent/config/config.go

+
+	// ImagePullPreferCachedBehavior specifies the behavior that agent will only attempt to pull
+	// the image if there is no cached image.
+	ImagePullPreferCachedBehavior


This can be just ImagePullPreferCached

See comment above.

sharanyad · 2018-04-27T17:30:08Z

agent/engine/docker_image_manager.go

@@ -155,6 +156,9 @@ func (imageManager *dockerImageManager) addContainerReferenceToNewImageState(con
 			Image:      sourceImage,
 			PulledAt:   time.Now(),
 			LastUsedAt: time.Now(),
+			// The PullSucceeded filed is false by default,


nit: field

sharanyad · 2018-04-27T17:54:03Z

agent/engine/docker_image_manager.go

@@ -155,6 +156,9 @@ func (imageManager *dockerImageManager) addContainerReferenceToNewImageState(con
 			Image:      sourceImage,
 			PulledAt:   time.Now(),
 			LastUsedAt: time.Now(),
+			// The PullSucceeded filed is false by default,
+			// one has to explicitly set it to be true when the pull image succeeds.
+			PullSucceeded: false,


bool is false by default. We need not explicitly set it here.

sharanyad · 2018-04-27T18:02:24Z

agent/engine/task_manager.go

+		// don't want to use cached image for both cases.
+		if mtask.cfg.ImagePullBehavior == config.ImagePullAlwaysBehavior ||
+			mtask.cfg.ImagePullBehavior == config.ImagePullOnceBehavior {
+			seelog.Criticalf("Managed task [%s]: Error while pulling container %s and image %s, task will fail: %v",


This can be Errorf and the message can be Managed task [%s]: Error while pulling image %s for container %s , moving task to STOPPED:

haikuoliu · 2018-04-27T23:56:19Z

Update: rebase and refactor to address the comments. Currently I am not sure about when this feature should be released, so I keep it 1.17.4 in changelog, will update the changelog before merged.

richardpen · 2018-05-01T21:14:50Z

README.md

@@ -169,6 +169,7 @@ additional details on each available environment variable.
 | `ECS_IMAGE_CLEANUP_INTERVAL` | 30m | The time interval between automated image cleanup cycles. If set to less than 10 minutes, the value is ignored. | 30m | 30m |
 | `ECS_IMAGE_MINIMUM_CLEANUP_AGE` | 30m | The minimum time interval between when an image is pulled and when it can be considered for automated image cleanup. | 1h | 1h |
 | `ECS_NUM_IMAGES_DELETE_PER_CYCLE` | 5 | The maximum number of images to delete in a single automated image cleanup cycle. If set to less than 1, the value is ignored. | 5 | 5 |
+| `ECS_IMAGE_PULL_BEHAVIOR` | &lt;default &#124; always &#124; once &#124; prefer-cached &gt; | The behavior used to customize the pull image process. If `default` is specified, the image will be pulled remotely, if the pull fails then the cached image will be used. If `always` is specified, the image will be pulled remotely, if the pull fails then the task will fail. If `once` is specified, the image will be pulled remotely if it has not been pulled before, otherwise the cached image will be used. If `prefer-cached` is specified, the image will be pulled remotely if there is no cached image, otherwise the cached image will be used. | default | default |


Is this reviewed by @nrdlngr? Also for once it should be: the image will be pulled remotely if it has not been pulled or it has been removed by image cleanup.

The README and env vars is reviewed by @joelbrandenburg. I think the image cleanup related stuff should be reflected in the tech doc, I will let @joelbrandenburg know about this.

It should also be reflected here, otherwise the description here is not correct and could cause confusion.

richardpen · 2018-05-01T21:38:34Z

agent/engine/docker_image_manager.go

+	// If the image pull behavior is prefer cached, don't clean up the image,
+	// because the cached image is needed.
+	if imageManager.imagePullBehavior == config.ImagePullPreferCachedBehavior {
+		seelog.Info("Pull behavior is set to always use cache. Disabling cleanup")


Can we point this out in the readme file?

I already told @joelbrandenburg about this, this should be also included in tech doc I think.

I think it might be valuable to add a link to the docs in README.

I will ask @joelbrandenburg if it's ok to have a link in readme. I will attach the link in readme if he has the link ready before I merge, otherwise I will attach it some time after I merge.

richardpen · 2018-05-01T21:40:22Z

agent/engine/docker_image_manager_integ_test.go

 	} else {
-		t.Logf("Found image state for %s", test1Image2Name)
+		t.Fatalf("Could not find image state for %s", test1Image1Name)


since you are here, can you change these to use assert or require?

richardpen · 2018-05-01T21:48:29Z

agent/engine/docker_task_engine.go

-	return engine.serialPull(task, container)
+	if engine.imagePullRequired(engine.cfg.ImagePullBehavior, container, task.Arn) {
+		// Record the pullStoppedAt timestamp
+		defer func() {


In this case what's the value of task PullStartedAt and pullStoppedAt?

I didn't quite get the question, the logic in imagePullRequired is to pull image, instead of using local cache, hence recording PullStartedAt and pullStoppedAt is necessary. (previously it's shouldUseLocalImage, and this is changed to imagePullRequired as per @aaithal's comments).

So if engine.imagePullRequired is false, then the image pullstartedAt and pullStoppedAt won't be set, right? Can you run a task in this scenario and check the pullstoppedAt and pullstartedat from describetask API that it isn't set to something unexpected?

yeah I see what you mean, I saw the comment here:

amazon-ecs-agent/agent/api/task.go

Line 112 in 159ae5c

// it won't be set if the pull never happens

And I test the describe task API (which you have seen), it works as expected, the pull started at and pull stopped at will not show up.

the pull started at and pull stopped at will now show up.

If image is not pulled, these values shouldn't show up right?

yeah it's not show up, typed wrong word.

richardpen · 2018-05-01T21:58:09Z

agent/engine/image/types.go

+
+// GetPullSucceeded safely returns the PullSucceeded of the imageState
+func (imageState *ImageState) GetPullSucceeded() bool {
+	imageState.lock.Lock()


This should be a read lock.

haikuoliu · 2018-05-02T17:21:06Z

Update: Address @richardpen 's comments, and squash the commits.

sharanyad

almost there!

sharanyad · 2018-05-02T17:05:45Z

README.md

@@ -169,6 +169,7 @@ additional details on each available environment variable.
 | `ECS_IMAGE_CLEANUP_INTERVAL` | 30m | The time interval between automated image cleanup cycles. If set to less than 10 minutes, the value is ignored. | 30m | 30m |
 | `ECS_IMAGE_MINIMUM_CLEANUP_AGE` | 30m | The minimum time interval between when an image is pulled and when it can be considered for automated image cleanup. | 1h | 1h |
 | `ECS_NUM_IMAGES_DELETE_PER_CYCLE` | 5 | The maximum number of images to delete in a single automated image cleanup cycle. If set to less than 1, the value is ignored. | 5 | 5 |
+| `ECS_IMAGE_PULL_BEHAVIOR` | &lt;default &#124; always &#124; once &#124; prefer-cached &gt; | The behavior used to customize the pull image process. If `default` is specified, the image will be pulled remotely, if the pull fails then the cached image will be used. If `always` is specified, the image will be pulled remotely, if the pull fails then the task will fail. If `once` is specified, the image will be pulled remotely if it has not been pulled before, otherwise the cached image will be used. If `prefer-cached` is specified, the image will be pulled remotely if there is no cached image, otherwise the cached image will be used. | default | default |


It would be good to specify that the cached image in the instance will be used.

sharanyad · 2018-05-02T17:13:10Z

agent/engine/docker_image_manager.go

+	// If the image pull behavior is prefer cached, don't clean up the image,
+	// because the cached image is needed.
+	if imageManager.imagePullBehavior == config.ImagePullPreferCachedBehavior {
+		seelog.Info("Pull behavior is set to always use cache. Disabling cleanup")


I think it might be valuable to add a link to the docs in README.

sharanyad · 2018-05-02T17:15:56Z

agent/engine/docker_task_engine.go

-	return engine.serialPull(task, container)
+	if engine.imagePullRequired(engine.cfg.ImagePullBehavior, container, task.Arn) {
+		// Record the pullStoppedAt timestamp
+		defer func() {


the pull started at and pull stopped at will now show up.

If image is not pulled, these values shouldn't show up right?

sharanyad · 2018-05-02T17:23:09Z

agent/engine/task_manager.go

+		// don't want to use cached image for both cases.
+		if mtask.cfg.ImagePullBehavior == config.ImagePullAlwaysBehavior ||
+			mtask.cfg.ImagePullBehavior == config.ImagePullOnceBehavior {
+			seelog.Errorf("Managed task [%s]: Error while pulling image %s for container %s , moving task to STOPPED: %v",


nit: Managed task [%s]: error while
same for below

sharanyad · 2018-05-02T17:35:25Z

CHANGELOG.md

@@ -1,6 +1,7 @@
 # Changelog

 ## 1.17.4-dev
+* Feature - Configurable agent pull behavior [#1348](https://github.com/aws/amazon-ecs-agent/pull/1348)


Could you please change this to Configurable container image pull behavior

richardpen

Minor comments, otherwise lgtm. Also please make sure all these behaviors are manually tested.

richardpen · 2018-05-02T20:44:46Z

README.md

@@ -169,6 +169,7 @@ additional details on each available environment variable.
 | `ECS_IMAGE_CLEANUP_INTERVAL` | 30m | The time interval between automated image cleanup cycles. If set to less than 10 minutes, the value is ignored. | 30m | 30m |
 | `ECS_IMAGE_MINIMUM_CLEANUP_AGE` | 30m | The minimum time interval between when an image is pulled and when it can be considered for automated image cleanup. | 1h | 1h |
 | `ECS_NUM_IMAGES_DELETE_PER_CYCLE` | 5 | The maximum number of images to delete in a single automated image cleanup cycle. If set to less than 1, the value is ignored. | 5 | 5 |
+| `ECS_IMAGE_PULL_BEHAVIOR` | &lt;default &#124; always &#124; once &#124; prefer-cached &gt; | The behavior used to customize the pull image process. If `default` is specified, the image will be pulled remotely, if the pull fails then the cached image will be used. If `always` is specified, the image will be pulled remotely, if the pull fails then the task will fail. If `once` is specified, the image will be pulled remotely if it has not been pulled before, otherwise the cached image will be used. If `prefer-cached` is specified, the image will be pulled remotely if there is no cached image, otherwise the cached image will be used. | default | default |


It should also be reflected here, otherwise the description here is not correct and could cause confusion.

haikuoliu · 2018-05-02T21:22:44Z

Update: address @richardpen and @sharanyad 's comments.

sharanyad · 2018-05-02T22:57:49Z

agent/engine/docker_image_manager.go

@@ -155,6 +156,8 @@ func (imageManager *dockerImageManager) addContainerReferenceToNewImageState(con
 			Image:      sourceImage,
 			PulledAt:   time.Now(),
 			LastUsedAt: time.Now(),
+			// PullSucceeded should be set to true when the pull image succeeds.
+			PullSucceeded: false,


bool value is false by default. This can be removed.

sharanyad · 2018-05-02T23:06:57Z

agent/engine/task_manager.go

+			mtask.SetDesiredStatus(api.TaskStopped)
+			return false
+		}
+		// If the agent pull behavior is prefer_cached, we receive the error because


nit: prefer-cached

haikuoliu added this to the 1.18.0 milestone Apr 17, 2018

haikuoliu requested review from aaithal, sharanyad and richardpen April 17, 2018 06:41

sharanyad reviewed Apr 18, 2018

View reviewed changes

aaithal reviewed Apr 18, 2018

View reviewed changes

aaithal reviewed Apr 19, 2018

View reviewed changes

richardpen reviewed Apr 19, 2018

View reviewed changes

sharanyad reviewed Apr 19, 2018

View reviewed changes

haikuoliu modified the milestones: 1.18.0, 1.17.4 Apr 25, 2018

haikuoliu force-pushed the branch_pull_image branch from 2d212de to 7e52650 Compare April 26, 2018 20:52

haikuoliu force-pushed the branch_pull_image branch from 7e52650 to 8af18ee Compare April 26, 2018 21:58

haikuoliu requested a review from petderek April 26, 2018 21:59

petderek reviewed Apr 26, 2018

View reviewed changes

aaithal reviewed Apr 27, 2018

View reviewed changes

haikuoliu removed this from the 1.17.4 milestone Apr 27, 2018

sharanyad reviewed Apr 27, 2018

View reviewed changes

haikuoliu force-pushed the branch_pull_image branch from 8af18ee to d67deff Compare April 27, 2018 23:52

richardpen reviewed May 1, 2018

View reviewed changes

haikuoliu force-pushed the branch_pull_image branch from d67deff to c7090f8 Compare May 2, 2018 17:20

sharanyad reviewed May 2, 2018

View reviewed changes

richardpen approved these changes May 2, 2018

View reviewed changes

haikuoliu force-pushed the branch_pull_image branch from c7090f8 to 8b4cb17 Compare May 2, 2018 21:22

sharanyad approved these changes May 2, 2018

View reviewed changes

engine: introduce a new env var to distinct pull image behavior

a8a9b98

haikuoliu force-pushed the branch_pull_image branch from 8b4cb17 to a8a9b98 Compare May 3, 2018 17:46

haikuoliu merged commit 961b2ef into aws:dev May 4, 2018

haikuoliu mentioned this pull request May 5, 2018

unit test failure: TestStartContainerTimeout #1352

Closed

engine: introduce a new env var to distinct pull image behavior #1348

engine: introduce a new env var to distinct pull image behavior #1348

Conversation

haikuoliu commented Apr 17, 2018 • edited Loading

Summary

Implementation details

Testing

Description for the changelog

Licensing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaithal commented Apr 18, 2018

haikuoliu commented Apr 18, 2018

aaithal commented Apr 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaithal commented Apr 18, 2018

haikuoliu commented Apr 18, 2018 • edited Loading

haikuoliu commented Apr 19, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richardpen Apr 19, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haikuoliu commented Apr 26, 2018

haikuoliu commented Apr 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

haikuoliu commented Apr 17, 2018 •

edited

Loading

aaithal commented Apr 18, 2018 •

edited

Loading

haikuoliu commented Apr 18, 2018 •

edited

Loading

richardpen Apr 19, 2018 •

edited

Loading