Changing Requires to Wants to make ECS more resilient and check if docker service is running in ECS service file

[mye956]

Summary

The goal of this PR is to make the ECS service more resilient in regards to when docker is up and available. There can be cases where docker goes down during instance bootstrapping (e.g. upgrades/restarts) and ECS service is trying to start up. However, it can't since docker is down and will fail due to a service dependency error which isn't taken into consideration for the restart policy that we defined. In other words, since the ECS service never starts up it's not consider a failure. To try to address this inconsistent race condition, we will be changing Requires to Wants and will have a check if docker is up and running during PreStart for the ECS service.

The reason why we're switching over to Wants is because it's less restrictive than Requires. Essentially, if the units within the ecs.service file are not found or fail to start, the current unit(ECS) will continue to function. This is the recommended way to configure most dependency relationships. Again, this implies a parallel activation unless modified by other directives. (ref)

In addition, we'll also have the ECS service to check if docker is running as part of its PreStart so that it can fail a bit faster. Otherwise, it will fail once ECS reaches its Start phase where it tries to start up the ECS Agent container.

Implementation details

• Changed Requires to Wants. These were made in the following files:
  • packaging/amazon-linux-ami-integrated/ecs.service
  • packaging/generic-deb-integrated/debian/ecs.service
  • packaging/generic-rpm-integrated/ecs.service
• Added check if docker is up and running within packaging/amazon-linux-ami-integrated/ecs.service via systemctl is-active docker
  • If it's not, we will exit with error code 1 (Note: We've defined ECS to ignore exit code 5)
  • We won't include this check in the other service files as the location of systemctl can vary across distros. Instead, we will look to change our public docs regarding installing ECS on non-AL AMIs (ref)

Testing

Built a custom AMI off of an older ami (specifically 20220304) where docker would do an auto-upgrade.

Load tested with 16600 instances and all were able to start up ECS after a docker upgrade

NEW INSTANCES LAUNCHED = 100 ; TOTAL INSTANCES LAUNCHED = 16600
TOTAL INSTANCES REGISTERED = 16600 / 16600

A few other edge cases that were tested:

• Stopping both ECS + Docker -> Start back up ECS
  Behavior: Once ECS tried starting back up, docker will also start back up
• Stopping just Docker while ECS is running
  Behavior:ECS service will fail. After a bit, ECS will restart which will also bring docker back up
• Disabling just Docker while ECS is running
  Behavior: ECS service will fail. After a bit, ECS will restart which will also bring docker back up
• Run ECS service on a vanilla AL2 AMI
  Behavior: ECS service will never reach a running state and will fail due to docker not being present when it tries starting up the ECS Agent docker container. It will retry starting up every 10 second which is expected based on what we’ve defined in our systemd service unit file for ECS.

Old Behavior:

• Stopping both ECS + Docker -> Start back up ECS
  Behavior: Once ECS starts back up, docker will also start back up
• Stopping just Docker while ECS is running
  Behavior: ECS will also go down and will remain down. It will only start back up if a user manually starts it back up.
• Disabling just Docker while ECS is running
  Behavior: ECS will also go down and will remain down. It will only start back up if a user manually starts it back up.
• Run ECS on a vanilla AL2 AMI
  Behavior: ECS fails to start up at all.

Description for the changelog

Enhancement: Change ECS service file from requires to wants and checking if docker is running during ECS service PreStart

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[yinyic]

Could we also add to the testing section what the current (old) behavior is for comparison? Otherwise the change looks good. Thanks Mike!

[ryzr]

Any chance this is related?
#4270

My ecs-agent service suddenly stopped on all of my instances and doesn't attempt to restart. Not sure if possibly due to SSM automatically applying updates?

EDIT: reverting this change fixed my issue

[juangc-wavyssa]

Same issue here. How can we force the agent to use 1.84.0? We are having this issue for all our 49 ECS.Happened on last two sundays in all ours ECS, and we had to reboot all the agents manually, and yesterday, only on one ECS.

[danehlim]

@ryzr @juangc-wavyssa thank you for reaching out. Our team is aware of this issue and currently in the process of addressing it as part of #4277.