Initialize-ECSAgent with -Version latest no longer works on Windows ECS AMI 2019

[alexspence]

Summary

Using -Version latest with Initialize-Agent in the ECSTools Powershell module does not work.

Description

Previously - to ensure I'm always installing the latest ECS Agent for my ECS hosts, I used the following commands for bootstrapping the ECS Host:

commands:
         00_import_powershell_module:
           command: !Sub powershell.exe -Command Import-Module ECSTools
         01_add_instance_to_cluster:
           command: !Sub powershell.exe -Command Initialize-ECSAgent -Cluster ${ECSCluster} -EnableTaskIAMRole -Version latest

This should work as documented in the readme of this repo:

PS C:\> # use agentVersion = "latest" for the latest available agent version

This command fails with an S3 403 Error. Probably becuase it is incorrectly prefixing the string "latest" with a v.

This is the file its trying to download that fails:
https://s3.amazonaws.com/amazon-ecs-agent/ecs-agent-windows-vlatest.zip

If you take the v out, the download succeeds:
https://s3.amazonaws.com/amazon-ecs-agent/ecs-agent-windows-latest.zip

I believe the offending code is on line 931 of EcsTools.psm1. Seems like it just needs to check if the string is "latest" before adding the v prefix.

 if (-not [string]::IsNullOrEmpty($Version)) {
            if ($Version -notlike 'v*') {
                $Version = 'v' + $Version
            }
            $Version = $Version.ToLower()
        } 

Expected Behavior

Initialize-ECSAgent -Cluster cluster -EnableTaskIAMRole -Version latest correctly installs the latest ECS Agent

Observed Behavior

Ecs Agent Fails to install with an 403 Error Downloading from S3.

Environment Details

AMI ID
Windows_Server-2019-English-Full-ECS_Optimized-2019.05.10 (ami-09a6b4fc9786621ef)

Supporting Log Snippets

PS C:\Users\Administrator> Initialize-ECSAgent -Cluster cluster -EnableTaskIAMRole -Version latest
2019-09-24T00:00:54Z - [INFO]:Runtime is already installed.
2019-09-24T00:00:54Z - [INFO]:Docker version 18.09.4, build c3516c43ef
2019-09-24T00:00:54Z - [INFO]:Configuring ECS Host...
2019-09-24T00:00:54Z - [INFO]:Checking Hyper-V Network adapter
2019-09-24T00:00:54Z - [INFO]:Default vEthernet adapter found for nat. Using this adapter.
2019-09-24T00:00:54Z - [INFO]:VMNetwork adapter found with mac: 
2019-09-24T00:00:54Z - [INFO]:Checking for network adatper with mac: 
2019-09-24T00:00:54Z - [INFO]:Network adapter found.
2019-09-24T00:00:54Z - [INFO]:Network adapter found with mac  on interface 11
2019-09-24T00:00:54Z - [INFO]:Getting subnet info from docker...
2019-09-24T00:00:54Z - [INFO]:Docker subnet: 
2019-09-24T00:00:54Z - [INFO]:Docker gateway: 
2019-09-24T00:00:56Z - [INFO]:Getting net ip address
2019-09-24T00:00:56Z - [INFO]:IP address available. 
Name                           Value                                                                                   
----                           -----                                                                                   
PrefixLength                   32                                                                                      
IPAddress                                                                                             
InterfaceIndex                 11
2019-09-24T00:00:57Z - [INFO]:netsh interface portproxy show all
2019-09-24T00:00:57Z - [INFO]:
2019-09-24T00:00:57Z - [INFO]:Listen on ipv4:             Connect to ipv4:
2019-09-24T00:00:57Z - [INFO]:
2019-09-24T00:00:57Z - [INFO]:Address         Port        Address         Port
2019-09-24T00:00:57Z - [INFO]:--------------- ----------  --------------- ----------
2019-09-24T00:00:57Z - [INFO]:
2019-09-24T00:00:57Z - [INFO]:
2019-09-24T00:00:57Z - [INFO]:Setting up new ipv4 interface proxy to forward traffic...
2019-09-24T00:00:57Z - [INFO]:  
2019-09-24T00:00:57Z - [INFO]:  
2019-09-24T00:00:57Z - [INFO]:
2019-09-24T00:00:57Z - [INFO]:Checking port forwarding...
2019-09-24T00:00:59Z - [INFO]:netsh interface portproxy show all
2019-09-24T00:00:59Z - [INFO]:
2019-09-24T00:00:59Z - [INFO]:Listen on ipv4:             Connect to ipv4:
2019-09-24T00:00:59Z - [INFO]:
2019-09-24T00:00:59Z - [INFO]:Address         Port        Address         Port
2019-09-24T00:00:59Z - [INFO]:--------------- ----------  --------------- ----------
2019-09-24T00:00:59Z - [INFO]:
2019-09-24T00:00:59Z - [INFO]:
2019-09-24T00:00:59Z - [INFO]:TcpTestSucceeded: True
2019-09-24T00:00:59Z - [INFO]:Port forwarding setup complete.
2019-09-24T00:00:59Z - [INFO]:ECS Host setup complete.
2019-09-24T00:01:00Z - [INFO]:Configuring agent environment variables... complete
2019-09-24T00:01:00Z - [INFO]:Downloading agent...
2019-09-24T00:01:00Z - [INFO]:Downloading file from S3: "https://s3.amazonaws.com/amazon-ecs-agent/ecs-agent-windows-vlatest.zip"
PS C:\Users\Administrator> TerminatingError(Invoke-RestMethod): "The remote server returned an error: (403) Forbidden."
2019-09-24T00:01:00Z - [ERROR]:There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden.
PS C:\Users\Administrator> TerminatingError(): "There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden."
>> TerminatingError(): "There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden."
>> TerminatingError(): "There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden."
>> TerminatingError(): "There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden."
>> TerminatingError(): "There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden."
>> TerminatingError(): "There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden."
There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden.
There was an issue downloading from S3. Message: The remote server returned an error: (403) Forbidden.
At C:\Program Files\WindowsPowerShell\Modules\ECSTools\ECSTools.psm1:68 char:5
+     Throw $Message
+     ~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (There was an is...403) Forbidden.:String) [], RuntimeException
    + FullyQualifiedErrorId : There was an issue downloading from S3. Message: The remote server returned an error: (4
   03) Forbidden.

[ubhattacharjya]

Thanks for letting us know. We are looking into this.

[sandeepindraganti]

There is a known issue using the “-Version” tag with the string “latest”. Using “-Version ‘latest’” will result in an error. To use the latest version, remove the “-Version” flag, and the module will default to the latest version unless there is a cached version available. We are working on a fix for this which is currently planned to rollout during next release.

[Smuggla]

We have added the following to our User Data (using the latest Amazon provided ECS Opimtized AMI, Windows_Server-2019-English-Full-ECS_Optimized-2019.10.09 at the time) as the 'cached version' is out of date.

<powershell>
Remove-Item -Recurse C:\ProgramData\Amazon\ECS\Cache
Import-Module ECSTools
Initialize-ECSAgent -Cluster '${CLUSTER_NAME}' -EnableTaskIAMRole
</powershell>

[ellenthsu]

This has been released in the November Windows AMI: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-ami-versions.html#windows-2019-ami-agent-versions

#2239

[mdunc]

Is this ECSTools module available anywhere outside of the official Windows ECS AMI? I've been searching, but haven't found anything except AWS.Tools.ECS which is something totally different despite the similar name. It would make building a custom Windows ECS AMI much easier if that module was available outside of the AMI (for various business and technical reasons, we cannot base our AMI on the on the official ECS optimized image even though that would make more sense.).