Unable to download firelens s3 config file

[drbly]

Summary

My ECS service gets stuck starting and stopping a task over and over with this stop reason:
Unable to download firelens s3 config file: unable to download s3 config extra.conf from bucket mybucket: MissingRegion: could not find region configuration

I have given both the task role and execution role the S3 permissions needed to download the config file. I don't think it is an S3 issue, and I'm not sure what "region configuration" it is referring to.

Description

I am trying to use FireLens to output logs from the container into 2 different CloudWatch log streams. I updated my task by adding a new firelens container definition, and updating my app to use firelens for logging. The firelens container has a FirelensConfiguration with a config file in S3.
It has all the S3 permissions it should need.

This is the firelens container definition:

{
	"Essential": true,
	"Image": "amazon/aws-for-fluent-bit:latest",
	"Name": "log_router",
	"FirelensConfiguration": {
		"Type": "fluentbit",
		"Options": {
			"config-file-type": "s3",
			"config-file-value": "arn:aws:s3:::mybucket/extra.conf"
		}
	},
	"LogConfiguration": {
		"LogDriver": "awslogs",
		"Options": {
			"awslogs-group": {"Ref": "apiBackendLogGroup"},
			"awslogs-region": {"Ref": "AWS::Region"},
			"awslogs-stream-prefix": "accessLogs"
		}
	},
	"MemoryReservation": 50
}

Both the task role and the execution role have this S3 policy added to them, suggested by this guide:

{
	"Effect": "Allow",
	"Action": [
		"s3:GetObject"
	],
	"Resource": [
		"arn:aws:s3:::mybucket/extra.conf"
	]
},
{
	"Effect": "Allow",
	"Action": [
		"s3:GetBucketLocation"
	],
	"Resource": [
		"arn:aws:s3:::mybucket"
	]
}

Expected Behavior

ECS service successfully starts the the tasks

Observed Behavior

ECS service gets stuck starting and stopping a task over and over with this stop reason:
Unable to download firelens s3 config file: unable to download s3 config extra.conf from bucket mybucket: MissingRegion: could not find region configuration

Environment Details

Here is the entire task definition:

"apiBackendTask": {
	"Condition": "notFirstRun",
	"Type": "AWS::ECS::TaskDefinition",
	"Properties": {
		"ContainerDefinitions": [
			{
				"Environment": [
					{
						"Name": "SPRING_PROFILES_ACTIVE",
						"Value": {"Ref": "target"}
					},
					{
						"Name": "DB_ENDPOINT",
						"Value": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "jdbc"]}
					},
					{
						"Name": "endpoint",
						"Value": {"Fn::Sub": "cognito-idp.${AWS::Region}.amazonaws.com"}
					},
					{
						"Name": "region",
						"Value": {"Ref": "AWS::Region"}
					},
					{
						"Name": "poolId",
						"Value": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "poolId"]}
					},
					{
						"Name": "identityPoolId",
						"Value": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "identityPoolId"]}
					},
					{
						"Name": "BUCKET_NAME",
						"Value": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "bucket"]}
					}
				],
				"Essential": true,
				"Image": {"Ref": "beimage"},
				"LogConfiguration": {
					"LogDriver": "awsfirelens"
				},
				"MemoryReservation": 320,
				"Name": {
					"Fn::Sub": [
						"${name}-${target}",
						{
							"name": {
								"Fn::FindInMap": ["all", "all", "beName"]
							}
						}
					]
				},
				"PortMappings": [
					{
						"ContainerPort": {"Fn::FindInMap": ["all", "all", "bePort"]}
					}
				],
				"Secrets": [
					{
						"Name": "clientId",
						"ValueFrom": {"Fn::FindInMap": [{"Ref": "target"}, {"Ref": "AWS::Region"}, "clientId"]}
					}
				]
			},
			{
				"Essential": true,
				"Image": "amazon/aws-for-fluent-bit:latest",
				"Name": "log_router",
				"FirelensConfiguration": {
					"Type": "fluentbit",
					"Options": {
						"config-file-type": "s3",
						"config-file-value": "arn:aws:s3:::mybucket/extra.conf"
					}
				},
				"LogConfiguration": {
					"LogDriver": "awslogs",
					"Options": {
						"awslogs-group": {"Ref": "apiBackendLogGroup"},
						"awslogs-region": {"Ref": "AWS::Region"},
						"awslogs-stream-prefix": "accessLogs"
					}
				},
				"MemoryReservation": 50
			}
		],
		"ExecutionRoleArn": {"Ref": "executionRole"},
		"NetworkMode": "bridge",
		"RequiresCompatibilities": ["EC2"],
		"Tags": [
			{
				"Key": "Owner",
				"Value": {"Ref": "contact"}
			}
		],
		"TaskRoleArn": {"Ref": "containerRole"}
	}
}

[petderek]

Thanks for reaching out to us, I'll try to see if I can repro this on my end.

Which region is your task running in? and in which region is the S3 bucket located?

[drbly]

It is all in us-east-1.

[fenxiong]

Thanks for reporting. I found that there's a bug such that bucket in us-east-1 is not working. Will work on a fix for it.

As a workaround, you should be able to use a bucket that's in a region other than us-east-1.

[fenxiong]

This is fixed in agent 1.37.0. Please update to latest version, and if you still find issue let us know. Thanks.