Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce ErrorResponse to TMDEv4 taskWithTags responses #2789

Merged
merged 1 commit into from
Jan 12, 2021

Conversation

chienhanlin
Copy link
Contributor

@chienhanlin chienhanlin commented Jan 7, 2021

Summary

This PR adds a new type ErrorResponse into the task metadata endpoint version 4 (TMDEv4) taskWithTags responses. When the path {ECS_CONTAINER_METADATA_URI_V4}/taskWithTag is queried, Agent writes a json response with task metadata, task and container instance tags that can be retrieved from ECS ListTagsForResource API, and errors returned from the ECS API.

To date, Agent shows a partial/no list of tags in TMDEv2/v3/v4 taskWithTags responses if it cannot get a complete list of tags from ECS. As no error code, error message or status code is mentioned in responses, it hinders customers from differentiating whether there is no task/container tag, or an exception has been encountered. To resolve this issue, An ErrorResponse with an error field, an error code, an error message, a status code, a request ID and a resources ARN is introduced to TMDEv4 taskWithTags responses.

Implementation details

  1. Define ErrorResponse, the schema for an error response json object, and make it be part of the TaskResponse
  2. Pass the boolean flag includeV4Metadata to function propagateTagsToMetadata
  3. Append ErrorResponses to the TaskResponse if includeV4Metadata is true and an ErrorResponse exists
  4. Define function newErrorResponse to return a new ErrorResponse object based on the error and the resource ARN
  5. Define function metadataErrorHandling to format errors for ecs-agent.log, and call function newErrorResponse for TMDEv4

Testing

Unit tests

New tests cover the changes: yes
A new unit test TestTaskResponseWithV4TagsError is added to cover this change.

=== RUN   TestTaskResponseWithV4TagsError
1610057606730023000 [Error] V2 response: unable to get 'ContainerInstanceTags' for 'containerInstance-test': ThrottlingException: Rate exceeded status code: 400, request id: cef9da77-aee7-431d-84d5-f92b2d342c51
caused by: 
1610057606730063000 [Error] V2 response: unable to get 'TaskTags' for 't1': ThrottlingException: Rate exceeded status code: 400, request id: 45dbbc67-0c60-4248-855e-14fdf4c11870
caused by: 
--- PASS: TestTaskResponseWithV4TagsError (0.00s)
PASS
ok      github.com/aws/amazon-ecs-agent/agent/handlers/v2       0.832s

Manual tests

TMDEv4 taskWithTags

  1. Response when no exception is hit
{
   "Cluster":"default",
   "TaskARN":"xxx",
   ...
   "TaskTags":{
      "SleeppyKey":"SleeppyValue",
      "SleepyKey":"SleepyValue",
      "SleepyKeyy":"SleepyValueee",
      "SleepyyKey":"SleepyyValue",
      "aws:ecs:clusterName":"default"
   },
   "ContainerInstanceTags":{
      "tag_key":"tag_value"
   },
   "LaunchType":"EC2",
   "Containers":[{...}]
}
  1. Response when ThrottlingException is encountered for TaskTags
{
   "Cluster":"default",
   "TaskARN":"xxx",
   "ContainerInstanceTags":{
      "tag_key":"tag_value"
   },
   "LaunchType":"EC2",
   "Errors":[
      {
         "ErrorField":"TaskTags",
         "ErrorCode":"ThrottlingException",
         "ErrorMessage":"Rate exceeded",
         "StatusCode":400,
         "RequestId":"xxx",
         "ResourceARN":"taskARN"
      }
   ],
   "Containers":[{...}]
}

ecs-agent.log

level=info time=xxx msg="V4 taskMetadata handler: Writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': ThrottlingException: Rate exceeded\n\tstatus code: 400, request id: xxx" module=response.go
  1. Response when AccessDeniedException is encountered for both ContainerInstanceTags and TaskTags
{
   "Cluster":"default",
   "TaskARN":"xxx",
   ...
   "LaunchType":"EC2",
   "Errors":[
      {
         "ErrorField":"ContainerInstanceTags",
         "ErrorCode":"AccessDeniedException",
         "ErrorMessage":"User: iamARN is not authorized to perform: ecs:ListTagsForResource on resource: containerInstanceARN",
         "StatusCode":400,
         "RequestId":"xxx",
         "ResourceARN":"containerInstanceARN"
      },
      {
         "ErrorField":"TaskTags",
         "ErrorCode":"AccessDeniedException",
         "ErrorMessage":"User: iamARN is not authorized to perform: ecs:ListTagsForResource on resource: taskARN",
         "StatusCode":400,
         "RequestId":"xxx",
         "ResourceARN":"taskARN"
      }
   ],
   "Containers":[{...}]
}

ecs-agent.log

level=info time=xxx msg="V4 taskMetadata handler: Writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'ContainerInstanceTags' for 'containerInstanceARN': AccessDeniedException: User: iamARN is not authorized to perform: ecs:ListTagsForResource on resource: containerInstanceARN" module=response.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': AccessDeniedException: User: iamARN is not authorized to perform: ecs:ListTagsForResource on resource: taskARN" module=response.go
  1. Response when InvalidParameterException is encountered for TaskTags
{
   "Cluster":"default",
   "TaskARN":"xxx",
   ...
   "ContainerInstanceTags":{
      "tag_key":"tag_value"
   },
   "LaunchType":"EC2",
   "Errors":[
      {
         "ErrorField":"TaskTags",
         "ErrorCode":"InvalidParameterException",
         "ErrorMessage":"The specified task is stopped. Specify a running task and try again.",
         "StatusCode":400,
         "RequestId":"xxx",
         "ResourceARN":"taskARN"
      }
   ],
   "Containers":[{...}]
}

ecs-agent.log

level=info time=xxx msg="V4 taskMetadata handler: Writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': InvalidParameterException: The specified task is stopped. Specify a running task and try again." module=response.go
  1. Response when a non-awserr is encountered
{
   "Cluster":"default",
   "TaskARN":"xxx",
   ...
   "ContainerInstanceTags":{
      "tag_key":"tag_value"
   },
   "LaunchType":"EC2",
   "Errors":[
      {
         "ErrorField":"TaskTags",
         "ErrorMessage":"This is a non-awserr for testing: unable to find TaskTags for 'taskARN'",
         "ResourceARN":"taskARN"
      }
   ],
   "Containers":[{...}]
}

ecs-agent.log

level=info time=xxx msg="V4 taskMetadata handler: Writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': This is a non-awserr for testing: unable to find TaskTags for 'taskARN'" module=response.go

TMDEv3 taskWithTags

  1. Response when no exception is hit
{
   "Cluster":"default",
   "TaskARN":"xxx",
   ...
   "Containers":[{...}]
   ... 
   "TaskTags":{
      "SleeppyKey":"SleeppyValue",
      "SleepyKey":"SleepyValue",
      "SleepyKeyy":"SleepyValueee",
      "SleepyyKey":"SleepyyValue",
      "aws:ecs:clusterName":"default"
   },
   "ContainerInstanceTags":{
      "tag_key":"tag_value"
   }
}
  1. ecs-agent.log when ThrottlingException is encountered for ContainerInstanceTags
level=info time=xxx msg="V3 task metadata handler: writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'ContainerInstanceTags' for 'containerInstanceARN': ThrottlingException: Rate exceeded\n\tstatus code: 400, request id: xxx" module=response.go
  1. ecs-agent.log when AccessDeniedException is encountered for both ContainerInstanceTags and TaskTags
level=info time=xxx msg="V3 task metadata handler: writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'ContainerInstanceTags' for 'containerInstanceARN': AccessDeniedException: User: iamARN is not authorized to perform: ecs:ListTagsForResource on resource: containerInstanceARN" module=response.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': AccessDeniedException: User:iamARN is not authorized to perform: ecs:ListTagsForResource on resource: taskARN" module=response.go
  1. ecs-agent.log when InvalidParameterException is encountered for TaskTags
level=info time=xxx msg="V3 task metadata handler: writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': InvalidParameterException: The specified task is stopped. Specify a running task and try again." module=response.go
  1. ecs-agent.log when a non-awserr is encountered
level=info time=xxx msg="V3 task metadata handler: writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': This is a non-awserr for testing: unable to find TaskTags for 'taskARN'" module=response.go

Description for the changelog

Enhancement - Add error responses into TMDEv4 taskWithTags responses

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

// metadataErrorHandling writes an error to the logger, and append an error response
// to V4 metadata endpoint task response
func metadataErrorHandling(resp *TaskResponse, err error, field, resourceARN string, includeV4Metadata bool) {
seelog.Errorf("V2 response: unable to get '%s' for '%s': %s", field, resourceARN, err.Error())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: since this function is used by v3 and v4 task metadata endpoints, it might be confusing to prepend the log with "V2 response". Maybe something more accurate like "Task Metadata error:"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the error log message as suggested. Thanks!

level=info time=xxx msg="V4 taskMetadata handler: Writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'ContainerInstanceTags' for 'containerInstanceARN': AccessDeniedException: User: iamARN is not authorized to perform: ecs:ListTagsForResource on resource: containerInstanceARN" module=response.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': AccessDeniedException: User: iamARN is not authorized to perform: ecs:ListTagsForResource on resource: taskARN" module=response.go

func newErrorResponse(err error, field, resourceARN string) (*ErrorResponse, error) {
if awsErr, ok := err.(awserr.Error); ok {
if reqErr, ok := err.(awserr.RequestFailure); ok {
errResp := &ErrorResponse{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: no need for the errResp variable, you can return &ErrorResponse{...} directly

}
}

return nil, errors.Errorf("Unable to get an error response for resource '%s'", resourceARN)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we return something like below if the error is not awserr.Error?

return &ErrorResponse{
				ErrorField:   field,
				ErrorMessage: err.Error(),
				ResourceARN:  resourceARN,
			}

Copy link
Contributor Author

@chienhanlin chienhanlin Jan 8, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should. Update function newErrorResponse with a base ErrorResponse as code snippets below.

// newErrorResponse creates a new error response
func newErrorResponse(err error, field, resourceARN string) *ErrorResponse {
	errResp := &ErrorResponse{
		ErrorField:   field,
		ErrorMessage: err.Error(),
		ResourceARN:  resourceARN,
	}

	if awsErr, ok := err.(awserr.Error); ok {
		errResp.ErrorCode = awsErr.Code()
		errResp.ErrorMessage = awsErr.Message()
		if reqErr, ok := err.(awserr.RequestFailure); ok {
			errResp.StatusCode = reqErr.StatusCode()
			errResp.RequestId = reqErr.RequestID()
		}
	}

	return errResp
}

A mock non-awserr Error is tested, and the corresponding responses are shown as follows.
TMDE v4

{
   "Cluster":"default",
   "TaskARN":"xxx",
   ...
   "ContainerInstanceTags":{
      "tag_key":"tag_value"
   },
   "LaunchType":"EC2",
   "Errors":[
      {
         "ErrorField":"TaskTags",
         "ErrorMessage":"This is a non-awserr for testing: unable to find TaskTags for 'taskARN'",
         "ResourceARN":"taskARN"
      }
   ],
   "Containers":[{...}]
}

TMDE v3

level=info time=xxx msg="V3 task metadata handler: writing response for task 'taskARN'" module=task_metadata_handler.go
level=error time=xxx msg="Task Metadata error: unable to get 'TaskTags' for 'taskARN': This is a non-awserr for testing: unable to find TaskTags for 'taskARN'" module=response.go

Thanks!

@chienhanlin chienhanlin merged commit 6962e07 into aws:dev Jan 12, 2021
@chienhanlin chienhanlin deleted the taskWithTagsTMDEV4 branch January 12, 2021 07:09
@ubhattacharjya ubhattacharjya added this to the 1.50.0 milestone Jan 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants