tests/provider: Flaky Kinesis Analytics v2 tests on GovCloud

[YakDriver]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

These tests sometimes fail on GovCloud but other times succeed:

TestAccAWSKinesisAnalyticsV2Application_SQLApplicationConfiguration_VPCConfiguration_Delete
TestAccAWSKinesisAnalyticsV2Application_SQLApplicationConfiguration_VPCConfiguration_Add

With this error: InvalidArgumentException: Please check the role provided or validity of S3 location you provided. These do not fail in commercial but may only be because commercial is faster at resolving eventual consistency.

New or Affected Resource(s)

• aws_kinesisanalyticsv2_application

Terraform Configuration Files

Example config:

resource "aws_iam_role" "test" {
  count = 2

  name               = "yaktest.${count.index}"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Principal": {"Service": "firehose.amazonaws.com"}
    },
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Principal": {"Service": "kinesisanalytics.amazonaws.com"}
    },
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Principal": {"Service": "lambda.amazonaws.com"}
    }
  ]
}
EOF
}

resource "aws_iam_policy" "test" {
  name   = "yaktest"
  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["ec2:*"],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": ["firehose:*"],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": ["lambda:*"],
      "Resource": ["*"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:*"],
      "Resource": ["*"]
    }
  ]
}
EOF
}

resource "aws_iam_role_policy_attachment" "test" {
  count = 2

  role       = aws_iam_role.test[count.index].name
  policy_arn = aws_iam_policy.test.arn
}

resource "aws_vpc" "test" {
  cidr_block = "10.0.0.0/16"

  tags = {
    Name = "yaktest"
  }
}

resource "aws_subnet" "test" {
  count = 2

  vpc_id            = aws_vpc.test.id
  cidr_block        = cidrsubnet(aws_vpc.test.cidr_block, 2, count.index)
  availability_zone = data.aws_availability_zones.available.names[count.index]

  tags = {
    Name = "yaktest"
  }
}

resource "aws_security_group" "test" {
  count = 2

  vpc_id = aws_vpc.test.id

  tags = {
    Name = "yaktest"
  }
}

resource "aws_s3_bucket" "test" {
  bucket = "yaktest"
}

resource "aws_s3_bucket_object" "test" {
  count = 2

  bucket = aws_s3_bucket.test.bucket
  key    = "yaktest.${count.index}"
  source = "test-fixtures/flink-app.jar"
}

resource "aws_kinesisanalyticsv2_application" "test" {
  name                   = "yaktest"
  runtime_environment    = "FLINK-1_8"
  service_execution_role = aws_iam_role.test[0].arn

  application_configuration {
    application_code_configuration {
      code_content {
        s3_content_location {
          bucket_arn = aws_s3_bucket.test.arn
          file_key   = aws_s3_bucket_object.test[0].key
        }
      }

      code_content_type = "ZIPFILE"
    }

    vpc_configuration {
      security_group_ids = aws_security_group.test.*.id
      subnet_ids         = aws_subnet.test.*.id
    }
  }
}

Debug Output

TestAccAWSKinesisAnalyticsV2Application_SQLApplicationConfiguration_VPCConfiguration_Delete: resource_aws_kinesisanalyticsv2_application_test.go:2417: Step 1/3 error: Error running apply: 2020/11/09 15:46:43 [DEBUG] Using modified User-Agent: Terraform/0.12.29 HashiCorp-terraform-exec/0.10.0
        
        Error: error creating Kinesis Analytics v2 Application: InvalidArgumentException: Please check the role provided or validity of S3 location you provided. We are unable to get the specified fileKey: yaktest in the specified bucket: yaktest

--- FAIL: TestAccAWSKinesisAnalyticsV2Application_SQLApplicationConfiguration_VPCConfiguration_Delete (23.39s)

References

• #0000

[ewbankkit]

There is already code in the resource to handle IAM eventual consistency which can be updated

terraform-provider-aws/aws/resource_aws_kinesisanalyticsv2_application.go

Lines 2598 to 2640 in be8eff2

	// kinesisAnalyticsV2RetryIAMEventualConsistency retries the specified function for 1 minute
	// if the returned error indicates an IAM eventual consistency issue.
	// If the retries time out the specified function is called one last time.
	func kinesisAnalyticsV2RetryIAMEventualConsistency(f func() (interface{}, error)) (interface{}, error) {
	var output interface{}
	err := resource.Retry(1*time.Minute, func() *resource.RetryError {
	var err error
	output, err = f()
	// Kinesis Stream: https://github.com/hashicorp/terraform-provider-aws/issues/7032
	if isAWSErr(err, kinesisanalyticsv2.ErrCodeInvalidArgumentException, "Kinesis Analytics service doesn't have sufficient privileges") {
	return resource.RetryableError(err)
	}
	// Kinesis Firehose: https://github.com/hashicorp/terraform-provider-aws/issues/7394
	if isAWSErr(err, kinesisanalyticsv2.ErrCodeInvalidArgumentException, "Kinesis Analytics doesn't have sufficient privileges") {
	return resource.RetryableError(err)
	}
	// InvalidArgumentException: Given IAM role arn : arn:aws:iam::123456789012:role/xxx does not provide Invoke permissions on the Lambda resource : arn:aws:lambda:us-west-2:123456789012:function:yyy
	if isAWSErr(err, kinesisanalyticsv2.ErrCodeInvalidArgumentException, "does not provide Invoke permissions on the Lambda resource") {
	return resource.RetryableError(err)
	}
	if err != nil {
	return resource.NonRetryableError(err)
	}
	return nil
	})
	if tfresource.TimedOut(err) {
	output, err = f()
	}
	if err != nil {
	return nil, err
	}
	return output, nil
	}

I can take this.

The same change should be made to the v1 aws_kinesis_analytics_application resource.

[YakDriver]

Great! Thanks @ewbankkit

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!