aws_vpc_route_table: retry read on new resource #25793

patrickdillon · 2022-07-12T18:37:42Z

We are seeing failures to read a newly created Route Table:

time="2022-07-12T14:52:59Z" level=error
time="2022-07-12T14:52:59Z" level=error msg="Error: error reading Route Table (rtb-0f35a4184bea5e3dd): couldn't find resource"
time="2022-07-12T14:52:59Z" level=error
time="2022-07-12T14:52:59Z" level=error msg="  with module.vpc.aws_route_table.private_routes[0],"
time="2022-07-12T14:52:59Z" level=error msg="  on vpc/vpc-private.tf line 1, in resource \"aws_route_table\" \"private_routes\":"
time="2022-07-12T14:52:59Z" level=error msg="   1: resource \"aws_route_table\" \"private_routes\" {"
time="2022-07-12T14:52:59Z" level=error

#22927 instituted retry logic for route table associations. I'm just applying that same logic here.
We should retry reads in those cases to alow for eventual consistency.

Community Note

Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for pull request followers and do not help prioritize the request

Closes #25791

Output from acceptance testing:

$ make testacc TESTS=TestAccVPCRouteTable PKG=ec2
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/ec2/... -v -count 1 -parallel 20 -run='TestAccVPCRouteTable'  -timeout 180m
=== RUN   TestAccVPCRouteTableAssociation_Subnet_basic
=== PAUSE TestAccVPCRouteTableAssociation_Subnet_basic
=== RUN   TestAccVPCRouteTableAssociation_Subnet_changeRouteTable
=== PAUSE TestAccVPCRouteTableAssociation_Subnet_changeRouteTable
=== RUN   TestAccVPCRouteTableAssociation_Gateway_basic
=== PAUSE TestAccVPCRouteTableAssociation_Gateway_basic
=== RUN   TestAccVPCRouteTableAssociation_Gateway_changeRouteTable
=== PAUSE TestAccVPCRouteTableAssociation_Gateway_changeRouteTable
=== RUN   TestAccVPCRouteTableAssociation_disappears
=== PAUSE TestAccVPCRouteTableAssociation_disappears
=== RUN   TestAccVPCRouteTableDataSource_basic
=== PAUSE TestAccVPCRouteTableDataSource_basic
=== RUN   TestAccVPCRouteTableDataSource_main
=== PAUSE TestAccVPCRouteTableDataSource_main
=== RUN   TestAccVPCRouteTable_basic
=== PAUSE TestAccVPCRouteTable_basic
=== RUN   TestAccVPCRouteTable_disappears
=== PAUSE TestAccVPCRouteTable_disappears
=== RUN   TestAccVPCRouteTable_Disappears_subnetAssociation
=== PAUSE TestAccVPCRouteTable_Disappears_subnetAssociation
=== RUN   TestAccVPCRouteTable_ipv4ToInternetGateway
=== PAUSE TestAccVPCRouteTable_ipv4ToInternetGateway
=== RUN   TestAccVPCRouteTable_ipv4ToInstance
=== PAUSE TestAccVPCRouteTable_ipv4ToInstance
=== RUN   TestAccVPCRouteTable_ipv6ToEgressOnlyInternetGateway
=== PAUSE TestAccVPCRouteTable_ipv6ToEgressOnlyInternetGateway
=== RUN   TestAccVPCRouteTable_tags
=== PAUSE TestAccVPCRouteTable_tags
=== RUN   TestAccVPCRouteTable_requireRouteDestination
=== PAUSE TestAccVPCRouteTable_requireRouteDestination
=== RUN   TestAccVPCRouteTable_requireRouteTarget
=== PAUSE TestAccVPCRouteTable_requireRouteTarget
=== RUN   TestAccVPCRouteTable_Route_mode
=== PAUSE TestAccVPCRouteTable_Route_mode
=== RUN   TestAccVPCRouteTable_ipv4ToTransitGateway
=== PAUSE TestAccVPCRouteTable_ipv4ToTransitGateway
=== RUN   TestAccVPCRouteTable_ipv4ToVPCEndpoint
=== PAUSE TestAccVPCRouteTable_ipv4ToVPCEndpoint
=== RUN   TestAccVPCRouteTable_ipv4ToCarrierGateway
=== PAUSE TestAccVPCRouteTable_ipv4ToCarrierGateway
=== RUN   TestAccVPCRouteTable_ipv4ToLocalGateway
=== PAUSE TestAccVPCRouteTable_ipv4ToLocalGateway
=== RUN   TestAccVPCRouteTable_ipv4ToVPCPeeringConnection
=== PAUSE TestAccVPCRouteTable_ipv4ToVPCPeeringConnection
=== RUN   TestAccVPCRouteTable_vgwRoutePropagation
=== PAUSE TestAccVPCRouteTable_vgwRoutePropagation
=== RUN   TestAccVPCRouteTable_conditionalCIDRBlock
=== PAUSE TestAccVPCRouteTable_conditionalCIDRBlock
=== RUN   TestAccVPCRouteTable_ipv4ToNatGateway
=== PAUSE TestAccVPCRouteTable_ipv4ToNatGateway
=== RUN   TestAccVPCRouteTable_IPv6ToNetworkInterface_unattached
=== PAUSE TestAccVPCRouteTable_IPv6ToNetworkInterface_unattached
=== RUN   TestAccVPCRouteTable_IPv4ToNetworkInterfaces_unattached
=== PAUSE TestAccVPCRouteTable_IPv4ToNetworkInterfaces_unattached
=== RUN   TestAccVPCRouteTable_vpcMultipleCIDRs
=== PAUSE TestAccVPCRouteTable_vpcMultipleCIDRs
=== RUN   TestAccVPCRouteTable_vpcClassicLink
=== PAUSE TestAccVPCRouteTable_vpcClassicLink
=== RUN   TestAccVPCRouteTable_gatewayVPCEndpoint
=== PAUSE TestAccVPCRouteTable_gatewayVPCEndpoint
=== RUN   TestAccVPCRouteTable_multipleRoutes
=== PAUSE TestAccVPCRouteTable_multipleRoutes
=== RUN   TestAccVPCRouteTable_prefixListToInternetGateway
=== PAUSE TestAccVPCRouteTable_prefixListToInternetGateway
=== RUN   TestAccVPCRouteTablesDataSource_basic
=== PAUSE TestAccVPCRouteTablesDataSource_basic
=== CONT  TestAccVPCRouteTableAssociation_Subnet_basic
=== CONT  TestAccVPCRouteTable_Disappears_subnetAssociation
=== CONT  TestAccVPCRouteTableDataSource_basic
=== CONT  TestAccVPCRouteTableAssociation_Gateway_basic
=== CONT  TestAccVPCRouteTable_ipv4ToTransitGateway
=== CONT  TestAccVPCRouteTablesDataSource_basic
=== CONT  TestAccVPCRouteTable_prefixListToInternetGateway
=== CONT  TestAccVPCRouteTable_multipleRoutes
=== CONT  TestAccVPCRouteTable_gatewayVPCEndpoint
=== CONT  TestAccVPCRouteTable_vpcClassicLink
=== CONT  TestAccVPCRouteTable_vpcMultipleCIDRs
=== CONT  TestAccVPCRouteTable_IPv4ToNetworkInterfaces_unattached
=== CONT  TestAccVPCRouteTable_IPv6ToNetworkInterface_unattached
=== CONT  TestAccVPCRouteTable_ipv4ToNatGateway
=== CONT  TestAccVPCRouteTable_conditionalCIDRBlock
=== CONT  TestAccVPCRouteTable_vgwRoutePropagation
=== CONT  TestAccVPCRouteTableAssociation_Gateway_changeRouteTable
=== CONT  TestAccVPCRouteTable_ipv4ToVPCPeeringConnection
=== CONT  TestAccVPCRouteTable_ipv4ToLocalGateway
=== CONT  TestAccVPCRouteTableAssociation_disappears
=== CONT  TestAccVPCRouteTable_ipv4ToLocalGateway
    acctest.go:1323: skipping since no Outposts found
--- SKIP: TestAccVPCRouteTable_ipv4ToLocalGateway (1.68s)
=== CONT  TestAccVPCRouteTable_ipv4ToCarrierGateway
    wavelength_carrier_gateway_test.go:196: skipping since no Wavelength Zones are available
--- SKIP: TestAccVPCRouteTable_ipv4ToCarrierGateway (0.13s)
=== CONT  TestAccVPCRouteTable_ipv4ToVPCEndpoint
--- PASS: TestAccVPCRouteTable_vpcClassicLink (28.43s)
=== CONT  TestAccVPCRouteTable_basic
--- PASS: TestAccVPCRouteTablesDataSource_basic (31.31s)
=== CONT  TestAccVPCRouteTable_disappears
--- PASS: TestAccVPCRouteTable_Disappears_subnetAssociation (33.22s)
=== CONT  TestAccVPCRouteTable_tags
--- PASS: TestAccVPCRouteTableDataSource_basic (34.29s)
=== CONT  TestAccVPCRouteTable_Route_mode
--- PASS: TestAccVPCRouteTable_ipv4ToVPCPeeringConnection (37.15s)
=== CONT  TestAccVPCRouteTable_requireRouteTarget
--- PASS: TestAccVPCRouteTableAssociation_Subnet_basic (38.43s)
=== CONT  TestAccVPCRouteTable_requireRouteDestination
--- PASS: TestAccVPCRouteTableAssociation_Gateway_basic (40.84s)
=== CONT  TestAccVPCRouteTableAssociation_Subnet_changeRouteTable
--- PASS: TestAccVPCRouteTable_prefixListToInternetGateway (41.49s)
=== CONT  TestAccVPCRouteTable_ipv4ToInstance
--- PASS: TestAccVPCRouteTableAssociation_disappears (44.06s)
=== CONT  TestAccVPCRouteTable_ipv6ToEgressOnlyInternetGateway
--- PASS: TestAccVPCRouteTable_gatewayVPCEndpoint (49.16s)
=== CONT  TestAccVPCRouteTableDataSource_main
--- PASS: TestAccVPCRouteTable_IPv6ToNetworkInterface_unattached (50.28s)
=== CONT  TestAccVPCRouteTable_ipv4ToInternetGateway
--- PASS: TestAccVPCRouteTable_requireRouteTarget (14.41s)
--- PASS: TestAccVPCRouteTable_disappears (23.73s)
--- PASS: TestAccVPCRouteTable_vpcMultipleCIDRs (55.21s)
--- PASS: TestAccVPCRouteTable_basic (28.76s)
--- PASS: TestAccVPCRouteTableAssociation_Gateway_changeRouteTable (60.58s)
--- PASS: TestAccVPCRouteTable_conditionalCIDRBlock (67.48s)
--- PASS: TestAccVPCRouteTableDataSource_main (18.90s)
--- PASS: TestAccVPCRouteTableAssociation_Subnet_changeRouteTable (43.05s)
--- PASS: TestAccVPCRouteTable_tags (57.71s)
--- PASS: TestAccVPCRouteTable_ipv6ToEgressOnlyInternetGateway (47.35s)
--- PASS: TestAccVPCRouteTable_vgwRoutePropagation (91.87s)
--- PASS: TestAccVPCRouteTable_ipv4ToInternetGateway (41.86s)
--- PASS: TestAccVPCRouteTable_IPv4ToNetworkInterfaces_unattached (93.82s)
--- PASS: TestAccVPCRouteTable_Route_mode (62.47s)
--- PASS: TestAccVPCRouteTable_ipv4ToInstance (119.31s)
--- PASS: TestAccVPCRouteTable_multipleRoutes (217.35s)
--- PASS: TestAccVPCRouteTable_ipv4ToNatGateway (229.68s)
--- PASS: TestAccVPCRouteTable_ipv4ToVPCEndpoint (299.82s)
--- PASS: TestAccVPCRouteTable_ipv4ToTransitGateway (322.49s)
--- PASS: TestAccVPCRouteTable_requireRouteDestination (290.58s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/ec2	329.134s

...

We are seeing failures when trying to read a newly created resource. We should retry reads in those cases to allow for eventual consistency.

github-actions

Welcome @patrickdillon 👋

It looks like this is your first Pull Request submission to the Terraform AWS Provider! If you haven’t already done so please make sure you have checked out our CONTRIBUTING guide and FAQ to make sure your contribution is adhering to best practice and has all the necessary elements in place for a successful approval.

Also take a look at our FAQ which details how we prioritize Pull Requests for inclusion.

Thanks again, and welcome to the community! 😃

ewbankkit · 2022-07-15T16:15:11Z

@patrickdillon Thanks for the contribution 🎉 👏.

For the aws_route resource we want to address EC2 eventual consistency on resource creation via the call to WaitRouteTableReady:

terraform-provider-aws/internal/service/ec2/vpc_route_table.go

Lines 188 to 190 in 518f833

    
           if _, err := WaitRouteTableReady(conn, d.Id(), d.Timeout(schema.TimeoutCreate)); err != nil { 
        
           	return fmt.Errorf("error waiting for Route Table (%s) to become available: %w", d.Id(), err) 
        
           }

So instead of modifying resourceRouteTableRead could you please add ContinuousTargetOccurence: 2, to the StateChangeConf:

terraform-provider-aws/internal/service/ec2/wait.go

Lines 699 to 706 in 518f833

    
           func WaitRouteTableReady(conn *ec2.EC2, id string, timeout time.Duration) (*ec2.RouteTable, error) { 
        
           	stateConf := &resource.StateChangeConf{ 
        
           		Pending:        []string{}, 
        
           		Target:         []string{RouteTableStatusReady}, 
        
           		Refresh:        StatusRouteTable(conn, id), 
        
           		Timeout:        timeout, 
        
           		NotFoundChecks: RouteTableNotFoundChecks, 
        
           	}

This will ensure that the new route table is successfully read back twice in a row before declaring that the resource is ready.

This reverts commit d94464d.

…bleReady' and 'WaitRouteTableDeleted'.

ewbankkit

LGTM 🚀.

% make testacc TESTARGS='-run=TestAccVPCRouteTable_basic\|TestAccVPCRouteTable_ipv6' PKG=ec2 ACCTEST_PARALLELISM=3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/ec2/... -v -count 1 -parallel 3  -run=TestAccVPCRouteTable_basic\|TestAccVPCRouteTable_ipv6 -timeout 180m
=== RUN   TestAccVPCRouteTable_basic
=== PAUSE TestAccVPCRouteTable_basic
=== RUN   TestAccVPCRouteTable_ipv6ToEgressOnlyInternetGateway
=== PAUSE TestAccVPCRouteTable_ipv6ToEgressOnlyInternetGateway
=== CONT  TestAccVPCRouteTable_basic
=== CONT  TestAccVPCRouteTable_ipv6ToEgressOnlyInternetGateway
--- PASS: TestAccVPCRouteTable_basic (21.01s)
--- PASS: TestAccVPCRouteTable_ipv6ToEgressOnlyInternetGateway (40.82s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/ec2	44.885s

ewbankkit · 2022-07-20T12:19:53Z

@patrickdillon Thanks for the contribution 🎉 👏.

patrickdillon · 2022-07-20T14:00:58Z

@ewbankkit Thanks for taking care of this!

github-actions · 2022-07-22T02:59:20Z

This functionality has been released in v4.23.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2022-08-22T02:42:09Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

aws_vpc_route_table: retry read on new resource

d94464d

We are seeing failures when trying to read a newly created resource. We should retry reads in those cases to allow for eventual consistency.

github-actions bot added service/vpc Issues and PRs that pertain to the vpc service. needs-triage Waiting for first response or review from a maintainer. size/XS Managed by automation to categorize the size of a PR. labels Jul 12, 2022

update .changelog

4335c95

github-actions bot reviewed Jul 12, 2022

View reviewed changes

ewbankkit removed the needs-triage Waiting for first response or review from a maintainer. label Jul 15, 2022

ewbankkit requested review from ewbankkit and removed request for ewbankkit July 15, 2022 16:15

ewbankkit added bug Addresses a defect in current functionality. eventual-consistency Pertains to eventual consistency issues. labels Jul 20, 2022

ewbankkit added 3 commits July 20, 2022 08:09

Revert "aws_vpc_route_table: retry read on new resource"

1624fde

This reverts commit d94464d.

r/aws_route_table: Add 'ContinuousTargetOccurence: 2' to 'WaitRouteTa…

eca5035

…bleReady' and 'WaitRouteTableDeleted'.

Tweak CHANGELOG entry.

d337612

github-actions bot removed the service/vpc Issues and PRs that pertain to the vpc service. label Jul 20, 2022

ewbankkit approved these changes Jul 20, 2022

View reviewed changes

ewbankkit merged commit e977d00 into hashicorp:main Jul 20, 2022

github-actions bot added this to the v4.23.0 milestone Jul 20, 2022

github-actions bot locked as resolved and limited conversation to collaborators Aug 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws_vpc_route_table: retry read on new resource #25793

aws_vpc_route_table: retry read on new resource #25793

patrickdillon commented Jul 12, 2022

github-actions bot left a comment

ewbankkit commented Jul 15, 2022

ewbankkit left a comment

ewbankkit commented Jul 20, 2022

patrickdillon commented Jul 20, 2022

github-actions bot commented Jul 22, 2022

github-actions bot commented Aug 22, 2022

aws_vpc_route_table: retry read on new resource #25793

aws_vpc_route_table: retry read on new resource #25793

Conversation

patrickdillon commented Jul 12, 2022

Community Note

github-actions bot left a comment

Choose a reason for hiding this comment

ewbankkit commented Jul 15, 2022

ewbankkit left a comment

Choose a reason for hiding this comment

ewbankkit commented Jul 20, 2022

patrickdillon commented Jul 20, 2022

github-actions bot commented Jul 22, 2022

github-actions bot commented Aug 22, 2022