Add NotFound retries when tagging a new RouteTable

[tremble]

SUMMARY

Prior to botocore 1.17.24 it wasn't possible to tag the route tables as a part of the creation call. As such a separate create tags call is performed after the creation. However, Amazon being 'Eventually consistent" we sometimes see NotFound errors just after creating a route table.

Rather than adding hard to test logic, add InvalidRouteTableID.NotFound retries. The tests for RouteTable existing uses its own decorator, we can safely retry on InvalidRouteTableID.NotFound

Also move from custom tag handling code over to the shared code in ansible_collections.amazon.aws.plugins.module_utils.ec2

ISSUE TYPE

• Bugfix Pull Request

COMPONENT NAME

ec2_vpc_route_table

ADDITIONAL INFORMATION

TASK [ec2_vpc_endpoint : Create a minimal route table (with IGW)] **************
task path: /home/zuul/.ansible/collections/ansible_collections/community/aws/tests/integration/targets/ec2_vpc_endpoint/tasks/main.yml:54
<testhost> ESTABLISH LOCAL CONNECTION FOR USER: zuul
<testhost> EXEC /bin/sh -c 'echo ~zuul && sleep 0'
<testhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/zuul/.ansible/tmp `"&& mkdir "` echo /home/zuul/.ansible/tmp/ansible-tmp-1624964310.5067005-7213-234903855964036 `" && echo ansible-tmp-1624964310.5067005-7213-234903855964036="` echo /home/zuul/.ansible/tmp/ansible-tmp-1624964310.5067005-7213-234903855964036 `" ) && sleep 0'
Using module file /home/zuul/.ansible/collections/ansible_collections/community/aws/plugins/modules/ec2_vpc_route_table.py
<testhost> PUT /home/zuul/.ansible/tmp/ansible-local-7051vh18nelt/tmpdx500zwb TO /home/zuul/.ansible/tmp/ansible-tmp-1624964310.5067005-7213-234903855964036/AnsiballZ_ec2_vpc_route_table.py
<testhost> EXEC /bin/sh -c 'chmod u+x /home/zuul/.ansible/tmp/ansible-tmp-1624964310.5067005-7213-234903855964036/ /home/zuul/.ansible/tmp/ansible-tmp-1624964310.5067005-7213-234903855964036/AnsiballZ_ec2_vpc_route_table.py && sleep 0'
<testhost> EXEC /bin/sh -c 'ANSIBLE_DEBUG_BOTOCORE_LOGS=True /home/zuul/venv/bin/python3.6 /home/zuul/.ansible/tmp/ansible-tmp-1624964310.5067005-7213-234903855964036/AnsiballZ_ec2_vpc_route_table.py && sleep 0'
<testhost> EXEC /bin/sh -c 'rm -f -r /home/zuul/.ansible/tmp/ansible-tmp-1624964310.5067005-7213-234903855964036/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_ec2_vpc_route_table_payload_y_h0b0in/ansible_ec2_vpc_route_table_payload.zip/ansible_collections/community/aws/plugins/modules/ec2_vpc_route_table.py", line 380, in ensure_tags
  File "/tmp/ansible_ec2_vpc_route_table_payload_y_h0b0in/ansible_ec2_vpc_route_table_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/core.py", line 288, in deciding_wrapper
    return retrying_wrapper(*args, **kwargs)
  File "/tmp/ansible_ec2_vpc_route_table_payload_y_h0b0in/ansible_ec2_vpc_route_table_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 154, in retry_func
    raise e
  File "/tmp/ansible_ec2_vpc_route_table_payload_y_h0b0in/ansible_ec2_vpc_route_table_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 144, in retry_func
    return f(*args, **kwargs)
  File "/home/zuul/venv/lib/python3.6/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/zuul/venv/lib/python3.6/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidRouteTableID.NotFound) when calling the CreateTags operation: The routeTable ID 'rtb-08ce2a0fac35f2cc3' does not exist
fatal: [testhost]: FAILED! => {
    "boto3_version": "1.17.102",
    "botocore_version": "1.20.102",
    "changed": false,
    "error": {
        "code": "InvalidRouteTableID.NotFound",
        "message": "The routeTable ID 'rtb-08ce2a0fac35f2cc3' does not exist"
    },
    "invocation": {
        "module_args": {
            "aws_access_key": "ASIA6CCDWXDOL3X5ZIJP",
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "debug_botocore_endpoint_logs": true,
            "ec2_url": null,
            "lookup": "tag",
            "profile": null,
            "propagating_vgw_ids": null,
            "purge_routes": true,
            "purge_subnets": true,
            "purge_tags": false,
            "region": "us-east-1",
            "route_table_id": null,
            "routes": [
                {
                    "destination_cidr_block": "0.0.0.0/0",
                    "gateway_id": "igw-0c6e113ca6c5d9a2c"
                }
            ],
            "security_token": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "state": "present",
            "subnets": [],
            "tags": {
                "AnsibleRun": "ansible-test-56393648-centos-8-1vcpu-vexxhost-ca-ymq-1-00016024",
                "AnsibleTest": "ec2_vpc_endpoint",
                "Name": "ansible-test-56393648-centos-8-1vcpu-vexxhost-ca-ymq-1-00016024-igw"
            },
            "validate_certs": true,
            "vpc_id": "vpc-055d3d6ce9bec0dc3"
        }
    },
    "msg": "Couldn't create tags: An error occurred (InvalidRouteTableID.NotFound) when calling the CreateTags operation: The routeTable ID 'rtb-08ce2a0fac35f2cc3' does not exist",
    "resource_actions": [
        "ec2:CreateRouteTable",
        "ec2:CreateRoute",
        "ec2:CreateTags",
        "ec2:DescribeRouteTables",
        "ec2:DescribeTags"
    ],
    "response_metadata": {
        "http_headers": {
            "cache-control": "no-cache, no-store",
            "connection": "close",
            "content-type": "text/xml;charset=UTF-8",
            "date": "Tue, 29 Jun 2021 10:58:31 GMT",
            "server": "AmazonEC2",
            "strict-transport-security": "max-age=31536000; includeSubDomains",
            "transfer-encoding": "chunked",
            "vary": "accept-encoding",
            "x-amzn-requestid": "fbefb890-4384-4001-a374-cb13db9e0d6c"
        },
        "http_status_code": 400,
        "request_id": "fbefb890-4384-4001-a374-cb13db9e0d6c",
        "retry_attempts": 0
    }
}

[ansibullbot]

cc @jillr @s-hertel @wimnat
click here for bot help

[tremble]

recheck

[tremble]

Looks like ec2_vpc_endpoint also has the same issue, (hence the need for the recheck), I'll get something submitted once the migration to amazon.aws is completed