dynamodb_table - boto3 migration #726

tremble · 2021-09-20T12:52:19Z

SUMMARY

Existing boto3 migration (#65) seems to have been mangled by the migration to collections and has bit-rotted. Start from scratch and don't add new features.

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

dynamodb_table

ADDITIONAL INFORMATION

ansibullbot · 2021-09-20T13:09:34Z

cc @jillr @loia @markuman @s-hertel @wimnat
click here for bot help

tremble · 2021-09-22T09:18:47Z

@marknet15 if you have time it'd be appreciated if you could review/test this change. It looks like you've used the existing module?

marknet15 · 2021-09-22T10:39:51Z

For general provisioned tables it seems fine 👍🏻 I was able to create provisioned tables with and without indexes etc without an issue:

Task result:

changed: [localhost] => (item={'name': 'demoapp-example-indexes', 'hash_key_name': 'SomeId', 'hash_key_type': 'NUMBER', 'read_capacity': 1, 'write_capacity': 1, 'indexes': [{'name': 'TestId-Index', 'type': 'global_all', 'hash_key_name': 'TestId', 'hash_key_type': 'NUMBER'}]}) => {
    "ansible_loop_var": "table",
    "changed": true,
    "hash_key_name": "SomeId",
    "hash_key_type": "NUMBER",
    "indexes": [
        {
            "hash_key_name": "TestId",
            "hash_key_type": "NUMBER",
            "includes": [],
            "name": "TestId-Index",
            "range_key_name": null,
            "range_key_type": null,
            "read_capacity": 1,
            "type": "global_ALL",
            "write_capacity": 1
        }
    ],
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "debug_botocore_endpoint_logs": false,
            "ec2_url": null,
            "hash_key_name": "SomeId",
            "hash_key_type": "NUMBER",
            "indexes": [
                {
                    "hash_key_name": "TestId",
                    "hash_key_type": "NUMBER",
                    "includes": null,
                    "name": "TestId-Index",
                    "range_key_name": null,
                    "range_key_type": null,
                    "read_capacity": null,
                    "type": "ALL",
                    "write_capacity": null
                }
            ],
            "name": "demoapp-example-indexes",
            "profile": null,
            "purge_tags": true,
            "range_key_name": null,
            "range_key_type": null,
            "read_capacity": 1,
            "region": "eu-west-1",
            "security_token": null,
            "state": "present",
            "tags": {
                "Name": "demoapp",
                "smart_pipeline": "true"
            },
            "validate_certs": true,
            "wait": true,
            "wait_for_active_timeout": 60,
            "wait_timeout": 60,
            "write_capacity": 1
        }
    },
    "range_key_name": null,
    "range_key_type": null,
    "read_capacity": 1,
    "region": "eu-west-1",
    "table": {
        "hash_key_name": "SomeId",
        "hash_key_type": "NUMBER",
        "indexes": [
            {
                "hash_key_name": "TestId",
                "hash_key_type": "NUMBER",
                "name": "TestId-Index",
                "type": "global_all"
            }
        ],
        "name": "demoapp-example-indexes",
        "read_capacity": 1,
        "write_capacity": 1
    },
    "table_name": "demoapp-example-indexes",
    "table_status": "ACTIVE",
    "tags": {
        "Name": "demoapp",
        "smart_pipeline": "true"
    },
    "write_capacity": 1
}

However on editing the read / write capacity on an index I do get an error

"msg": "Failed to update table: Parameter validation failed:\nMissing required parameter in GlobalSecondaryIndexUpdates[0].Update.ProvisionedThroughput: \"WriteCapacityUnits\"",

Example:

- name: Create DynamoDB tables
  community.aws.dynamodb_table:
    name: "demoapp-example-indexes"
    read_capacity: 1
    write_capacity: 1
    hash_key_name: "SomeId"
    hash_key_type: "NUMBER"
    indexes:
      - name: TestId-Index
        type: global_all
        hash_key_name: TestId
        hash_key_type: NUMBER
        read_capacity: 5
        write_capacity: 1
    region: "eu-west-1"
    state: present

Task result:

The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_community.aws.dynamodb_table_payload_z5__r9u5/ansible_community.aws.dynamodb_table_payload.zip/ansible_collections/community/aws/plugins/modules/dynamodb_table.py", line 739, in _update_table
  File "/tmp/ansible_community.aws.dynamodb_table_payload_z5__r9u5/ansible_community.aws.dynamodb_table_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/core.py", line 334, in deciding_wrapper
    return retrying_wrapper(*args, **kwargs)
  File "/tmp/ansible_community.aws.dynamodb_table_payload_z5__r9u5/ansible_community.aws.dynamodb_table_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 125, in _retry_wrapper
    base_class=cls.base_class,
  File "/tmp/ansible_community.aws.dynamodb_table_payload_z5__r9u5/ansible_community.aws.dynamodb_table_payload.zip/ansible_collections/amazon/aws/plugins/module_utils/cloud.py", line 68, in _retry_func
    return func()
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 678, in _make_api_call
    api_params, operation_model, context=request_context)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 726, in _convert_to_request_dict
    api_params, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/validate.py", line 337, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Missing required parameter in GlobalSecondaryIndexUpdates[0].Update.ProvisionedThroughput: "WriteCapacityUnits"
fatal: [localhost]: FAILED! => {
    "boto3_version": "1.18.0",
    "botocore_version": "1.21.45",
    "changed": false,
    "invocation": {
        "module_args": {
            "aws_access_key": null,
            "aws_ca_bundle": null,
            "aws_config": null,
            "aws_secret_key": null,
            "debug_botocore_endpoint_logs": false,
            "ec2_url": null,
            "hash_key_name": "SomeId",
            "hash_key_type": "NUMBER",
            "indexes": [
                {
                    "hash_key_name": "TestId",
                    "hash_key_type": "NUMBER",
                    "includes": null,
                    "name": "TestId-Index",
                    "range_key_name": null,
                    "range_key_type": null,
                    "read_capacity": 5,
                    "type": "global_all",
                    "write_capacity": 1
                }
            ],
            "name": "demoapp-example-indexes",
            "profile": null,
            "purge_tags": true,
            "range_key_name": null,
            "range_key_type": null,
            "read_capacity": 1,
            "region": "eu-west-1",
            "security_token": null,
            "state": "present",
            "tags": null,
            "validate_certs": true,
            "wait": true,
            "wait_timeout": 120,
            "write_capacity": 1
        }
    },
    "msg": "Failed to update table: Parameter validation failed:\nMissing required parameter in GlobalSecondaryIndexUpdates[0].Update.ProvisionedThroughput: \"WriteCapacityUnits\""
}

plugins/modules/dynamodb_table.py

alinabuzachis · 2021-09-22T14:31:48Z

recheck

alinabuzachis · 2021-09-22T15:46:03Z

@tremble

TASK [dynamodb_table : Check results - Tag table] ******************************
task path: /home/zuul/.ansible/collections/ansible_collections/community/aws/tests/integration/targets/dynamodb_table/tasks/main.yml:139
fatal: [testhost]: FAILED! => {
    "assertion": "tag_table.tags == tags_default",
    "changed": false,
    "evaluated_to": false,
    "msg": "Assertion failed"
}

tremble · 2021-09-22T17:56:16Z

@alinabuzachis That shows up when the module's blocked from accessing tags.

I've run the code locally against a copy of the perms and it's working. Maybe @jillr could double-check that the permissions from mattclay/aws-terminator#163 are live...

tremble · 2021-09-23T08:51:41Z

@marknet15 Many thanks for catching that bug, I've added a fix and some extra tests.

marknet15 · 2021-10-01T16:19:33Z

@tremble I had a look through because I was curious and was working on the billing_mode and it's definitely permissions related:

TASK [dynamodb_table : Create table] *******************************************
task path: /home/zuul/.ansible/collections/ansible_collections/community/aws/tests/integration/targets/dynamodb_table/tasks/main.yml:33
Using module file /home/zuul/.ansible/collections/ansible_collections/community/aws/plugins/modules/dynamodb_table.py
Pipelining is enabled.
<testhost> ESTABLISH LOCAL CONNECTION FOR USER: zuul
<testhost> EXEC /bin/sh -c 'ANSIBLE_DEBUG_BOTOCORE_LOGS=True /home/zuul/venv/bin/python && sleep 0'

[WARNING]: Permission denied when listing tags
changed: [testhost] => {
    "changed": true,

locally yeah I see all tags etc, I created mattclay/aws-terminator#171 with a slight adjustment, maybe that would help

plugins/modules/dynamodb_table.py

…s updated.

ansible-zuul

LGTM!

nick-zh · 2021-10-12T07:07:56Z

Many thanks for this 🙏

tremble mentioned this pull request Sep 20, 2021

[WIP] dynamodb_table: boto3, AnsibleAwsModule, billing_mode, point_in_time_recovery and more #65

Closed

ansibullbot added WIP Work in progress feature This issue/PR relates to a feature request integration tests/integration module module needs_triage plugins plugin (any type) tests tests labels Sep 20, 2021

tremble force-pushed the boto3/dynamodb_table branch from dac8672 to 1fb3f35 Compare September 22, 2021 09:16

tremble changed the title ~~[WIP] dynamodb_table - boto3 migration~~ dynamodb_table - boto3 migration Sep 22, 2021

tremble removed the WIP Work in progress label Sep 22, 2021

tremble requested review from markuman and alinabuzachis September 22, 2021 09:19

ansibullbot added the community_review label Sep 22, 2021

marknet15 reviewed Sep 22, 2021

View reviewed changes

plugins/modules/dynamodb_table.py Outdated Show resolved Hide resolved

nick-zh mentioned this pull request Sep 23, 2021

Create PAY_PER_REQUEST dynamodb table #609

Closed

marknet15 mentioned this pull request Oct 1, 2021

DynamoDB permissions fix mattclay/aws-terminator#171

Merged

ansibullbot added the has_issue label Oct 1, 2021

marknet15 reviewed Oct 1, 2021

View reviewed changes

plugins/modules/dynamodb_table.py Show resolved Hide resolved

tremble added 3 commits October 9, 2021 09:14

Drop retries, boto3 will use waiters

ee42d55

Drop most bug related test exceptions

7b11b5f

dynamodb - Initial migration to boto3

2c5268f

tremble added 2 commits October 9, 2021 09:14

changelog

9b68256

Ensure we can update indexes when only a one of read/write capacity i…

5603d4e

…s updated.

tremble force-pushed the boto3/dynamodb_table branch from 0c8df9f to 5603d4e Compare October 9, 2021 08:45

marknet15 approved these changes Oct 9, 2021

View reviewed changes

alinabuzachis approved these changes Oct 11, 2021

View reviewed changes

tremble added the gate label Oct 11, 2021

ansible-zuul bot approved these changes Oct 11, 2021

View reviewed changes

ansible-zuul bot merged commit e3a50c4 into ansible-collections:main Oct 11, 2021

marknet15 mentioned this pull request Oct 16, 2021

community.aws.dynamodb_table ignores AWS role #486

Closed

tremble deleted the boto3/dynamodb_table branch November 26, 2021 12:44

tremble mentioned this pull request Jul 9, 2022

AWS_URL isn't passed into dynamodb_table #146

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dynamodb_table - boto3 migration #726

dynamodb_table - boto3 migration #726

tremble commented Sep 20, 2021 •

edited

Loading

ansibullbot commented Sep 20, 2021

tremble commented Sep 22, 2021

marknet15 commented Sep 22, 2021 •

edited

Loading

alinabuzachis commented Sep 22, 2021

alinabuzachis commented Sep 22, 2021

tremble commented Sep 22, 2021

tremble commented Sep 23, 2021

marknet15 commented Oct 1, 2021

ansible-zuul bot left a comment

nick-zh commented Oct 12, 2021

dynamodb_table - boto3 migration #726

dynamodb_table - boto3 migration #726

Conversation

tremble commented Sep 20, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

ansibullbot commented Sep 20, 2021

tremble commented Sep 22, 2021

marknet15 commented Sep 22, 2021 • edited Loading

alinabuzachis commented Sep 22, 2021

alinabuzachis commented Sep 22, 2021

tremble commented Sep 22, 2021

tremble commented Sep 23, 2021

marknet15 commented Oct 1, 2021

ansible-zuul bot left a comment

Choose a reason for hiding this comment

nick-zh commented Oct 12, 2021

tremble commented Sep 20, 2021 •

edited

Loading

marknet15 commented Sep 22, 2021 •

edited

Loading