Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

route53_info - fix max_items when not paginating #1384

Merged

Conversation

mcoot
Copy link
Contributor

@mcoot mcoot commented Aug 2, 2022

SUMMARY

As reported in #1383, the route53_info module presently fails to run with a boto3 parameter validation error if run with particular combinations of parameters, specifically:

  • query: hosted_zone parameter with hosted_zone_method: list_by_name
  • query: reusable_delegation_set without specifying a delegation_set_id

I believe this is a regression introduced in #813

ISSUE TYPE
  • Bugfix Pull Request
COMPONENT NAME

route53_info

ADDITIONAL INFORMATION

Some further information is described in the issue but tl;dr the prior PR converted all cases in the module where params['MaxItems'] was set to instead pass a params['PaginationConfig'], however this should only be done if a boto3 paginator is actually being used, and will fail (as noted above, due to parameter validation) if called with a regular boto3 client method.

Hence this PR switches back to directly setting MaxItems on the methods that do not use a paginator.

@ansibullbot
Copy link

@ansibullbot ansibullbot added bug This issue/PR relates to a bug community_review module module needs_triage new_contributor Help guide this first time contributor plugins plugin (any type) labels Aug 2, 2022
@tremble
Copy link
Contributor

tremble commented Aug 2, 2022

Hi @mcoot,

Thanks for taking the time to submit this PR. We have integration tests in tests/integration/targets/route53 (just an Ansible role) it would be helpful if you could expand the tests to include these use cases, this will help to avoid future regressions.

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.

✔️ ansible-galaxy-importer SUCCESS in 3m 26s
✔️ build-ansible-collection SUCCESS in 4m 48s
✔️ ansible-test-sanity-docker-devel SUCCESS in 10m 14s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 10m 25s
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 10m 07s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 9m 46s
✔️ ansible-test-units-community-aws-python38 SUCCESS in 5m 43s
✔️ ansible-test-units-community-aws-python39 SUCCESS in 5m 55s
✔️ ansible-test-splitter SUCCESS in 2m 26s
✔️ integration-community.aws-1 SUCCESS in 6m 59s
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED

@mcoot
Copy link
Contributor Author

mcoot commented Aug 2, 2022

No worries - I was considering doing so initially, but it didn't obviously fit into the existing flow (and there were no existing integration tests using the relevant operations).

I've added some steps here to try one of the non-paginated operations and assert on the result (which also tests for regression against what #813 aimed to fix) – albeit I don't have a setup for running these against my own account so I'm relying on the CI run here to check the test is correct.

@tremble
Copy link
Contributor

tremble commented Aug 2, 2022

Unfortunately, some of our older (and often more complex) modules have very patchy integration test coverage. While "complete" coverage would be nice, slowly targetting 'known' bugs for additional tests is better than nothing.

@softwarefactory-project-zuul
Copy link
Contributor

Build failed.

✔️ ansible-galaxy-importer SUCCESS in 4m 07s
✔️ build-ansible-collection SUCCESS in 4m 57s
✔️ ansible-test-sanity-docker-devel SUCCESS in 10m 28s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 10m 33s
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 10m 50s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 10m 38s
✔️ ansible-test-units-community-aws-python38 SUCCESS in 6m 49s
✔️ ansible-test-units-community-aws-python39 SUCCESS in 7m 48s
✔️ ansible-test-splitter SUCCESS in 2m 25s
integration-community.aws-1 FAILURE in 6m 42s
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED

@ansibullbot
Copy link

@ansibullbot ansibullbot added integration tests/integration tests tests and removed needs_triage labels Aug 2, 2022
@mcoot
Copy link
Contributor Author

mcoot commented Aug 2, 2022

Well good thing I did add the test, seeing as it failed. Fix up shortly.

The error is a little surprising - Parameter validation failed:\nInvalid type for parameter MaxItems, value: 1, type: <class 'int'>, valid types: <class 'str'>; the code pre-#813 did not convert it to a string for these cases so I'm surprised that fails, albeit the boto3 docs do say it has to be passed as a string.

@tremble tremble added backport-3 PR should be backported to the stable-3 branch backport-4 PR should be backported to the stable-4 branch labels Aug 2, 2022
@github-actions
Copy link

github-actions bot commented Aug 2, 2022

Docs Build 📝

Thank you for contribution!✨

This PR has been merged and your docs changes will be incorporated when they are next published.

@tremble
Copy link
Contributor

tremble commented Aug 2, 2022

Prior to #813 the input parameter didn't have a type defined, this was updated in #813 to require that an integer was passed (max_items=dict(type='int'),). Ansible defaults to assuming that this is supposed to be a string and automatically converts it from the Integer to the String. Passing anything other than an Integer (which would be converted to the string representation) would have triggered an API layer error.

@tremble tremble removed the backport-3 PR should be backported to the stable-3 branch label Aug 2, 2022
@softwarefactory-project-zuul
Copy link
Contributor

Build failed.

✔️ ansible-galaxy-importer SUCCESS in 3m 25s
✔️ build-ansible-collection SUCCESS in 5m 09s
ansible-test-sanity-docker-devel RETRY_LIMIT in 1m 53s (non-voting)
ansible-test-sanity-docker-milestone RETRY_LIMIT in 1m 35s
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 10m 41s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 8m 56s
✔️ ansible-test-units-community-aws-python38 SUCCESS in 7m 02s
✔️ ansible-test-units-community-aws-python39 SUCCESS in 6m 57s
✔️ ansible-test-splitter SUCCESS in 2m 28s
integration-community.aws-1 FAILURE in 6m 17s
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED

@mcoot
Copy link
Contributor Author

mcoot commented Aug 2, 2022

Ah, IAM this time:

An error occurred (AccessDenied) when calling the ListHostedZonesByName operation: User: arn:aws:sts::966509639900:assumed-role/ansible-core-ci-test-prod/prod=remote=zuul-cloud is not authorized to perform: route53:ListHostedZonesByName because no identity-based policy allows the route53:ListHostedZonesByName action

Hmm, how are the IAM perms for the integration tests configured here?

@tremble
Copy link
Contributor

tremble commented Aug 2, 2022

@tremble
Copy link
Contributor

tremble commented Aug 2, 2022

Since I'm in the process of preparing the next minor release (and it'd be good to get this added), I've tested this change locally, I'm going to comment out the integration test and follow up with the relevant folks to get the tests and CI permissions updated.

@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded.

✔️ ansible-galaxy-importer SUCCESS in 4m 03s
✔️ build-ansible-collection SUCCESS in 5m 01s
✔️ ansible-test-sanity-docker-devel SUCCESS in 10m 43s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 9m 07s
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 10m 12s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 10m 26s
✔️ ansible-test-units-community-aws-python38 SUCCESS in 6m 42s
✔️ ansible-test-units-community-aws-python39 SUCCESS in 6m 34s
✔️ ansible-test-splitter SUCCESS in 2m 38s
✔️ integration-community.aws-1 SUCCESS in 7m 20s
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED

@tremble tremble added mergeit Merge the PR (SoftwareFactory) backport-3 PR should be backported to the stable-3 branch labels Aug 2, 2022
@softwarefactory-project-zuul
Copy link
Contributor

Build succeeded (gate pipeline).

✔️ ansible-galaxy-importer SUCCESS in 4m 10s
✔️ build-ansible-collection SUCCESS in 5m 01s
✔️ ansible-test-sanity-docker-devel SUCCESS in 10m 53s (non-voting)
✔️ ansible-test-sanity-docker-milestone SUCCESS in 10m 01s
✔️ ansible-test-sanity-docker-stable-2.12 SUCCESS in 10m 44s
✔️ ansible-test-sanity-docker-stable-2.13 SUCCESS in 9m 43s
✔️ ansible-test-units-community-aws-python38 SUCCESS in 6m 02s
✔️ ansible-test-units-community-aws-python39 SUCCESS in 5m 56s
✔️ ansible-test-splitter SUCCESS in 2m 38s
✔️ integration-community.aws-1 SUCCESS in 8m 10s
⚠️ integration-community.aws-2 SKIPPED
⚠️ integration-community.aws-3 SKIPPED
⚠️ integration-community.aws-4 SKIPPED
⚠️ integration-community.aws-5 SKIPPED
⚠️ integration-community.aws-6 SKIPPED
⚠️ integration-community.aws-7 SKIPPED
⚠️ integration-community.aws-8 SKIPPED
⚠️ integration-community.aws-9 SKIPPED
⚠️ integration-community.aws-10 SKIPPED
⚠️ integration-community.aws-11 SKIPPED
⚠️ integration-community.aws-12 SKIPPED
⚠️ integration-community.aws-13 SKIPPED

@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit 569fff4 into ansible-collections:main Aug 2, 2022
@patchback
Copy link

patchback bot commented Aug 2, 2022

Backport to stable-3: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-3/569fff4c802b226277dfee882d253821a1e1a5d9/pr-1384

Backported as #1387

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback bot pushed a commit that referenced this pull request Aug 2, 2022
route53_info - fix max_items when not paginating

SUMMARY
As reported in #1383, the route53_info module presently fails to run with a boto3 parameter validation error if run with particular combinations of parameters, specifically:

query: hosted_zone parameter with hosted_zone_method: list_by_name
query: reusable_delegation_set without specifying a delegation_set_id

I believe this is a regression introduced in #813
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
route53_info
ADDITIONAL INFORMATION
Some further information is described in the issue but tl;dr the prior PR converted all cases in the module where params['MaxItems'] was set to instead pass a params['PaginationConfig'], however this should only be done if a boto3 paginator is actually being used, and will fail (as noted above, due to parameter validation) if called with a regular boto3 client method.
Hence this PR switches back to directly setting MaxItems on the methods that do not use a paginator.

Reviewed-by: Mark Chappell <None>
(cherry picked from commit 569fff4)
@patchback
Copy link

patchback bot commented Aug 2, 2022

Backport to stable-4: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-4/569fff4c802b226277dfee882d253821a1e1a5d9/pr-1384

Backported as #1388

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback bot pushed a commit that referenced this pull request Aug 2, 2022
route53_info - fix max_items when not paginating

SUMMARY
As reported in #1383, the route53_info module presently fails to run with a boto3 parameter validation error if run with particular combinations of parameters, specifically:

query: hosted_zone parameter with hosted_zone_method: list_by_name
query: reusable_delegation_set without specifying a delegation_set_id

I believe this is a regression introduced in #813
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
route53_info
ADDITIONAL INFORMATION
Some further information is described in the issue but tl;dr the prior PR converted all cases in the module where params['MaxItems'] was set to instead pass a params['PaginationConfig'], however this should only be done if a boto3 paginator is actually being used, and will fail (as noted above, due to parameter validation) if called with a regular boto3 client method.
Hence this PR switches back to directly setting MaxItems on the methods that do not use a paginator.

Reviewed-by: Mark Chappell <None>
(cherry picked from commit 569fff4)
@tremble
Copy link
Contributor

tremble commented Aug 2, 2022

Thanks @mcoot,

Hopefully we can get this out in a release in the next week or so.

softwarefactory-project-zuul bot pushed a commit that referenced this pull request Aug 2, 2022
[PR #1384/569fff4c backport][stable-3] route53_info - fix max_items when not paginating

This is a backport of PR #1384 as merged into main (569fff4).
SUMMARY
As reported in #1383, the route53_info module presently fails to run with a boto3 parameter validation error if run with particular combinations of parameters, specifically:

query: hosted_zone parameter with hosted_zone_method: list_by_name
query: reusable_delegation_set without specifying a delegation_set_id

I believe this is a regression introduced in #813
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
route53_info
ADDITIONAL INFORMATION
Some further information is described in the issue but tl;dr the prior PR converted all cases in the module where params['MaxItems'] was set to instead pass a params['PaginationConfig'], however this should only be done if a boto3 paginator is actually being used, and will fail (as noted above, due to parameter validation) if called with a regular boto3 client method.
Hence this PR switches back to directly setting MaxItems on the methods that do not use a paginator.

Reviewed-by: Mark Chappell <None>
softwarefactory-project-zuul bot pushed a commit that referenced this pull request Aug 2, 2022
[PR #1384/569fff4c backport][stable-4] route53_info - fix max_items when not paginating

This is a backport of PR #1384 as merged into main (569fff4).
SUMMARY
As reported in #1383, the route53_info module presently fails to run with a boto3 parameter validation error if run with particular combinations of parameters, specifically:

query: hosted_zone parameter with hosted_zone_method: list_by_name
query: reusable_delegation_set without specifying a delegation_set_id

I believe this is a regression introduced in #813
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
route53_info
ADDITIONAL INFORMATION
Some further information is described in the issue but tl;dr the prior PR converted all cases in the module where params['MaxItems'] was set to instead pass a params['PaginationConfig'], however this should only be done if a boto3 paginator is actually being used, and will fail (as noted above, due to parameter validation) if called with a regular boto3 client method.
Hence this PR switches back to directly setting MaxItems on the methods that do not use a paginator.

Reviewed-by: Mark Chappell <None>
@mcoot
Copy link
Contributor Author

mcoot commented Aug 2, 2022

Awesome, thanks for your help with this!

softwarefactory-project-zuul bot pushed a commit that referenced this pull request Aug 3, 2022
route53_info - enable max_items tests

SUMMARY
Enables the tests from #1384
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
route53_info
ADDITIONAL INFORMATION
abikouo pushed a commit to abikouo/community.aws that referenced this pull request Oct 24, 2023
…#1384)

route53_info - fix max_items when not paginating

SUMMARY
As reported in ansible-collections#1383, the route53_info module presently fails to run with a boto3 parameter validation error if run with particular combinations of parameters, specifically:

query: hosted_zone parameter with hosted_zone_method: list_by_name
query: reusable_delegation_set without specifying a delegation_set_id

I believe this is a regression introduced in ansible-collections#813
ISSUE TYPE

Bugfix Pull Request

COMPONENT NAME
route53_info
ADDITIONAL INFORMATION
Some further information is described in the issue but tl;dr the prior PR converted all cases in the module where params['MaxItems'] was set to instead pass a params['PaginationConfig'], however this should only be done if a boto3 paginator is actually being used, and will fail (as noted above, due to parameter validation) if called with a regular boto3 client method.
Hence this PR switches back to directly setting MaxItems on the methods that do not use a paginator.

Reviewed-by: Mark Chappell <None>

This commit was initially merged in https://github.com/ansible-collections/community.aws
See: ansible-collections@569fff4
abikouo pushed a commit to abikouo/community.aws that referenced this pull request Oct 24, 2023
route53_info - enable max_items tests

SUMMARY
Enables the tests from ansible-collections#1384
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
route53_info
ADDITIONAL INFORMATION

This commit was initially merged in https://github.com/ansible-collections/community.aws
See: ansible-collections@7427a0d
abikouo pushed a commit to abikouo/community.aws that referenced this pull request Oct 24, 2023
Allow to disable encryption on cloudtrail

SUMMARY

Allow to disable encryption on cloudtrail.
Note: Tests are not run in CI because of missing policy.

ISSUE TYPE


Bugfix Pull Request
Docs Pull Request
Feature Pull Request
New Module Pull Request

COMPONENT NAME

ADDITIONAL INFORMATION

Reviewed-by: Mark Chappell
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-3 PR should be backported to the stable-3 branch backport-4 PR should be backported to the stable-4 branch bug This issue/PR relates to a bug community_review integration tests/integration mergeit Merge the PR (SoftwareFactory) module module new_contributor Help guide this first time contributor plugins plugin (any type) tests tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants