Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rds_instance - fix check_mode and idempotence bugs and support adding/removing iam roles #1002

Conversation

jatorcasso
Copy link
Contributor

@jatorcasso jatorcasso commented Mar 17, 2022

SUMMARY

Depends-On ansible-collections/amazon.aws#714

ISSUE TYPE
  • Feature Pull Request
COMPONENT NAME

rds_instance

ADDITIONAL INFORMATION

Wasn't sure the best way to go about deleting IAM roles - ended up using a purge_iam_roles param that defaults to False, which seems consistent with other modules I've looked at.

@ansibullbot
Copy link

@markuman
Copy link
Member

Not all RDS engines supports IAM roles. E.g. mariadb.
Do we plan to cover this?

@jatorcasso
Copy link
Contributor Author

Not all RDS engines supports IAM roles. E.g. mariadb. Do we plan to cover this?

That's a good point. I suppose that can be added as a pre-check in rds_instance or as a caught error in amazon.aws.module_utils.rds.handle_errors. I would guess add it to the utils handle_errors since that already exists, but im never really sure which collection I should be modifying..

@alinabuzachis
Copy link
Contributor

alinabuzachis commented Mar 25, 2022

Not all RDS engines supports IAM roles. E.g. mariadb. Do we plan to cover this?

That's a good point. I suppose that can be added as a pre-check in rds_instance or as a caught error in amazon.aws.module_utils.rds.handle_errors. I would guess add it to the utils handle_errors since that already exists, but im never really sure which collection I should be modifying..

I guess a pre-check inside rds_instance will be enough. Please also document the engines supporting iam roles.

@jatorcasso
Copy link
Contributor Author

Not all RDS engines supports IAM roles. E.g. mariadb. Do we plan to cover this?

That's a good point. I suppose that can be added as a pre-check in rds_instance or as a caught error in amazon.aws.module_utils.rds.handle_errors. I would guess add it to the utils handle_errors since that already exists, but im never really sure which collection I should be modifying..

I guess a pre-check inside rds_instance will be enough. Please also document the engines supporting aim roles.

I actually think it makes more sense in this case to add an extra conditional in the handle_errors method, similar to what is done for when the user passes in an invalid engine. something like:

    elif method_name == 'add_role_to_db_instance' and error_code == 'InvalidParameterValue':
        if 'is not valid for the engine' in to_text(exception) and parameters.get('Engine') not in accepted_engines:
            module.fail_json_aws(exception, msg='It appears you are trying to modify IAM roles for an engine that does not support IAM roles.')
        else:
            module.fail_json_aws(exception, msg='Unable to {0}'.format(get_rds_method_attribute(method_name, module).operation_description))

Only issue is the accepted engines here are not well documented

@alinabuzachis
Copy link
Contributor

alinabuzachis commented Mar 25, 2022

Not all RDS engines supports IAM roles. E.g. mariadb. Do we plan to cover this?

That's a good point. I suppose that can be added as a pre-check in rds_instance or as a caught error in amazon.aws.module_utils.rds.handle_errors. I would guess add it to the utils handle_errors since that already exists, but im never really sure which collection I should be modifying..

I guess a pre-check inside rds_instance will be enough. Please also document the engines supporting aim roles.

I actually think it makes more sense in this case to add an extra conditional in the handle_errors method, similar to what is done for when the user passes in an invalid engine. something like:

    elif method_name == 'add_role_to_db_instance' and error_code == 'InvalidParameterValue':
        if 'is not valid for the engine' in to_text(exception) and parameters.get('Engine') not in accepted_engines:
            module.fail_json_aws(exception, msg='It appears you are trying to modify IAM roles for an engine that does not support IAM roles.')
        else:
            module.fail_json_aws(exception, msg='Unable to {0}'.format(get_rds_method_attribute(method_name, module).operation_description))

Only issue is the accepted engines here are not well documented

Where did you find that iam roles are not supported by all the engines https://boto3.amazonaws.com/v1/documentation/api/1.9.185/reference/services/rds.html#RDS.Client.add_role_to_db_instance? This only happens for rds cluster (https://boto3.amazonaws.com/v1/documentation/api/1.9.185/reference/services/rds.html#RDS.Client.add_role_to_db_cluster).

@jatorcasso
Copy link
Contributor Author

its convoluted, but some engines dont support any features. for example from https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_DBEngineVersion.html:

SupportedFeatureNames.member.N
A list of features supported by the DB engine.

The supported features vary by DB engine and DB engine version.

To determine the supported features for a specific DB engine and DB engine version using the AWS CLI, use the following command:

aws rds describe-db-engine-versions --engine <engine_name> --engine-version <engine_version>

For example, to determine the supported features for RDS for PostgreSQL version 13.3 using the AWS CLI, use the following command:

aws rds describe-db-engine-versions --engine postgres --engine-version 13.3

The supported features are listed under SupportedFeatureNames in the output.

Type: Array of strings

Required: No

I had to learn this in order to properly test.. many of the db engines simply return an empty list of supported features.

@alinabuzachis
Copy link
Contributor

its convoluted, but some engines dont support any features. for example from https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_DBEngineVersion.html:

SupportedFeatureNames.member.N
A list of features supported by the DB engine.
The supported features vary by DB engine and DB engine version.
To determine the supported features for a specific DB engine and DB engine version using the AWS CLI, use the following command:
aws rds describe-db-engine-versions --engine <engine_name> --engine-version <engine_version>
For example, to determine the supported features for RDS for PostgreSQL version 13.3 using the AWS CLI, use the following command:
aws rds describe-db-engine-versions --engine postgres --engine-version 13.3
The supported features are listed under SupportedFeatureNames in the output.
Type: Array of strings
Required: No

I had to learn this in order to properly test.. many of the db engines simply return an empty list of supported features.

Ok. yeah! So I guess you need to handle that as you suggested directly inside module_utils in amazon.aws. Cover it by a unit test.

@markuman
Copy link
Member

Where did you find that iam roles are not supported by all the engines ...

... just experiences :)

@markuman
Copy link
Member

markuman commented Mar 25, 2022

Ok. yeah! So I guess you need to handle that as you suggested directly inside module_utils in amazon.aws. Cover it by a unit test.

I'm not sure .... if rds_instance is moved soon into amazon.aws, ok. otherwise it feels easier to maintain those technical deprecation in rds_instance itself.
Because when AWS maybe support this one e.g. on 1st april, you need to wait for a fix in amazon.aws and just backport might be also not that easy.

@jatorcasso
Copy link
Contributor Author

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/describe-db-engine-versions.html

looks like the valid db engines need to be updated as well. I can make that a separate issue/corresponding PR?

@jatorcasso
Copy link
Contributor Author

Ok. yeah! So I guess you need to handle that as you suggested directly inside module_utils in amazon.aws. Cover it by a unit test.

I'm not sure .... if rds_instance is moved soon into amazon.aws, ok. otherwise it feels easier to maintain those technical deprecation in rds_instance itself. Because when AWS maybe support this one e.g. on 1st april, you need to wait for a fix in amazon.aws and just backport might be also not that easy.

I'll add in rds_instance. It's not much anyway

@markuman
Copy link
Member

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/describe-db-engine-versions.html

looks like the valid db engines need to be updated as well. I can make that a separate issue/corresponding PR?

If we want to land this in 3.2.0, it might be easier to include the updated engine list also in this PR. But it's upt to you.

@jatorcasso jatorcasso marked this pull request as draft March 29, 2022 15:48
@jatorcasso jatorcasso marked this pull request as ready for review April 6, 2022 13:44
@jatorcasso
Copy link
Contributor Author

jatorcasso commented Apr 11, 2022

@markuman @jillr I suppose I should change version_added to 3.3.0 and add backport-3 label ?

@jatorcasso jatorcasso added the backport-3 PR should be backported to the stable-3 branch label Apr 11, 2022
@markuman markuman added the mergeit Merge the PR (SoftwareFactory) label Apr 12, 2022
@softwarefactory-project-zuul softwarefactory-project-zuul bot merged commit c403552 into ansible-collections:main Apr 12, 2022
@patchback
Copy link

patchback bot commented Apr 12, 2022

Backport to stable-3: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-3/c403552f8d8d2da6be0a1797d6ac85ab1cd7b22c/pr-1002

Backported as #1055

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

patchback bot pushed a commit that referenced this pull request Apr 12, 2022
…/removing iam roles (#1002)

rds_instance - fix check_mode and idempotence bugs and support adding/removing iam roles

SUMMARY

Support the addition and deletion of iam roles to db instances
Fixes #464
Fixes #1013
Integration tests to test both this and the amazon.aws module_util rds changes

Depends-On ansible-collections/amazon.aws#714
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
rds_instance
ADDITIONAL INFORMATION
Wasn't sure the best way to go about deleting IAM roles - ended up using a purge_iam_roles param that defaults to False, which seems consistent with other modules I've looked at.

Reviewed-by: Mark Woolley <mw@marknet15.com>
Reviewed-by: Joseph Torcasso <None>
Reviewed-by: Gonéri Le Bouder <goneri@lebouder.net>
Reviewed-by: Alina Buzachis <None>
Reviewed-by: Markus Bergholz <git@osuv.de>
(cherry picked from commit c403552)
jatorcasso added a commit to jatorcasso/amazon.aws that referenced this pull request Apr 12, 2022
…ing/removing IAM roles (ansible-collections#714)

rds module_util - fix check_mode and idempotence bugs and support adding/removing IAM roles

SUMMARY

Add waiter for promoting read replica to fix idempotence bug in community.aws integration testing
Support modifying IAM roles to a db instance for ansible-collections/community.aws#1002
Add necessary waiters for both adding and removing an IAM role
compare and ensure iam_roles methods for idempotency
unit & integration tests for coverage (integration tests in community.aws PR)

ISSUE TYPE

Feature Pull Request

COMPONENT NAME

module_util/rds
community.aws.rds_instance

ADDITIONAL INFORMATION
Refs:

(adding role) https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/add-role-to-db-instance.html
(removing role) https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/remove-role-from-db-instance.html
(waiters) https://awscli.amazonaws.com/v2/documentation/api/latest/reference/rds/describe-db-instances.html

Reviewed-by: Alina Buzachis <None>
Reviewed-by: Joseph Torcasso <None>
Reviewed-by: Mark Woolley <mw@marknet15.com>
Reviewed-by: Mandar Kulkarni <mandar242@gmail.com>
Reviewed-by: Jill R <None>
Reviewed-by: Mike Graves <mgraves@redhat.com>
(cherry picked from commit 00c752e)
softwarefactory-project-zuul bot pushed a commit that referenced this pull request Apr 16, 2022
…/removing iam roles (#1002) (#1055)

[PR #1002/c403552f backport][stable-3] rds_instance - fix check_mode and idempotence bugs and support adding/removing iam roles

This is a backport of PR #1002 as merged into main (c403552).
SUMMARY

Support the addition and deletion of iam roles to db instances
Fixes #464
Fixes #1013
Integration tests to test both this and the amazon.aws module_util rds changes

Depends-On ansible-collections/amazon.aws#714
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
rds_instance
ADDITIONAL INFORMATION
Wasn't sure the best way to go about deleting IAM roles - ended up using a purge_iam_roles param that defaults to False, which seems consistent with other modules I've looked at.

Reviewed-by: Mark Chappell <None>
Reviewed-by: Markus Bergholz <git@osuv.de>
abikouo pushed a commit to abikouo/community.aws that referenced this pull request Oct 24, 2023
…/removing iam roles (ansible-collections#1002)

rds_instance - fix check_mode and idempotence bugs and support adding/removing iam roles

SUMMARY

Support the addition and deletion of iam roles to db instances
Fixes ansible-collections#464
Fixes ansible-collections#1013
Integration tests to test both this and the amazon.aws module_util rds changes

Depends-On ansible-collections/amazon.aws#714
ISSUE TYPE

Feature Pull Request

COMPONENT NAME
rds_instance
ADDITIONAL INFORMATION
Wasn't sure the best way to go about deleting IAM roles - ended up using a purge_iam_roles param that defaults to False, which seems consistent with other modules I've looked at.

Reviewed-by: Mark Woolley <mw@marknet15.com>
Reviewed-by: Joseph Torcasso <None>
Reviewed-by: Gonéri Le Bouder <goneri@lebouder.net>
Reviewed-by: Alina Buzachis <None>
Reviewed-by: Markus Bergholz <git@osuv.de>

This commit was initially merged in https://github.com/ansible-collections/community.aws
See: ansible-collections@c403552
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport-3 PR should be backported to the stable-3 branch community_review feature This issue/PR relates to a feature request integration tests/integration mergeit Merge the PR (SoftwareFactory) module module plugins plugin (any type) tests tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

rds_instance - Broken check_mode and idempotence rds_instance module should support adding IAM roles
6 participants