[libbeat][aws] Fix AWS config initialization issue when using a role

[zmoog]

What does this PR do?

Set up the AssumeRoleProvider after the AWS region value from Filebeat settings is applied to the AWS SDK configuration.

Why is it important?

When Filebeat uses a role ARN, it sets up AssumeRoleProvider before evaluating the region value from its settings.

If the AWS SDK configuration loaded from ~/.aws/config does not contain a region, the error described in #30999 happens.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Requirements:

• A SQS queue (in this example, https://sqs.eu-west-1.amazonaws.com/000123456789/elastic-cloudtrail-logs),
• an IAM role ARN (in this example, arn:aws:iam::000123456789:role/elastic-agent-role)

Configure the IAM Role as EC2 instance role with the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Also add some permission to the role to access SQS and S3 ("Amazon SQS Full Access" and "SQS Read only Access" are probably fine for a quick test).

Here's an example command line arguments for Filebeat:

-e
-v
-d
*
--strict.perms=false
--path.home
/Users/zmoog/code/projects/zmoog/beats/x-pack/filebeat
-E
cloud.id=<CLOUD_ID>
-E
cloud.api_key=<CLOUD_API_KEY>
-E
gc_percent=100
-E
setup.ilm.enabled=false
-E
setup.template.enabled=false
--modules
aws
-M
aws.s3access.enabled=true
-M
aws.s3access.input.default_region=eu-west-1
-M
aws.s3access.input.queue_url=https://sqs.eu-west-1.amazonaws.com/000123456789/elastic-cloudtrail-logs
-M
aws.s3access.input.role_arn=arn:aws:iam::000123456789:role/elastic-agent-role

Related issues

• Closes [Filebeat] sqs ReceiveMessage failed: unknown endpoint, could not resolve endpoint #30999

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label. Could you fix it @zmoog? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-04-05T21:40:25.357+0000

• Duration: 109 min 29 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8270
Skipped	532
Total	8802

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[kaiyan-sheng]

Looks good to me! Could you add a changelog to it please? Thanks!

[zmoog]

Looks good to me! Could you add a changelog to it please? Thanks!

Oh! 🤦

Thanks for the heads up! 🙇

[zmoog]

@kaiyan-sheng could you please double check the CHANGELOG entry is in the right place for this issue?

[kaiyan-sheng]

The changelog entry looks good to me! Thanks!

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b zmoog/fix-aws-region-on-assume-role upstream/zmoog/fix-aws-region-on-assume-role
git merge upstream/main
git push upstream zmoog/fix-aws-region-on-assume-role