Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide more ways to set AWS credentials in Functionbeat #23344

Merged
merged 8 commits into from
Jan 7, 2021

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Jan 4, 2021

What does this PR do?

This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced:

  1. access_key_id, secret_access_key and/or session_token for tokens
functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}'
functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}'
  1. role_arn for assuming IAM roles
functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb
  1. credential_profile_name and/or shared_credential_file for credential files
functionbeat.provider.aws.credential_profile_name: fnb-aws
functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials

Why is it important?

Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat.

Checklist

  • My code follows the style guidelines of this project
    - [ ] I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
    - [ ] I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Based on #17658
Closes #12464

@kvch kvch added Functionbeat Team:Elastic-Agent Label for the Agent team labels Jan 4, 2021
@kvch kvch requested a review from kaiyan-sheng January 4, 2021 15:28
@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 4, 2021
@kvch kvch force-pushed the feature-functionbeat-add-aws-credentials branch from 5719971 to 80b3a6e Compare January 4, 2021 15:31
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jan 4, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #23344 updated

  • Start Time: 2021-01-06T22:02:47.452+0000

  • Duration: 51 min 8 sec

Test stats 🧪

Test Results
Failed 0
Passed 5602
Skipped 396
Total 5998

Steps errors 2

Expand to view the steps failures

Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 15 sec . View more details on here
  • Description: terraform apply -auto-approve
Terraform Apply on x-pack/metricbeat/module/aws
  • Took 0 min 15 sec . View more details on here
  • Description: terraform apply -auto-approve

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 5602
Skipped 396
Total 5998

@kvch kvch force-pushed the feature-functionbeat-add-aws-credentials branch from 80b3a6e to 85fe0bd Compare January 4, 2021 15:56
@kvch
Copy link
Contributor Author

kvch commented Jan 4, 2021

Some tests need adjustment.

@kvch kvch force-pushed the feature-functionbeat-add-aws-credentials branch 2 times, most recently from 05c6938 to 6019970 Compare January 5, 2021 14:22
@kvch kvch force-pushed the feature-functionbeat-add-aws-credentials branch from 6019970 to 7f1ed12 Compare January 5, 2021 14:46
@kvch kvch force-pushed the feature-functionbeat-add-aws-credentials branch from f7cdcd5 to b6f0335 Compare January 6, 2021 12:07
@kvch kvch force-pushed the feature-functionbeat-add-aws-credentials branch from b6f0335 to 43e0ef7 Compare January 6, 2021 12:26
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, not sure why doc is not happy though:

12:32:16 INFO:build_docs:asciidoctor: ERROR: config-options-aws.asciidoc: line 219: include file not found: /tmp/docsbuild/eSoev3Xy72/beats/x-pack/libbeat/docs/aws-credentials-config.asciidoc

@bmorelli25
Copy link
Member

Wow, this was a fun doc build error to try and hunt down. I think you're seeing this issue because no file named aws-credentials-examples.asciidoc exists at /functionbeat/docs.

You're including aws-credentials-config.asciidoc: /beats/x-pack/libbeat/docs/aws-credentials-config.asciidoc which has this include on line 46: include::../../../{beatname_lc}/docs/aws-credentials-examples.asciidoc[]

@bmorelli25
Copy link
Member

bmorelli25 commented Jan 6, 2021

Pushed a partial fix in dd080ba. From here, we'll merge elastic/docs#2035, run a doc build, and finally rerun the preview build on this PR.

@bmorelli25
Copy link
Member

@elasticmachine, run elasticsearch-ci/docs

@bmorelli25
Copy link
Member

New error:

14:46:32 INFO:build_docs:asciidoctor: ERROR: ../../libbeat/docs/aws-credentials-config.asciidoc: line 59: include file not found: /tmp/docsbuild/rCldsYRINg/beats/filebeat/docs/aws-credentials-examples.asciidoc

@bmorelli25
Copy link
Member

@elasticmachine, run elasticsearch-ci/docs

@kvch kvch merged commit 5e6558b into elastic:master Jan 7, 2021
@kvch kvch added the v7.12.0 label Jan 7, 2021
kvch added a commit to kvch/beats that referenced this pull request Jan 7, 2021
This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced:

1. `access_key_id`, `secret_access_key` and/or `session_token` for tokens

```yaml
functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}'
functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}'
```
2. `role_arn` for assuming IAM roles
```yaml
functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb
```

3. `credential_profile_name` and/or `shared_credential_file` for credential files
```yaml
functionbeat.provider.aws.credential_profile_name: fnb-aws
functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials
```

Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat.

Based on elastic#17658
Closes elastic#12464

Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>
(cherry picked from commit 5e6558b)
kvch added a commit that referenced this pull request Jan 7, 2021
…3386)

This PR makes credential settings when deploying Lambdas to AWS more flexible. New options are introduced:

1. `access_key_id`, `secret_access_key` and/or `session_token` for tokens

```yaml
functionbeat.provider.aws.access_key_id: '${AWS_ACCESS_KEY_ID:""}'
functionbeat.provider.aws.secret_access_key: '${AWS_SECRET_ACCESS_KEY:""}'
functionbeat.provider.aws.session_token: '${AWS_SESSION_TOKEN:""}'
```
2. `role_arn` for assuming IAM roles
```yaml
functionbeat.provider.aws.role_arn: arn:aws:iam::123456789012:role/test-fnb
```

3. `credential_profile_name` and/or `shared_credential_file` for credential files
```yaml
functionbeat.provider.aws.credential_profile_name: fnb-aws
functionbeat.provider.aws.shared_credential_file: /etc/functionbeat/aws_credentials
```

Credential configuration becomes more flexible and follows the same pattern as in Filebeat and Metricbeat.

Based on #17658
Closes #12464

Co-authored-by: Brandon Morelli <brandon.morelli@elastic.co>
(cherry picked from commit 5e6558b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Please support AWS IAM Instance profiles!
4 participants