Added Secret Manager support

[AlexanderWert]

Closes #105
Closes #206

Description

Added two new configuration options:

• ELASTIC_APM_SECRETS_MANAGER_SECRET_TOKEN_ID
• ELASTIC_APM_SECRETS_MANAGER_API_KEY_ID

If a secret manager id is provided through the above config options, the extension retrieves the corresponding secret from the AWS Secret manager. The secret must be in the same AWS region.
If both, a plain text secret is provided (i.e. ELASTIC_APM_SECRET_TOKEN) and the ELASTIC_APM_SECRETS_MANAGER_SECRET_TOKEN_ID, then the value from the Secret Manager takes precedence.

Retrieving the secret from the Secret Manager happens once at the cold start of the extension.

Steps for end to end testing:

Step 1: Create a secret in the AWS Secret Manager.

• The secret must be in the same AWS region as the target lambda function
• Create the secret either for the APM secret token OR the APM API key
• The secret type must be Plain Text
• Optionally, you can select a custom KMS key for encryption (Note: in this case you will need additional key permissions on your Lambda functions)
  • Otherwise: use the key provided by AWS: aws/secretsmanager (No additional key permissions needed for this)

Step 2: Create and setup the Lambda function

• Follow these docs for general setup of the Lambda function with Elastic APM
• However, create and use a custom extension layer based on this PR
• In the environment variables configuration instead of providing the ELASTIC_APM_SECRET_TOKEN or ELASTIC_APM_API_KEY specify the ELASTIC_APM_SECRETS_MANAGER_SECRET_TOKEN_ID or ELASTIC_APM_SECRETS_MANAGER_API_KEY_ID variables. Use the secret-name as the value.

Step 3: Add permissions

• Under Configuration > Permissions > Execution Role click on the execution role to edit the permissions
• Add a new inline policy with the following JSON-based permission definition:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "HERE_GOES_THE_ARN_OF_THE_SECRET"
            ]
        },
        // The following is only needed if a custom key is used
        {
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "HERE_GOES_THE_ARN_OF_THE_CUSTOM_KEY"
            ]
        }
    ]
}

Step 4: Invoke the lambda function and validate data in Elastic APM UI

[apmmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-06-07T16:36:32.073+0000

• Duration: 8 min 31 sec

Test stats 🧪

Test	Results
Failed	0
Passed	200
Skipped	4
Total	204

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[stuartnelson3]

is there any sort of linting for import order? other projects separate out stdlib / elastic / 3rd party

[stuartnelson3]

might want to check region != "" and return a descriptive error

[stuartnelson3]

len is a builtin and, while the compiler might let you use it, it'd probably be better to choose a different variable name

[stuartnelson3]

according to the docs, [Encoding.Decode] writes at most DecodedLen(len(src)) bytes to dst, so the [:len] shouldn't be necessary -- it should only differ if there's an error and there's a short write, in which case you'll return the error anyway.

[stuartnelson3]

the else can be dropped, and just executed regularly (the Fatalf will abort the program if there's an error)

[stuartnelson3]

same as above

[simitt]

@elastic/apm-server could you please help pushing this over the finish line; I suggest for this PR rather than only reviewing, commit small code improvements directly to free up @AlexanderWert.
Thanks Alex for this contribution.

[jlvoiseux]

LGTM, aside from the small test correction.
I was also able to get the PR to work end-to-end by following the repro steps.

[jlvoiseux]

It seems that the Unit test, when run sequentially together with the whole test suite, fails due to previously set values of the SECRET_TOKEN and the API_KEY.

Suggested change

	func TestGetSecretCalled(t *testing.T) {
	func TestGetSecretCalled(t *testing.T) {
	os.Clearenv()

[stuartnelson3]

I updated the test actually to confirm that there's a preference between the managed and unmanaged secrets, since that wasn't being tested