A Python-based CLI tool to simplify the process of fetching and injecting environment variables from AWS Parameter Store.
- Description
- Installation
- Usage
- Setup
- Permissions
- Concept
- Access-parameter-store-entries
- Authentication
- Todos
- Acknowledgments
- License
This CLI tool (aenv
) allows you to fetch environment variables from AWS Parameter Store, injecting them into your local environment for use in your applications. The tool also supports authentication with MFA, including Yubikeys.
-
Install python3 and pip
-
Boto3 (Will be installed with the aenv package)
- Windows: setup Boto3 credentials
pip install aenv
aenv --help
# or
aenv -h
All current options:
aenv [-s <service/application>] [-i] [-n] [-e <env>] [-t <2fa key>] [-T] [-Y] [-u <aws username>] [-a <account number>] [-p <aws profile>] [-r <region>] <command>
Option | Explanation | Sample | Comment |
---|---|---|---|
-h | Shows help | aenv -h |
|
-i | Starts aenv in interactive mode | aenv -i |
Gives you a command line that you can interact with |
-s <service/application> | For which service should the environment variables be loaded? | aenv -s CustomerService |
|
-S | Sets a default service for aenv and writes it to a config file | aenv -S CustomerService |
from now on "CustomerService" is the default service which means "-s CustomerService" is redundant |
-n | Do not query the parameter store at all | aenv -n |
Can be used to auth the current session with MFA |
-e <env> | For which environment should the environment variables be loaded? For example Dev, Test or Prod (permission required) | aenv -e Prod |
|
-t <2fa key> | Takes the 2FA key from your aws account | aenv -t 987123 |
|
-T | Lets you type in the 2FA key from your aws account during runtime | aenv -T |
When you run your command aenv will ask for the token |
-Y | Uses Yubikey for MFA auth | aenv -Y |
During runtime aenv will use ykman to fetch the MFA-Key from your yubikey |
-r <region> | Overwrites temporary the awscli default region | aenv -r eu-central-1 |
aenv will use the given region for example Frankfurt |
-v | Verbose mode (more output) | aenv -v |
|
-u <aws username> | Sets a specific username combined with -a gives you a faster runtime (otherwise this data needs to be retrieved via aws) | aenv -u user@example.de |
|
-a <account number> | Sets a specific account number combined with -u gives you a faster runtime (otherwise this data needs to be retrieved via aws) | aenv -a 999999999999 |
|
-p <aws profile> | If multiple aws profiles are available you can choose the profile otherwise aenv will use the default profile | aenv -p testUser1 |
|
-c <aws profile> | Container mode(enable this to make aenv work in ecs and codebuild) | aenv -c |
permissions |
<command> | Is the command to execute with environment variables injected. | aenv code |
Will run VS Code with access to given environment variables |
Lets start with setting up a simple example service. Let's call it UserService.
The UserService needs a database hostname, a database username and a database password.
Let's assume, we have two environments, "Dev" and "Test" and we want to inject the correct UserService, database hostname,... for the related environment.
Step 1 is to create the AWS parameter store entries. Let's create two entries:
/Dev/UserService/DB/hostname
# and
/Test/UserService/DB/hostname
Both are SecureString, that hold the values:
db.dev.example.com for /Dev/UserService/DB/hostname
and
db.test.example.com for /Test/UserService/DB/hostname
Step 2 is to fetch those values and have them available as environment variables.
For the UserService variables on Dev run:
aenv -e Dev -s UserService
This command fetches all entries from the parameter store with the path /Dev/UserService/* and makes them available as environment variables.
If you just want to echo out the DB hostname we created (/Dev/UserService/DB/hostname) you can run:
aenv -e Dev -s UserService echo '$SECRET_USERSERVICE_DB_HOSTNAME'
You can also run your service with aenv to have the correct DB hostname available in the service as an environment variable. The call for a Python or JVM service would be:
aenv -e Dev -s UserService java -jar service.jar
# or
aenv -e Dev -s UserService python service2.py
Both services now have access to the environment variable "SECRET_USERSERVICE_DB_HOSTNAME" containing the value that we defined for "/Dev/UserService/DB/hostname".
Here is the minimal suggested set of IAM permissions to use aenv for all services that can be found for our Dev environment:
(Do not forget to adapt to account ID(123456789098) to your own )
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:GetParametersByPath",
"ssm:GetParameters",
"ssm:ListTagsForResource",
"ssm:GetParameter"
],
"Resource": [
"arn:aws:ssm:*:123456789098:parameter/Dev/*"
]
},
{
"Effect": "Allow",
"Action": "ssm:DescribeParameters",
"Resource": "*"
}
]
}
These permissions allow to fetch all entries for any service in the Dev environment.
For example, fetching all environment variables for a service called UserService:
aenv -e Dev -s UserService
To further limit access to only allow loading environment variables for a specific service like "UserService" we need to adapt the "Resource":
"Resource": [
"arn:aws:ssm:*:123456789098:parameter/Dev/UserService/*"
]
This is a work in progress!
Currently, the yubikey needs to be added as a virtual mfa and needs to be the first device in our Multi-factor authentication devices. Feel free to also add your Yubikey as a hardware mfa afterwards.(The AWS web console works flawless with multiple mfa's)
pip install --user yubikey-manager
Permission | Used in the code | Documentation | Comment |
---|---|---|---|
"ec2:DescribeTags" | clientEC2.describe_tags() | https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeTags.html | |
"sts:GetCallerIdentity" | clientSTS.get_caller_identity() | https://docs.aws.amazon.com/STS/latest/APIReference/API_GetCallerIdentity.html | Optional(No permissions are required to perform this operation.) |
"sts:GetSessionToken" | clientSTS.get_session_token() | https://docs.aws.amazon.com/STS/latest/APIReference/API_GetSessionToken.html | |
"ssm:GetParametersByPath" | clientSSMMFA.get_parameters_by_path() | https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-paramstore-access.html | |
"iam:ListMFADevices" | boto3.client('iam').list_mfa_devices() | https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_examples_iam_mfa-selfmanage.html | Optional! (At the moment not in use but as soon aws API supports hardware tokens this can be enabled to let aenv support hardware MFA's) |
tldr Minimal permissions:
“ec2:DescribeTags”
“sts:GetSessionToken”
“ssm:GetParametersByPath”
To enforce MFA authentication for all Prod parameters you can make use of the condition "MultiFactorAuthPresent" in your IAM permission.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ssm:PutParameter",
"ssm:DeleteParameter",
"ssm:GetParameterHistory",
"ssm:GetParametersByPath",
"ssm:GetParameters",
"ssm:ListTagsForResource",
"ssm:GetParameter",
"ssm:DeleteParameters"
],
"Resource": [
"arn:aws:ssm:*:123456789098:parameter/Prod/*"
],
"Condition": {
"Bool": {
"aws:MultiFactorAuthPresent": "true"
}
}
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "ssm:DescribeParameters",
"Resource": "*"
}
]
}
Add the condition "MultiFactorAuthPresent" to your IAM permission:
"Condition": {"Bool": {"aws:MultiFactorAuthPresent": "true"}}
Sample for sts:AssumeRole:
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Principal": {"AWS": "ACCOUNT-B-ID"},
"Action": "sts:AssumeRole",
"Condition": {"Bool": {"aws:MultiFactorAuthPresent": "true"}}
}
}
Now you need MFA authentication to run assume role commands. Sample call for this would be:
aenv -v -n -Y aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/example-role" --role-session-name AWSCLI-Session
# -v enables verbose mode
# -n puts aenv in only authentication mode
# -Y authenticates the session with your YubiKey, alternatively you could use -t or -T
AEnv uses the parameter store path to define the environment and service name, following this schema:
/<Environment>/<Service-Name>/
# Which could look like:
/Prod/CustomerManagement/DB/USER
# Or
/Prod/CustomerManagement/DB/PASSWORD
Having those two in place would enable our CustomerManagement service running in our Prod environment to access, the environment variables: SECRET_CUSTOMERMANAGEMENT_DB_USER and SECRET_CUSTOMERMANAGEMENT_DB_PASSWORD
With both parameters in place, your CustomerManagement application/service, launched with aenv, could now access the database with the provided username and password.
aenv -e Prod -s CustomerManagement java -jar service.jar
Every environment variable that is loaded with aenv starts with "SECRET_".
Then the service-name and path, separated by underliners. (of course in upper case)
For example:
/Prod/CustomerManagement/DB/USER
would be accessible with:
SECRET_CUSTOMERMANAGEMENT_DB_USER
or
/Prod/CustomerManagement/DB/PASSWORD/USER1
would be accessible with:
SECRET_CUSTOMERMANAGEMENT_DB_PASSWORD_USER1
More about environment variables: Guide to Unix/Environment Variables
Linux/Mac
aenv -e Dev -s UserService echo '$SECRET_USERSERVICE_UI_URL'
or
Windows
aenv -e Dev -s UserService echo %SECRET_USERSERVICE_UI_URL%
To access those environment variables you have to run your application/service with aenv.
aenv -e Dev -s UserService java -jar service.jar
//or
aenv -e Dev -s UserService python service2.py
Now these two services have access to all Dev environment variables for the UserService. Here are easy examples for Python and Kotlin:
Python:
import os
os.getenv('SECRET_USERSERVICE_HOSTNAME')
# For example conneting to a host depending on the environment:
....
hostname = os.getenv('SECRET_USERSERVICE_HOSTNAME')
....
Kotlin:
val envVar : String? = System.getenv("SECRET_USERSERVICE_HOSTNAME")
Linux/Mac running IntelliJ with Test environment variables for the UserService
aenv -e Test -s UserService "/Applications/IntelliJ\ IDEA\ CE.app/Contents/MacOS/idea"
This can come in handy if you want to debug something that only seems to occure in the Test environment.
AWS Server:
Easy!
Done by boto3. Boto3 automatically uses the in the instance role defined permissions.
(Details "Permissions" section)
Developer:
boto3 uses the aws CLI's authentication so make sure you set this up before ;)
By default, aenv uses the aws CLI default profile, but of course, you can choose the profile, that you want to use, simply do:
aenv -p <awscli profile name>
#or
aenv -h
#to see more configuration options
(More details in the "Usage" section)
MFA
Multi-factor authentication is highly suggested!
https://lmgtfy.com/?q=why+mfa+is+important
Ok, all jokes aside especially for production parameters your IAM users should require MFA authentication at least for production parameters.
At least in my humble opinion, this should be a "better be safe than sorry" point.
Especially for your production systems!
AEnv supports multiple MFA options, details in the "Usage" section, here the short overview:
# Normal virtual mfa token:
aenv -t <TOKEN>
# Asks for the token during runtime:
aenv -T
# leads to an interactive token query during runtime:
$ aenv -T
$ Please enter token:
#Yubikey Authenticator:
aenv -Y
- Add managing mode with TerminalMenu to read specific values
- Function to add new entries
- Function to update existing entries
- Function to delete existing entries
- refactor whole code base
- Add better permission error handling (Especially the auth part with Yubikey handling)
- Add check if service exists/can be read with proper error message
- Update and correct -h / --help output
- Add regex filter for only loading specific variables
- Add regex filter to leave out variables from loading
- add -P to save a default profile
- Update initial setup instructions + console output for this(ykman + output for missing service)
- Option to list all available environments / services(discover/list env / list services)
- Check for ykman on -Y calls improve output
- Currently only the fist MFA device of any given account is used -> add mfa device selection + option for default selection
- cleanup/refactor documentation / improve overall structure
- add auth only mode(no env and no service name given)
- Add more information about container mode and necessary IAM permissions
- Enhance local profile/config setup/usage
- Handle the region in the same way services are handled
- Load multiple services at once instead of concatenating multiple aenv calls ( "aenv -s Service1 aenv -s Service2 ")
- Add feature for only loading certain variables to speed up loading
- Add assume role feature to support this setup more ease -> https://aws.amazon.com/de/blogs/security/enhance-programmatic-access-for-iam-users-using-yubikey-for-multi-factor-authentication/
- Add testing
Inspired by:
Bug reports:
MIT Link
If this project provided value, and you want to give something back, you can give the repo a star or support by buying me a coffee.