AWS: Adding Webidentityprovider

[DagW]

When running monstache in EKS,
the recommended auth method is using IRSA (IAM Roles for service accounts).

Using this method of authentication will populate two ENV vars in your Pod,
AWS_ROLE_ARN - the role to assume
AWS_WEB_IDENTITY_TOKEN_FILE - the token to do it with

To enable that, we need to also enable the stscreds.WebIdentityRoleProvider

[tlierdotfr]

Hi @rwynn, do you know by when this feature will be released (maybe in v6.7.12 ) ?
In my use case, I have deployed monstache on EKS and my AWS IAM Role is binded to the Pod ServiceAccount.
I would prefer this auth method rather than using a static AccessKey / SecretKey.

[DagW]

We are currently using it by specifying a commit hash, and the auth method works great 👍 In a release would be even better :) lör 27 maj 2023 kl. 14:03 skrev Thomas ***@***.***>:

…

Hi @rwynn <https://github.com/rwynn>, do you know by when this feature will be released (maybe in v6.7.12 ) ? In my use case, I have deployed monstache on K8S and my AWS IAM Role is binded to the Pod ServiceAccount. I would prefer this auth method rather than using a static AccessKey / SecretKey. — Reply to this email directly, view it on GitHub <#676 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACGXG2P7WWLSBBBQBOHPB3XIHUSNANCNFSM6AAAAAAWPNOF6A> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tlierdotfr]

@DagW I have just tried and it's not working for me out of the box.
I've just adapted the toml config file for

# AWS Connection Strategy
[aws-connect]
region = "eu-west-3"
strategy = 5

Is there anything else to do for the ServiceAccount IAM role binding ot be used by monstache ?

[DagW]

You should make sure the env vars AWS_ROLE_ARN , AWS_WEB_IDENTITY_TOKEN_FILE and AWS_REGION are populated by the service account binding to your pod. Other than that, like you mentioned - I have this at the end of my conf file: [aws-connect] region = "x" strategy = 5 Den mån 29 maj 2023 kl 11:21 skrev Thomas ***@***.***>:

…

@DagW <https://github.com/DagW> I have just tried and it's not working for me out of the box. I just adapt the toml config file for # AWS Connection Strategy [aws-connect] region = "eu-west-3" strategy = 5 Is there anything else to do for the ServiceAccount IAM role binding ot be used by monstache ? — Reply to this email directly, view it on GitHub <#676 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACGXG2NQWDBTAMGSILJZDTXIRTBPANCNFSM6AAAAAAWPNOF6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DagW]

Whats the error that you're seeing? Den mån 29 maj 2023 kl 22:56 skrev Dag Wullt ***@***.***>:

…

You should make sure the env vars AWS_ROLE_ARN , AWS_WEB_IDENTITY_TOKEN_FILE and AWS_REGION are populated by the service account binding to your pod. Other than that, like you mentioned - I have this at the end of my conf file: [aws-connect] region = "x" strategy = 5 Den mån 29 maj 2023 kl 11:21 skrev Thomas ***@***.***>: > @DagW <https://github.com/DagW> I have just tried and it's not working > for me out of the box. > I just adapt the toml config file for > > # AWS Connection Strategy > [aws-connect] > region = "eu-west-3" > strategy = 5 > > Is there anything else to do for the ServiceAccount IAM role binding ot > be used by monstache ? > > — > Reply to this email directly, view it on GitHub > <#676 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AACGXG2NQWDBTAMGSILJZDTXIRTBPANCNFSM6AAAAAAWPNOF6A> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[tlierdotfr]

I confirm that the 3 env variables are correctly populated on my pod.

Here is the error I have : Unable to create Elasticsearch client: health check timeout: no Elasticsearch node available.

[DagW]

Might be a connection issue? Also, your Role is either the master role arn or correctly mapped to an opensearch internal role? mån 29 maj 2023 kl. 23:17 skrev Thomas ***@***.***>:

…

I confirm that the 3 env variables are correctly populated on my pod. Here is the error I have : Unable to create Elasticsearch client: health check timeout: no Elasticsearch node available. — Reply to this email directly, view it on GitHub <#676 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACGXG3RF7YVDYI2I5OAK5TXIUG5XANCNFSM6AAAAAAWPNOF6A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[DagW]

I do remember having that issue, maybe you can diagnose by doing the request manually first, with a sigv4-curl request; note You would get the credentials with STS $ aws sts assume-role-with-web-identity \ --role-arn $AWS_ROLE_ARN \ --role-session-name mh9test \ --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \ --duration-seconds 1000 > /tmp/irp-cred.txt $ export AWS_ACCESS_KEY_ID="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.AccessKeyId")" $ export AWS_SECRET_ACCESS_KEY="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SecretAccessKey")" $ export AWS_SESSION_TOKEN="$(cat /tmp/irp-cred.txt | jq -r ".Credentials.SessionToken")" $ rm /tmp/irp-cred.txt and then do a curl like; curl \ --verbose \ --request GET "https://...us-west-2.es.amazonaws.com" \ --aws-sigv4 "aws:amz:us-west-2:es" \ --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \ -H "x-amz-security-token:$AWS_SESSION_TOKEN" Sources: https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ https://code.dblock.org/2022/07/11/making-sigv4-authenticated-requests-to-managed-opensearch.html Den tis 30 maj 2023 kl 01:50 skrev Dag Wullt ***@***.***>:

…

Might be a connection issue? Also, your Role is either the master role arn or correctly mapped to an opensearch internal role? mån 29 maj 2023 kl. 23:17 skrev Thomas ***@***.***>: > I confirm that the 3 env variables are correctly populated on my pod. > > Here is the error I have : Unable to create Elasticsearch client: health > check timeout: no Elasticsearch node available. > > — > Reply to this email directly, view it on GitHub > <#676 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AACGXG3RF7YVDYI2I5OAK5TXIUG5XANCNFSM6AAAAAAWPNOF6A> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

[tlierdotfr]

Thanks a lot for this advice @DagW !

The error in curl response was :

{
	"error": {
		"root_cause": [
			{
				"type": "security_exception",
				"reason": "no permissions for [cluster:monitor/main] and User [name=arn:aws:iam::000000000000:role/iam-role-monstache, backend_roles=[arn:aws:iam::000000000000:role/iam-role-monstache], requestedTenant=null]"
			}
		],
		"type": "security_exception",
		"reason": "no permissions for [cluster:monitor/main] and User [name=arn:aws:iam::000000000000:role/iam-role-monstache, backend_roles=[arn:aws:iam::000000000000:role/iam-role-monstache], requestedTenant=null]"
	},
	"status": 403
}

And the root cause was quite simple ....
I forgot to bind the IAM role to an OpenSerch role..

After making this configuration in the "Security / Roles" menu, everything is now working perfectly fine !