Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SigV4 sign arbitrary requests #1784

Open
rhboyd opened this issue Jul 16, 2019 · 16 comments
Open

SigV4 sign arbitrary requests #1784

rhboyd opened this issue Jul 16, 2019 · 16 comments
Labels
feature-request This issue requests a feature. needs-review This issue or pull request needs review from a core team member. p2 This is a standard priority issue

Comments

@rhboyd
Copy link

rhboyd commented Jul 16, 2019

Botocore Sigv4 signs all AWS API Requests. It should expose the ability to sign arbitrary https requests.

API Gateway has an AWS_IAM auth mechanism and this results in needed to sigv4 sign https requests to domains not included in the usual Python SDK. As more databases (like Neptune) offer AWS IAM auth to manage database permissions, this problem will appear more frequently. The current "recommended" approach is to use a random third-party library named "aws-requests-auth" but that library isn't maintained by AWS and I'd really like to avoid asking new devs to Google "AWS SIgv4 Python", which returns about a dozen different repos. Should AWS be encouraging people to use 3P libraries for something as important as signing requests?

I propose that botocore expose some functionality to attach SigV4 Auth to any request. I think this can be accomplished by adding a def __call__(self, request) method to the SigV4Auth class and I'd even be willing to implement and test this feature if the botocore team agrees that this feature is needed.

I hacked together a way to perform the signing with the existing functionality, but it's pretty sloppy and requires building two requests in parallel then stripping the auth bits out of one so that we can send the other.

https://gist.github.com/rhboyd/1e01190a6b27ca4ba817bf272d5a5f9a

@benkehoe
Copy link

benkehoe commented Jul 16, 2019

In my ideal world, this is available on the Session object, so I could do something like session = botocore.Session(); response = requests.get(url, auth=session.signer(service='execute-api')) or something similar. And then also made available on the boto3 Session object, so I don't have to muck about getting the botocore session out of it.

@swetashre swetashre self-assigned this Aug 19, 2019
@swetashre
Copy link
Contributor

@rhboyd -Thank you for your post. I would mark this as a feature request.

@swetashre swetashre added the feature-request This issue requests a feature. label Aug 19, 2019
@tyldavis
Copy link

tyldavis commented Feb 11, 2020

I was surprised to find this wasn't already a feature! This would be hugely beneficial for AWS_IAM secured API Gateway endpoints.

Edit: I am especially surprised after finding this is a feature of the Go SDK, and in fact it appears all other SDKs support this already.

@ryansb
Copy link

ryansb commented Apr 21, 2020

I'd also be happy to do implementation work on this, as long so the botocore team is ok with the new __call__ on SigV4Auth.

@benkehoe
Copy link

It could be a new wrapper class with a __call__ method

@richardhboyd
Copy link

I've discovered a cleaner way to do this lately

import boto3
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
import requests

session = boto3.Session()
credentials = session.get_credentials()
creds = credentials.get_frozen_credentials()

def signed_request(method, url, data=None, params=None, headers=None):
    request = AWSRequest(method=method, url=url, data=data, params=params, headers=headers)
    # "service_name" is generally "execute-api" for signing API Gateway requests
    SigV4Auth(creds, "service_name", REGION).add_auth(request)
    return requests.request(method=method, url=url, headers=dict(request.headers), data=data)

def main():
    url = f"my.url.example.com/path"
    data = {"environmentId": self._environment_id}
    headers = {'Content-Type': 'application/x-amz-json-1.1'}
    response = signed_request(method='POST', url=url, data=data, headers=headers)

if __name__ == "__main__":
    main()

@tyldavis
Copy link

I've discovered a cleaner way to do this lately

[...]

Isn't this basically the same thing as your original Gist (I'm assuming @rhboyd and @richardhboyd are the same person... apologies if not), but without calling AWSRequest.prepare()? Looking at the source it seems like it isn't doing much anyway.

@richardhboyd
Copy link

yeah, the request.headers part can also be re-used for websockets that need IAM Auth (Neptune Gremlin connections) and the credentials work in a more environment agnostic way as well.

@RahulPATK
Copy link

@rhboyd can we also pass security token as well? I need to pass assume role credentials along with signed header?

@richardhboyd
Copy link

can you clarify the question a little bit? you can sign requests using the security token as well but you shouldn't have to pass the actual credentials in the header. the credentials should never be sent with a request (only a signature made from the credentials should be sent)

@RahulPATK
Copy link

@richardhboyd thanks for coming back, so when I tried your botocore with AWSRequest and Sig4Vuth approach I was getting that 'code': 'ACCESS_DENIED', 'message': 'User: arn:aws:sts::myrole is not authorized to perform: execute-api:Invoke on resource: soyrcerole /GET/sites with an explicit deny'}

I tried second approach where I manually signed the signature using aws documentation(https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html) its saying security token is invalid 'message': 'The security token included in the request is invalid, so I thought to pass security token and provided signed again, I was getting INVALID_REQUEST The request signature we calculated does not match the signature you provided.

Checkyour AWSSecretAccessKey and signingmethod.Consult the service documentation for details.TheCanonical String for this request should have been. and just for FYI, its working on POSTMAN when I'm passing assume role credns.

@richardhboyd
Copy link

the first message meant that the request was signed properly but you didn't have permission to invoke the API. what does the API's resource policy look like?

@RahulPATK
Copy link

RahulPATK commented Nov 17, 2021

I confronted same to use but they have added my lambda exn role in there config, and api resource policy expects credentials need to be added to a header in the request somehow. But the kind of header the request needs is to do with sigv4 signing

@richardhboyd
Copy link

richardhboyd commented Nov 17, 2021 via email

@RahulPATK
Copy link

-API Gateway API
-API Gateway execution Role (let's call this RoleA) -- Configured
-Lambda Function Execution Role (RoleB) -- Configured
-Role the signs requests that are sent to the API (RoleC)

The API Gateway resource policy needs to allow RoleC to "execute-api" for any path in the API that is needed -- need to check

RoleA needs permission to invoke the Lambda Function and needs to have a trust policy that lets the API Gateway service assume that role -- Configured

@RahulPATK
Copy link

@richardhboyd its done, issue was with headers,

so, first, I manually signed, using aws documentation, https://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html,
second, please make sure to add security token when using assume role
third, please make sure canonical String will in appropriate order

bingo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request This issue requests a feature. needs-review This issue or pull request needs review from a core team member. p2 This is a standard priority issue
Projects
None yet
Development

No branches or pull requests

9 participants