Issue Generating PreSignedURL

[stack72]

Hi, I am trying to add support to Terraform for Cross Region Encrypted Replicas in RDS

I have the following code:

req, out := conn.CreateDBInstanceReadReplicaRequest(&opts)
preSignedUrl, presign_err := req.Presign(30 * time.Minute)
if presign_err != nil {
	return fmt.Errorf(`provider.aws: aws_db_instance: %s: error encountered
	calling Presign on request: %s `, identifier, presign_err)
}
opts.PreSignedUrl = aws.String(preSignedUrl)
err := req.Send()

The request looks as follows:

[DEBUG] [aws-sdk-go] DEBUG: Request rds/CreateDBInstanceReadReplica Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /?Action=CreateDBInstanceReadReplica&CopyTagsToSnapshot=false&DBInstanceClass=db.t2.large&DBInstanceIdentifier=tf-replica-db-1&DBSubnetGroupName=foobarbaz-test-1&DestinationRegion=us-west-2&KmsKeyId=arn%3Aaws%3Akms%3Aus-west-2%3A187416307283%3Akey%2Fc17820d6-bf40-40e2-b32a-dd0f9f431637&PubliclyAccessible=false&SourceDBInstanceIdentifier=arn%3Aaws%3Ards%3Aeu-central-1%3A187416307283%3Adb%3Afoobarbaz-test-terraform-1&Tags.member.1.Key=Name&Tags.member.1.Value=tf-replica-db-1&Version=2014-10-31&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI2TTU7QGVZGXTG4Q%2F20170223%2Fus-west-2%2Frds%2Faws4_request&X-Amz-Date=20170223T175605Z&X-Amz-Expires=1800&X-Amz-SignedHeaders=host&X-Amz-Signature=5e1183c0a53a786bde7a9aae77953a6ff8a76aecb0dc97f62b2c80d8ba15eb6f HTTP/1.1
Host: rds.us-west-2.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/0.9.0-dev
Accept-Encoding: gzip
-----------------------------------------------------

The response is as follows:

[aws-sdk-go] DEBUG: Response rds/CreateDBInstanceReadReplica Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 292
Content-Type: text/xml
Date: Thu, 23 Feb 2017 17:56:08 GMT
X-Amzn-Requestid: 591c7388-f9f1-11e6-9e33-b1f64c0ec258

<ErrorResponse xmlns="http://rds.amazonaws.com/doc/2014-10-31/">
  <Error>
    <Type>Sender</Type>
    <Code>InvalidParameterValue</Code>
    <Message>PreSignedUrl could not be authenticated.</Message>
  </Error>
  <RequestId>591c7388-f9f1-11e6-9e33-b1f64c0ec258</RequestId>
</ErrorResponse>
-----------------------------------------------------

This seems to be a similar issue to this

Any thoughts on what I seem to be doing wrong here?

Thanks

Paul

[xibz]

Hello @stack72, thank you for reaching out to us. I'll take a look into this. Once I am able to replicate it, I'll post back here.

Can you post what your input looks like?

[xibz]

@stack72 - I am unable to reproduce this, and behaves properly from what I can tell. Here is my source

package main                                                                                                               
                                                                                                                           
import (                                                                                                                   
  "fmt"                                                                                                                    
  "time"                                                                                                                   
                                                                                                                           
  "github.com/aws/aws-sdk-go/aws"                                                                                          
  "github.com/aws/aws-sdk-go/aws/session"                                                                                  
  "github.com/aws/aws-sdk-go/service/rds"                                                                                  
)                                                                                                                          
                                                                                                                           
func main() {                                                                                                              
  sess := session.New((&aws.Config{                                                                                        
    Region: aws.String("us-east-1"),                                                                                       
  }).WithLogLevel(aws.LogDebugWithRequestRetries | aws.LogDebugWithRequestErrors))                                         
                                                                                                                           
  svc := rds.New(sess)                                                                                                     
  req, out := svc.CreateDBInstanceReadReplicaRequest(&rds.CreateDBInstanceReadReplicaInput{                                
    DBInstanceIdentifier:       aws.String("fooinstance"),                                                                
    SourceDBInstanceIdentifier: aws.String("arn to instance"),                          
    SourceRegion:               aws.String("us-west-2"),                                                                   
    KmsKeyId:                   aws.String("us-east-1 key arn"),              
  })                                                                                                                       
  url, err := req.Presign(time.Minute * 5)                                                                                 
  fmt.Println(url)                                                                                                         
  fmt.Println(err)                                                                                                         
  fmt.Println(out)                                                                                                         
} 

Please let me know if you any issues with that example.

[stack72]

@xibz

thanks for this - so there is no need to run the req.Send()?

P.

[xibz]

@stack72 - If you are just looking to use the presigned URL, you do not need to send the request :).

[stack72]

got it - let me try that now

Will report back within the hour or so

Thanks for this

[stack72]

@xibz you are correct - that fixes my issue :) Thanks so much!

[stack72]

@xibz ok, this code works but it doesn't actually create the replica - we just go into a waiting pattern for it to actually get deployed and nothing happens:

The code looks as follows now:

req, out := conn.CreateDBInstanceReadReplicaRequest(&opts)
_, presign_err := req.Presign(time.Minute * 5)
if presign_err != nil {
	return fmt.Errorf(`provider.aws: aws_db_instance: %s: error encountered calling Presign on request: %s `, identifier, presign_err)
}
log.Printf("[DEBUG] CreateDBInstanceReadReplicaRequest Options: %#v", opts)
log.Printf("[DEBUG] CreateDBInstanceReadReplicaRequest Request: %#v", req)
log.Printf("[DEBUG] CreateDBInstanceReadReplicaRequest Output: %#v", out)

The responses are as follows:

 2017/02/24 01:04:22 [DEBUG] CreateDBInstanceReadReplicaRequest Options: {
   CopyTagsToSnapshot: false,
   DBInstanceClass: "db.t2.large",
   DBInstanceIdentifier: "tf-replica-db-1",
   DBSubnetGroupName: "foobarbaz-test-1",
   DestinationRegion: "us-west-2",
   KmsKeyId: "arn:aws:kms:us-west-2:187416307283:key/86185527-b4b7-47e6-9740-bd853ea52945",
   PubliclyAccessible: false,
   SourceDBInstanceIdentifier: "arn:aws:rds:eu-central-1:187416307283:db:foobarbaz-test-terraform-1",
   SourceRegion: "eu-central-1",
   Tags: [{
       Key: "Name",
       Value: "tf-replica-db-1"
     }]
 }

[DEBUG] CreateDBInstanceReadReplicaRequest Url: "https://rds.us-west-2.amazonaws.com/?Action=CreateDBInstanceReadReplica&CopyTagsToSnapshot=false&DBInstanceClass=db.t2.large&DBInstanceIdentifier=tf-replica-db-1&DBSubnetGroupName=foobarbaz-test-1&DestinationRegion=us-west-2&KmsKeyId=arn%3Aaws%3Akms%3Aus-west-2%3A187416307283%3Akey%2F86185527-b4b7-47e6-9740-bd853ea52945&PubliclyAccessible=false&SourceDBInstanceIdentifier=arn%3Aaws%3Ards%3Aeu-central-1%3A187416307283%3Adb%3Afoobarbaz-test-terraform-1&Tags.member.1.Key=Name&Tags.member.1.Value=tf-replica-db-1&Version=2014-10-31&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI2TTU7QGVZGXTG4Q%2F20170223%2Fus-west-2%2Frds%2Faws4_request&X-Amz-Date=20170223T231812Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=ef25d5c566a0d70b9e0594eab7c85ad6372243c2a2e0554a858bab14803c7c30"

2017/02/24 01:04:22 [DEBUG] CreateDBInstanceReadReplicaRequest Output: {
 
 }

The output is empty

Does your code actually create the replica? or just happen to do the same thing?

[xibz]

@stack72 - Yea, that just returns the URL. You must wget or call http.Get on the url. Let me know if that helps!

[stack72]

Hi @xibz

When i do the following now:

req, out := conn.CreateDBInstanceReadReplicaRequest(&opts)
presignUrl, presign_err := req.Presign(time.Minute * 5)
if presign_err != nil {
  return fmt.Errorf(`provider.aws: aws_db_instance: %s: error encountered
  calling Presign on request: %s `, identifier, presign_err)
}
resp, err := http.Get(presignUrl)
if err != nil {
  return err
}
log.Printf("[DEBUG] CreateDBInstanceReadReplicaRequest Url: %#v", presignUrl)
log.Printf("[DEBUG] CreateDBInstanceReadReplicaRequest Resp: %#v", resp)
log.Printf("[DEBUG] CreateDBInstanceReadReplicaRequest Options: %#v", opts)
log.Printf("[DEBUG] CreateDBInstanceReadReplicaRequest Output: %#v", out)

The response from the http.Get returns the following:

[DEBUG] CreateDBInstanceReadReplicaRequest Resp: &http.Response{Status:"400 Bad Request", StatusCode:400, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"X-Amzn-Requestid":[]string{"1196037c-fa20-11e6-b950-2db38d0d26b9"}, "Content-Type":[]string{"text/xml"}, "Content-Length":[]string{"292"}, "Date":[]string{"Thu, 23 Feb 2017 23:30:36 GMT"}}, Body:(*http.bodyEOFSignal)(0xc4210b1d40), ContentLength:292, TransferEncoding:[]string(nil), Close:true, Trailer:http.Header(nil), Request:(*http.Request)(0xc4210b2270), TLS:(*tls.ConnectionState)(0xc420f619e0)}

Any reason why the presigned url above would return a 400? Is it because my opts being passed to the Request have more than the 4 fields you have above?

[xibz]

@stack72 - What go version are you using? Can you provide what your input looks like when calling CreateDBInstanceReadReplicaRequest? Im wondering if it is something that is being specified in the opts like you had eluded to.

[stack72]

% go version                                                                                                     
go version go1.7.4 darwin/amd64

ok, so by only supplying the 4 options as follows:

opts := rds.CreateDBInstanceReadReplicaInput{
  SourceDBInstanceIdentifier: aws.String(v.(string)),
  DBInstanceIdentifier: aws.String(identifier),
  KmsKeyId: aws.String(kms_attr.(string)),
  SourceRegion: aws.String(sourceRegion.(string)),
}

I then get a URL that returns a 200 response

[DEBUG] CreateDBInstanceReadReplicaRequest Url: "https://rds.us-west-2.amazonaws.com/?Action=CreateDBInstanceReadReplica&DBInstanceIdentifier=tf-replica-db-1&DestinationRegion=us-west-2&KmsKeyId=arn%3Aaws%3Akms%3Aus-west-2%3A187416307283%3Akey%2F86185527-b4b7-47e6-9740-bd853ea52945&PreSignedUrl=https%3A%2F%2Frds.eu-central-1.amazonaws.com%2F%3FAction%3DCreateDBInstanceReadReplica%26DBInstanceIdentifier%3Dtf-replica-db-1%26DestinationRegion%3Dus-west-2%26KmsKeyId%3Darn%253Aaws%253Akms%253Aus-west-2%253A187416307283%253Akey%252F86185527-b4b7-47e6-9740-bd853ea52945%26SourceDBInstanceIdentifier%3Darn%253Aaws%253Ards%253Aeu-central-1%253A187416307283%253Adb%253Afoobarbaz-test-terraform-1%26Version%3D2014-10-31%26X-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Credential%3DAKIAI2TTU7QGVZGXTG4Q%252F20170223%252Feu-central-1%252Frds%252Faws4_request%26X-Amz-Date%3D20170223T234328Z%26X-Amz-Expires%3D300%26X-Amz-SignedHeaders%3Dhost%26X-Amz-Signature%3D6f8ca48d7ffb5ff18f8e75ecbd04b984e154cc50dc42c04159a2591b1f33b670&SourceDBInstanceIdentifier=arn%3Aaws%3Ards%3Aeu-central-1%3A187416307283%3Adb%3Afoobarbaz-test-terraform-1&Version=2014-10-31&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI2TTU7QGVZGXTG4Q%2F20170223%2Fus-west-2%2Frds%2Faws4_request&X-Amz-Date=20170223T234328Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=7ad992cea3fae626ed9c8a61af63e6d462b61c91f463c98180f9add36e5fcd50"

[DEBUG] CreateDBInstanceReadReplicaRequest Resp: &http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"X-Amzn-Requestid":[]string{"e0f6e40d-fa21-11e6-99e0-5d0069e0435b"}, "Content-Type":[]string{"text/xml"}, "Vary":[]string{"Accept-Encoding"}, "Date":[]string{"Thu, 23 Feb 2017 23:43:34 GMT"}}, Body:(*http.bodyEOFSignal)(0xc4210d0900), ContentLength:-1, TransferEncoding:[]string{"chunked"}, Close:false, Trailer:http.Header(nil), Request:(*http.Request)(0xc421028410), TLS:(*tls.ConnectionState)(0xc420e04e40)}

But the output from the request is still empty at this point so, no replica is being created:

[DEBUG] CreateDBInstanceReadReplicaRequest Output: {
 }

[xibz]

The output is not going to be populated due to it not being unmarshaled from the http.Get. You would need to read the response body from the http.Get and unmarshal into the out shape. So, Im certain if you look at your instances, you'll see a new instance being created.

[stack72]

yes i see it being created now - thanks!

[xibz]

Yep! No problem @stack72. If you run into any other issue, please feel free to reach out :)