Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS provider crash on 0.7.5 (multiple resources) #9272

Closed
akerl opened this issue Oct 7, 2016 · 10 comments
Closed

AWS provider crash on 0.7.5 (multiple resources) #9272

akerl opened this issue Oct 7, 2016 · 10 comments

Comments

@akerl
Copy link

akerl commented Oct 7, 2016

Terraform Version

ubuntu@box878:~/aws-account$ terraform version
Terraform v0.7.5

ubuntu@box878:~/aws-account$ uname -a
Linux box878.localdomain 3.13.0-91-generic #138-Ubuntu SMP Fri Jun 24 17:00:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Affected Resource(s)

Seems to change w/ each run. I've seen failures for any of the following, though cloudfront seems most common:

  • aws_cloudfront_distribution
  • aws_iam_user_policy
  • aws_s3_bucket

Terraform Configuration Files

The errors are occurring in my CircleCI builds for https://github.com/akerl/aws-account/

Example failure: https://circleci.com/gh/akerl/aws-account/23

Debug Output

https://gist.github.com/anonymous/365a7fab5342ea10724f7040f595ba91

Panic Output

https://gist.github.com/anonymous/3e19fa70b2536e8c4db87a7d6106feec

Expected Behavior

Plan should have completed successfully

Actual Behavior

Crashes with:

panic: runtime error: invalid memory address or nil pointer dereference
2016/10/07 03:21:51 [DEBUG] plugin: terraform: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x97776b]

Steps to Reproduce

  1. terraform plan

Appears to need to be run from Linux system; I couldn't reproduce on my Mac with 0.7.5 (also didn't occur on 0.7.4 on Linux or OSX)

Important Factoids

Running in CircleCI, test user has privilege set described here: https://github.com/akerl/aws-account/blob/master/akerl/aws-account/main.tf

@kwilczynski
Copy link
Contributor

@akerl hi there! Thank you for letting us know, and I am very sorry that you have problems with Terraform! We are going to have a look and see what can be the problem and keep you updated.

@kwilczynski
Copy link
Contributor

@akerl hi!

The failure in https://circleci.com/gh/akerl/aws-account/23 is related to the lack of IAM user permissions:

2016/10/07 03:17:19 [DEBUG] plugin: terraform: -----------------------------------------------------
2016/10/07 03:17:19 [DEBUG] plugin: terraform: aws-provider (internal) 2016/10/07 03:17:19 [DEBUG] [aws-sdk-go] DEBUG: Response cloudfront/ListTagsForResource2016_09_07 Details:
2016/10/07 03:17:19 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2016/10/07 03:17:19 [DEBUG] plugin: terraform: HTTP/1.1 403 Forbidden
2016/10/07 03:17:19 [DEBUG] plugin: terraform: Connection: close
2016/10/07 03:17:19 [DEBUG] plugin: terraform: Content-Length: 370
2016/10/07 03:17:19 [DEBUG] plugin: terraform: Content-Type: text/xml
2016/10/07 03:17:19 [DEBUG] plugin: terraform: Date: Fri, 07 Oct 2016 03:17:19 GMT
2016/10/07 03:17:19 [DEBUG] plugin: terraform: X-Amzn-Requestid: 8e448477-8c3c-11e6-ae51-1b2f2a25b5df
2016/10/07 03:17:19 [DEBUG] plugin: terraform: 
2016/10/07 03:17:19 [DEBUG] plugin: terraform: <?xml version="1.0"?>
2016/10/07 03:17:19 [DEBUG] plugin: terraform: <ErrorResponse xmlns="http://cloudfront.amazonaws.com/doc/2016-09-07/"><Error><Type>Sender</Type><Code>AccessDenied</Code><Message>User: arn:aws:iam::764218738161:user/akerl-aws-account-circleci is not authorized to perform: cloudfront:ListTagsForResource</Message></Error><RequestId>8e448477-8c3c-11e6-ae51-1b2f2a25b5df</RequestId></ErrorResponse>

You would need to add necessary permissions so that the IAM user. Whereas, we need to address the panic by handling this error gracefully (so this is a bug for us to fix).

@kwilczynski
Copy link
Contributor

Content of the stack trace for reference:

panic: runtime error: invalid memory address or nil pointer dereference
2016/10/07 02:47:56 [DEBUG] plugin: terraform: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x97776b]
2016/10/07 02:47:56 [DEBUG] plugin: terraform: 
2016/10/07 02:47:56 [DEBUG] plugin: terraform: goroutine 752 [running]:
2016/10/07 02:47:56 [DEBUG] plugin: terraform: panic(0x289bca0, 0xc4200140e0)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/go/src/runtime/panic.go:500 +0x1a1
2016/10/07 02:47:56 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/builtin/providers/aws.tagsToMapCloudFront(0x0, 0x29)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/gopath/src/github.com/hashicorp/terraform/builtin/providers/aws/tagsCloudFront.go:93 +0x5b
2016/10/07 02:47:56 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/builtin/providers/aws.resourceAwsCloudFrontDistributionRead(0xc421248660, 0x25fa680, 0xc42044a780, 0x0, 0x1b)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/gopath/src/github.com/hashicorp/terraform/builtin/providers/aws/resource_aws_cloudfront_distribution.go:555 +0x477
2016/10/07 02:47:56 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc420ba4c60, 0xc421358ff0, 0x25fa680, 0xc42044a780, 0xc4209b9128, 0x1, 0x18)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/gopath/src/github.com/hashicorp/terraform/helper/schema/resource.go:259 +0x131
2016/10/07 02:47:56 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc420c08480, 0xc420f1f540, 0xc421358ff0, 0x0, 0xc42072dc20, 0xc42076d9a8)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/gopath/src/github.com/hashicorp/terraform/helper/schema/provider.go:203 +0x91
2016/10/07 02:47:56 [DEBUG] plugin: terraform: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc4208be2c0, 0xc42135ad60, 0xc42135bb20, 0x0, 0x0)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/gopath/src/github.com/hashicorp/terraform/plugin/resource_provider.go:482 +0x4e
2016/10/07 02:47:56 [DEBUG] plugin: terraform: reflect.Value.call(0xc420342840, 0xc420029358, 0x13, 0x2e5b318, 0x4, 0xc420e97ed0, 0x3, 0x3, 0x44e7d40, 0xc420f1f500, ...)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/go/src/reflect/value.go:434 +0x5c8
2016/10/07 02:47:56 [DEBUG] plugin: terraform: reflect.Value.Call(0xc420342840, 0xc420029358, 0x13, 0xc420e97ed0, 0x3, 0x3, 0xc4211d3194, 0xc420f393e0, 0xc420f39380)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/go/src/reflect/value.go:302 +0xa4
2016/10/07 02:47:56 [DEBUG] plugin: terraform: net/rpc.(*service).call(0xc420c3c140, 0xc420c3c100, 0xc4208c2858, 0xc420ba6680, 0xc4210c44a0, 0x25fc840, 0xc42135ad60, 0x16, 0x25fc880, 0xc42135bb20, ...)
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/go/src/net/rpc/server.go:383 +0x148
2016/10/07 02:47:56 [DEBUG] plugin: terraform: created by net/rpc.(*Server).ServeCodec
2016/10/07 02:47:56 [DEBUG] plugin: terraform:  /opt/go/src/net/rpc/server.go:477 +0x421

@kwilczynski
Copy link
Contributor

@akerl I had a look at other builds failing, and it is the same resource each time:

aws_cloudfront_distribution

This is due to lack of permission on the IAM user side, which causes nil pointer dereference at the following line:

/opt/gopath/src/github.com/hashicorp/terraform/builtin/providers/aws/tagsCloudFront.go:93

@akerl
Copy link
Author

akerl commented Oct 7, 2016

Interesting. My apologies for not catching that; I'm too used to the AWS provider calling out 403s directly as errors.

I'll dig a bit and get my user policy lined up to sort this out, so I can confirm which new perm is required that it doesn't have. For the failures on the other resources, is it possible that's just other stuff it happened to be processing when it crashed (which would explain why the other resources that fail do so transiently)?

@akerl
Copy link
Author

akerl commented Oct 7, 2016

Confirmed that "cloudfront:List*" is what I needed to fix this

@kwilczynski
Copy link
Contributor

@akerl hi! I am glad you were able to fix the issue.

I am going to work on a fix to handle the lack of permissions to tag more gracefully, so that it does not cause a crash.

@kwilczynski
Copy link
Contributor

@akerl hi there! I have send a Pull Request which should address the panic due to nil pointer dereference due to lack of permission to use EC2 tags with CloudFront Distribution.

I have tested it manually, and it seem to do the right thing, but I would be very grateful if you could test your setup and/or configuration using patched Terraform, if possible.

If you have a moment, would you be able to help with this?

@radeksimko
Copy link
Member

Fixed via #9298

@ghost
Copy link

ghost commented Apr 21, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 21, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants