Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform 0.13 crash on aws_cloudfront_distribution import #14798

Closed
peterpakos opened this issue Aug 24, 2020 · 9 comments · Fixed by #14844
Closed

Terraform 0.13 crash on aws_cloudfront_distribution import #14798

peterpakos opened this issue Aug 24, 2020 · 9 comments · Fixed by #14844
Assignees
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/cloudfront Issues and PRs that pertain to the cloudfront service.
Milestone

Comments

@peterpakos
Copy link

peterpakos commented Aug 24, 2020

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v0.13.0
+ provider registry.terraform.io/hashicorp/aws v3.3.0

Affected Resource(s)

  • aws_cloudfront_distribution

Terraform Configuration Files

resource "aws_cloudfront_distribution" "default" {
  origin {
    domain_name = aws_s3_bucket.default.bucket_regional_domain_name
    origin_id   = "S3-${aws_s3_bucket.default.id}"

    s3_origin_config {
      origin_access_identity = aws_cloudfront_origin_access_identity.default.cloudfront_access_identity_path
    }
  }

  enabled             = true
  is_ipv6_enabled     = false
  default_root_object = "index.html"

  aliases = [var.bucket_name]

  default_cache_behavior {
    allowed_methods  = ["GET", "HEAD"]
    cached_methods   = ["GET", "HEAD"]
    target_origin_id = "S3-${aws_s3_bucket.default.id}"

    forwarded_values {
      query_string = false

      cookies {
        forward = "none"
      }
    }

    viewer_protocol_policy = "redirect-to-https"
    min_ttl                = 0
    default_ttl            = 0
    max_ttl                = 0

    lambda_function_association {
      event_type   = "origin-request"
      include_body = false
      lambda_arn   = "arn:aws:lambda:us-east-1:225237029829:function:s3-cdn:2"
    }
  }

Debug Output

https://gist.github.com/peterpakos/eeffe7d7b94a0b0928832d4225e6bcdd

Panic Output

https://gist.github.com/peterpakos/05b9cf9d66195ddcc3c24f48e1da1880

Expected Behavior

Terraform import command successful with Cloudfront Distribution successfully imported to the state file.

Actual Behavior

Terraform import command crashed.

Steps to Reproduce

  1. terraform import module.s3-cdn.aws_cloudfront_distribution.default E32EAQRW369ZYR

Important Factoids

Cloudfront Distribution E32EAQRW369ZYR contains 2 Origins:

  • S3 Origin
  • Custom Origin

It also has 2 Behaviours configured:

  • Default (*) for Custom Origin
  • Path ui/* for S3 Origin.

The latter Behaviour also has Lambda@Edge Function associated with Viewer Request.

@ghost ghost added the service/cloudfront Issues and PRs that pertain to the cloudfront service. label Aug 24, 2020
@github-actions github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Aug 24, 2020
@bflad bflad added bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. and removed needs-triage Waiting for first response or review from a maintainer. labels Aug 24, 2020
@bflad
Copy link
Contributor

bflad commented Aug 24, 2020

Relevant portion of the crash log:

2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: panic: runtime error: invalid memory address or nil pointer dereference
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x38173bb]
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: goroutine 73 [running]:
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenForwardedValues(0x0, 0xc00152a4b0)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:430 +0x2b
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenCacheBehavior(0xc001825980, 0x5839f60)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:304 +0x22a
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenCacheBehaviors(0xc001520e40, 0x59fc4f0, 0x15, 0x5839f60)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:185 +0x89
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenDistributionConfig(0xc001a86800, 0xc0000f9200, 0xc001496740, 0x0)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:123 +0xa5b
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsCloudFrontDistributionRead(0xc001a86800, 0x4e5c540, 0xc00033b900, 0x97c6030, 0x5ae3920)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_cloudfront_distribution.go:729 +0x35e
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc00070cb00, 0x6490b00, 0xc001ab4700, 0xc001a86800, 0x4e5c540, 0xc00033b900, 0x0, 0x0, 0x0)
2020-08-24T11:05:34.544+0100 [DEBUG] plugin.terraform-provider-aws_v3.3.0_x5: 	/opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:281 +0x88

Looks like the query_string handling is missing a nil check here:

https://github.com/terraform-providers/terraform-provider-aws/blob/116a671279121171385dca2a9a6cc2bfdd15a612/aws/cloudfront_distribution_configuration_structure.go#L430

@anGie44
Copy link
Contributor

anGie44 commented Aug 26, 2020

The fix for this has been merged and will release with version 3.4.0 of the Terraform AWS Provider, likely out tomorrow.

@peterpakos
Copy link
Author

Great stuff, thanks!

@ghost
Copy link

ghost commented Aug 27, 2020

This has been released in version 3.4.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

@peterpakos
Copy link
Author

Unfortunately it still doesn't work with the latest terraform/provider:

Terraform v0.13.1
+ provider registry.terraform.io/hashicorp/aws v3.4.0

crash.log: https://gist.github.com/peterpakos/9b0a15f84a5e1afb9b1c136288218782

✗ terraform import 'module.s3-cdn[0].aws_cloudfront_distribution.default' E32EAQRW369ZYR
module.s3-cdn[0].aws_cloudfront_distribution.default: Importing from ID "E32EAQRW369ZYR"...
module.s3-cdn[0].aws_cloudfront_distribution.default: Import prepared!
  Prepared aws_cloudfront_distribution for import
module.s3-cdn[0].aws_cloudfront_distribution.default: Refreshing state... [id=E32EAQRW369ZYR]

Error: rpc error: code = Unavailable desc = transport is closing


panic: runtime error: invalid memory address or nil pointer dereference
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x445734f]
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: goroutine 88 [running]:
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenForwardedValues(0x0, 0xc0015f3b00)
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:430 +0x2f
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenCacheBehavior(0xc0007b5580, 0x64a34e0)
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:304 +0x245
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenCacheBehaviors(0xc001c76a20, 0x666980e, 0x15, 0x64a34e0)
2020-09-02T20:33:07.083+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:185 +0x89
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.flattenDistributionConfig(0xc0014cea00, 0xc000e330e0, 0xc001e30220, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/cloudfront_distribution_configuration_structure.go:123 +0xa5b
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsCloudFrontDistributionRead(0xc0014cea00, 0x5aba380, 0xc000908000, 0xa488e20, 0x6751da8)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_cloudfront_distribution.go:729 +0x35e
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0xc0000c26e0, 0x710b300, 0xc0014e9100, 0xc0014cea00, 0x5aba380, 0xc000908000, 0x0, 0x0, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:281 +0x88
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).RefreshWithoutUpgrade(0xc0000c26e0, 0x710b300, 0xc0014e9100, 0xc0019cb1f0, 0x5aba380, 0xc000908000, 0xc00000fe80, 0x0, 0x0, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema/resource.go:555 +0x1c2
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/internal/helper/plugin.(*GRPCProviderServer).ReadResource(0xc0007a91a0, 0x710b300, 0xc0014e9100, 0xc001e0e0c0, 0xc0007a91a0, 0xc0007a91b0, 0x6751d98)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/helper/plugin/grpc_provider.go:572 +0x42f
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5._Provider_ReadResource_Handler.func1(0x710b300, 0xc0014e9100, 0x6420300, 0xc001e0e0c0, 0xc0014e9100, 0x5c2c540, 0xc001e0e001, 0xc0014bb500)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5/tfplugin5.pb.go:3276 +0x86
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/plugin.Serve.func3.1(0x710b3c0, 0xc0015bee10, 0x6420300, 0xc001e0e0c0, 0xc0014bb4e0, 0xc0014bb500, 0xc0010a8ba0, 0x108fe18, 0x62896c0, 0xc0015bee10)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/plugin/serve.go:76 +0x87
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5._Provider_ReadResource_Handler(0x64f6280, 0xc0007a91a0, 0x710b3c0, 0xc0015bee10, 0xc001e0e060, 0xc00108a280, 0x710b3c0, 0xc0015bee10, 0xc001e10000, 0x200)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform-plugin-sdk/v2/internal/tfplugin5/tfplugin5.pb.go:3278 +0x14b
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000162700, 0x712c8a0, 0xc001095080, 0xc0014d6100, 0xc001482510, 0xa44ec50, 0x0, 0x0, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:1171 +0x50a
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: google.golang.org/grpc.(*Server).handleStream(0xc000162700, 0x712c8a0, 0xc001095080, 0xc0014d6100, 0x0)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:1494 +0xccd
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc000d1e200, 0xc000162700, 0x712c8a0, 0xc001095080, 0xc0014d6100)
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:834 +0xa1
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5: created by google.golang.org/grpc.(*Server).serveStreams.func1
2020-09-02T20:33:07.084+0100 [DEBUG] plugin.terraform-provider-aws_v3.4.0_x5:   /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/terraform-providers/terraform-provider-aws/vendor/google.golang.org/grpc/server.go:832 +0x204
2020/09/02 20:33:07 [ERROR] eval: *terraform.EvalRefresh, err: rpc error: code = Unavailable desc = transport is closing
2020/09/02 20:33:07 [ERROR] eval: *terraform.EvalSequence, err: rpc error: code = Unavailable desc = transport is closing
2020/09/02 20:33:07 [TRACE] [walkImport] Exiting eval tree: import module.s3-cdn[0].aws_cloudfront_distribution.default result
2020/09/02 20:33:07 [TRACE] vertex "import module.s3-cdn[0].aws_cloudfront_distribution.default result": visit complete
2020/09/02 20:33:07 [TRACE] vertex "module.s3-cdn[0].aws_cloudfront_distribution.default (import id \"E32EAQRW369ZYR\")": dynamic subgraph encountered errors
2020/09/02 20:33:07 [TRACE] vertex "module.s3-cdn[0].aws_cloudfront_distribution.default (import id \"E32EAQRW369ZYR\")": visit complete
2020-09-02T20:33:07.088+0100 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020/09/02 20:33:07 [TRACE] dag/walk: upstream of "module.s3-cdn (close)" errored, so skipping
2020-09-02T20:33:07.088+0100 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.4.0/darwin_amd64/terraform-provider-aws_v3.4.0_x5 pid=87888 error="exit status 2"
2020/09/02 20:33:07 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/hashicorp/aws\"] (close)" errored, so skipping
2020/09/02 20:33:07 [TRACE] dag/walk: upstream of "root" errored, so skipping
2020-09-02T20:33:07.287+0100 [DEBUG] plugin: plugin exited



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

SECURITY WARNING: the "crash.log" file that was created may contain
sensitive information that must be redacted before it is safe to share
on the issue tracker.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

@anGie44
Copy link
Contributor

anGie44 commented Sep 2, 2020

ahh thanks for following up on this @peterpakos! Looks like we'll have to add an explicit check for the nil value returned from the API; in this case the *cloudfront.ForwardedValues param seems to be the culprit

@dShringi
Copy link

I'm facing the same issue and yes I can confirm it is still not resolved after the update. Please let us know if there is a workaround for now. At the moment I have just removed the code for aws_cloudfront distribution and the state from the state file and it is working fine.

@ewbankkit
Copy link
Contributor

@dShringi Could you please open a new issue for this so that we can track and fix?
Thanks.

@ghost
Copy link

ghost commented Sep 26, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Sep 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/cloudfront Issues and PRs that pertain to the cloudfront service.
Projects
None yet
5 participants