plan crahes updating aws_db_parameter_group #4294
Labels
Comments
|
@j4ybee is this the first time this has happened? Have you recently upgraded? |
|
Based on where the crash occurred it would appear that the RDS API returned a nil Seems like |
|
Yes, this is the first time. I worked on 0.6.6 the past weeks. When the
|
|
@j4ybee 1 more question, do you have any custom db_parameter_groups or are you using the defaults? |
|
I'm experiencing same issue but with MySQL. Everything worked well last week. I'm using terraform 0.6.8. Here is my RDS definition: |
|
Ah! Thanks @marcinwalczyk. So it looks like empty parameter groups are the issue! Sending a PR now to fix |
|
My parameter group is an unchanged copy of the "default.mysql5.5" standard On Mon, Dec 14, 2015 at 9:09 AM, Paul Stack notifications@github.com
|
|
so this is very strange, I have a test that covers this in Terraform. The test looks as follows: When I pass the following config into the test, it passes: Test Result: The only thing I am going to do differently in a PR is to check that there are Parameters in the group before trying to flatten them |
|
Sorry, I have to correct myself. @stack72 per definition I am also using an empty parameter group! |
|
Thanks @j4ybee and @marcinwalczyk - there is a PR now open that should fix this :) |
|
I've built terraform with your fix and the issue still exists. Just in case I'm including output from my execution. What additional information can I post in order to help with investigation |
|
Hello, Same for me, tried the fix and issue still here. I was in 0.6.7 and everything was working fine until few hours ago. I pass a parametergroup to all my databases. Here is my trace if it can help : |
|
Thanks @Seraf - so you were in 0.6.7 and then upgraded and got the error? Or the error still exists in 0.6.7? |
|
@stack72 I got the error without changing terraform or anything. I did an apply the 08 December and it worked. Without changing anything, today it doesn't work. I guess Amazon changed something on their side ... |
|
interesting! Looking down that line of thought now |
|
thanks for testing this @marcinwalczyk - that behaviour follows along the same line of thought that a test was already in place for this. I am trying to investigate what else it could be right now |
|
@stack72 if it can help, here is a And the parameter group : |
|
@stack72 I don't know I'm now able to do a plan |
|
@Seraf I just ran an acceptance test with your exact config from above and all passed :( This is most strange The config was as follows: |
|
@stack72 could it be that the API behaves differently in different AWS regions? I am working in |
|
Changing now to use that region.... stand by :) |
|
@j4ybee config is now this And I get the following result: |
|
@stack72 but we are few to have the error even if the tests PASS. So I guess tests missed something. As said in my last comment, I found that removing the flattenParameters and using Maybe tests are faking a AWSClient and hardcode a rdsconn property, that could explain why tests are okay on the structure, but fails to flat the datastructure ? These are just guesses, I have never even looked terraform code and I'm not a Go developper ;-) |
|
@Seraf the tests are actually creating real resources in the AWS console. Flatten parameters is a method that is required to turn an array of []*rds.Parameter into a map for string representation: e.g. By using this flattened structure, we can easily identify changes to the parameters. I am still trying to investigate more here P. |
|
so I just ran this code with Terraform 0.6.8 in the same way you will have - plan, apply, destroy (not acceptance tests) I get the following https://gist.github.com/stack72/ea3af8e87971c62a2a6e P. |
|
@stack72 the problem is not on the parameter group itself, but on the rds database WITH this group :) I can terraform postgresql rds which doesn't have parameter groups and it works. |
|
@Seraf the rds instance only takes a |
|
@stack72 but |
|
+1 |
|
On the params passed to flattenParameters() I get : There's no ParameterValue attribute for the last item with the server time zone. Here is the error I guess ;) : https://github.com/hashicorp/terraform/blob/master/builtin/providers/aws/structure.go#L404 |
|
@Seraf is this from your defined config? |
|
@stack72 I confirm that was the problem : That makes my plan OK. I don't know where this time_zone parameter come from and if it was present before. But I guess that was added by AWS recently as we are a lot with different version to have the same bug. I'm not sure how the parameter Here is my db instance : |
This seems to be the crux of it, and likely the simple fix. I'm testing things now, I'd like to understand why this was added, and when, but we may not find out... I suspect it's on the API side either way. This seems to only be added if the db parameter group is actually used by a database(?). Attempts to reproduce it have failed when only using the db group itself... still testing things out |
|
Hey friends – @stack72 and I have been unable to reproduce this, but clearly something is up 😱 The Can I get a summary of what regions this has been found in, and are you using a classic or VPC environment? I've tested several regions in VPCs with no luck, I'm trying class but it's pretty slow going... Thanks! |
|
@catsby I guess it's a combination of differents parameters as you said as there's not a lot of activity on this thread. Anyway thanks for the support :) |
|
Found in EU (Ireland) for me. That is |
|
@catsby VPC @ I didn't find any API changes, but there have been additional features released to AWS RDS on december 4th and 12th that could account for a change in behavior: I will try to get a quote on that from AWS... |
|
I'm using a VPC on eu-west-1 |
|
I've opened a patch for this here #4318 Unfortunately I can't reproduce the issue, so while I'm sure it will behave correctly, I can't verify. Can someone who's experiencing this issue try it out? |
catsby
added
the
waiting-response
|
@catsby I successfully tested your patch. Thx for the support! |
jen20
removed
the
waiting-response
|
It does the job. Thanks ! |
|
Hey y'all, @catsby, AWS has confirmed that this was a bug in the RDS API that is now resolved. I can confirm that the parameter in question ("time_zone") doesn't have an empty "ParameterValue" anymore. But I guess it still is a good thing to have the fix in place to be more resilient here. |
|
Thanks @j4ybee ! Is there a publicly accessible (or at least accessible with any AWS Account) link for that response, just to record for posterity? |
This was referenced Dec 15, 2015
Closed
|
Goog point @catsby. Unfortunately, I only have a business support ticket which is not public. I bounced this back to AWS... |
|
I'd like to bill them for a day's debugging LOL (In actual seriousness, glad to see it's actually fixed at the source now) |
|
Thank you @j4ybee |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
There is a crash in AWS provider's
aws_db_parameter_group. I did not change anything in my tf file so maybe there is a change in the underlying AWS API. I use release is 0.6.8.The text was updated successfully, but these errors were encountered: