Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long secrets cause an EOF error #6

Closed
jasonamyers opened this issue Jul 7, 2017 · 7 comments
Closed

Long secrets cause an EOF error #6

jasonamyers opened this issue Jul 7, 2017 · 7 comments

Comments

@jasonamyers
Copy link

jasonamyers commented Jul 7, 2017

I've encountered an odd issue as we've begun to use this software more and more. We have a password in the list below it's data.credstash_secret.redshift_password that is 64 characters long. We can use the module for all of the secrets but it. We've put several eyes on it looking for a reason this would occur: (context, missing secret, etc), and we've failed to find anything. This became more confusing when I tried to add that value in the decrypt tests in secrets_tests.go, but it didn't error there. If we remove that one long secret from our terraform config, all the other secrets work great. I'm totally happy to help try to assist with this issue code wise, I'm just struggling for where to start looking for this issue since the tests passed.

The key is digest SHA256. We're using TF 0.8.8 and the Version 1.0 of your provider.

Here is the output from a terraform plan:

Error refreshing state: 4 error(s) occurred:

  • data.credstash_secret.redshift_password: unexpected EOF
  • data.credstash_secret.rds_password: unexpected EOF
  • data.credstash_secret.juicebox_password: unexpected EOF
  • data.credstash_secret.pager_duty_api_key: unexpected EOF

TF_LOG=debug output:

2017/07/07 07:32:40 [DEBUG] plugin: terraform: terraform-provider (internal) 2017/07/07 07:32:40 [DEBUG] Initializing remote state client: s3
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: panic: runtime error: invalid memory address or nil pointer dereference
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x9a4d7]
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash:
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: goroutine 51 [running]:
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: panic(0x5e4360, 0xc42000c180)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/opt/boxen/homebrew/Cellar/go/1.7.4_1/libexec/src/runtime/panic.go:500 +0x1a1
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/credstash.getString(0xc4204ba750, 0x6924e2, 0x4, 0x4, 0x6a1e18, 0x18, 0xc420198b60)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/credstash/secret.go:260 +0x67
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/credstash.getStringAndDecode(0xc4204ba750, 0x6924e2, 0x4, 0x6f6390, 0x13, 0x0, 0x0, 0xc42030f000, 0xc42035f110)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/credstash/secret.go:264 +0x43
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/credstash.keyMaterialFromDBItem(0xc4204ba750, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/credstash/secret.go:230 +0x1b9
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/credstash.getLatestVersion(0x9ec140, 0xc420318140, 0xc4203160c0, 0x24, 0xc42000cc30, 0x10, 0x0, 0x0, 0x0, 0x0, ...)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/credstash/secret.go:204 +0x4c3
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/credstash.getKeyMaterial(0x9ec140, 0xc420318140, 0xc4203160c0, 0x24, 0x0, 0x0, 0xc42000cc30, 0x10, 0x0, 0x0, ...)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/credstash/secret.go:157 +0xc3
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/credstash.secret.get(0x9ec140, 0xc420318140, 0x9e8200, 0xc420318130, 0xc4203160c0, 0x24, 0xc42000cc30, 0x10, 0x0, 0x0, ...)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/credstash/secret.go:59 +0x9e
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/credstash.GetSecret(0xc4203160c0, 0x24, 0xc42000cc30, 0x10, 0x0, 0x0, 0xc42000cc80, 0x9, 0xc4203163f0, 0x0, ...)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/credstash/secret.go:50 +0x11a
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: main.dataSourceSecretRead(0xc42032e120, 0x60d1e0, 0xc420015260, 0x28, 0xc4203003d1)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/datasource_secrets.go:61 +0x503
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).ReadDataApply(0xc420012b40, 0xc420310140, 0x60d1e0, 0xc420015260, 0xc4202c2978, 0x1, 0x18)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:207 +0xda
2017/07/07 07:32:41 [DEBUG] plugin: terraform: aws-provider (internal) 2017/07/07 07:32:41 [WARN] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).ReadDataApply(0xc420012ba0, 0xc420314050, 0xc420310140, 0x0, 0x0, 0x0)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:351 +0x91
2017/07/07 07:32:41 [DEBUG] plugin: terraform: aws-provider (internal) 2017/07/07 07:32:41 [INFO] AWS Auth provider used: "EnvProvider"
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: github.com/sspinc/terraform-provider-credstash/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).ReadDataApply(0xc420014920, 0xc42030c0c0, 0xc42030c1b0, 0x0, 0x0)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/Users/tomi/src/github.com/sspinc/terraform-provider-credstash/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:565 +0x4e
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: reflect.Value.call(0xc420083b60, 0xc4200ac828, 0x13, 0x692086, 0x4, 0xc42035fed0, 0x3, 0x3, 0x0, 0x0, ...)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/opt/boxen/homebrew/Cellar/go/1.7.4_1/libexec/src/reflect/value.go:434 +0x5c8
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: reflect.Value.Call(0xc420083b60, 0xc4200ac828, 0x13, 0xc42035fed0, 0x3, 0x3, 0x0, 0x0, 0x0)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/opt/boxen/homebrew/Cellar/go/1.7.4_1/libexec/src/reflect/value.go:302 +0xa4
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: net/rpc.(*service).call(0xc4202d4880, 0xc4202d4840, 0xc4202ceae8, 0xc4202cca80, 0xc4202dad20, 0x57ef60, 0xc42030c0c0, 0x16, 0x57efa0, 0xc42030c1b0, ...)
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/opt/boxen/homebrew/Cellar/go/1.7.4_1/libexec/src/net/rpc/server.go:383 +0x148
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: created by net/rpc.(*Server).ServeCodec
2017/07/07 07:32:41 [DEBUG] plugin: terraform-provider-credstash: 	/opt/boxen/homebrew/Cellar/go/1.7.4_1/libexec/src/net/rpc/server.go:477 +0x421```
@tmichel
Copy link

tmichel commented Jul 16, 2017

This is rather interesting. It seems like that the DynamoDB response somehow contains the key but the value is nil. Even more interesting that we have longer values than 64 characters. We store public keys and they work just fine. There are no integration tests at the moment so it's not a surprise that this works in the tests.

The issue is clearly that the code does not handle the nil value scenario. It blindly dereferences whatever comes out of the DynamoDB response. We can easily fix this but that would not solve the underlying issue. @jasonamyers if you could provide a key-value pair that produces the error that would be super helpful in debugging.

I will probably have time next weekend to look into this.

@jasonamyers
Copy link
Author

jasonamyers commented Jul 21, 2017

Hopefully this is what you are looking for. It's not the exact values but I preserved all the fidelity I could and replaced via 🍪 when I couldn't

  name = "cookybox.db.default.password.cookieprod"

  context = {
    app         = "cookiebox"
    app_id      = "e50656e9-5219-4799-adab-ea1ef51deb36"
    environment = "cookieprod"
  }
}

value of key:
CookIemhAf3cHBH0p4rmVxRll9KDfTHMmMcookiESWtQYFWH8pag7YnMoccOOkie

@tmichel
Copy link

tmichel commented Jul 23, 2017

I tried to reproduce the issue with no success.

I used the values provided by @jasonamyers in the following terraform configuration for testing:

provider "credstash" {
  region = "us-east-1"
  table  = "credstash-test"
}

data "credstash_secret" "db_password" {
  name = "cookybox.db.default.password.cookieprod"

  context = {
    app         = "cookiebox"
    app_id      = "e50656e9-5219-4799-adab-ea1ef51deb36"
    environment = "cookieprod"
  }
}

resource "null_resource" "echo" {
  triggers {
    id = "${data.credstash_secret.db_password.id}"
  }

  provisioner "local-exec" {
    command = "echo password=${data.credstash_secret.db_password.value}"
  }
}

Everything works as expected: the password is echoed to stdout. I tried both v0.1.0 (terraform 0.8.8) and v0.1.1 (terraform 0.9.9) versions.

@jasonamyers does credstash get work as expected for you? Maybe there is something about the data stored in the DynamoDB table?

@jasonamyers
Copy link
Author

jasonamyers commented Jul 23, 2017 via email

@tmichel
Copy link

tmichel commented Jul 24, 2017

I fixed the nil dereference in 272b0db. But this still means that if the DynamoDB response contains a nil value an empty string will be returned. This solves the panic but probably will not solve this particular issue.

@jasonamyers can you upgrade to the latest version? This is a pretty trivial change so I guess backporting would not be a big problem but I would not want to do that unless absolutely necessary.

As I cannot reproduce the issue this seems to be specific to your environment. To further narrow down the issue you could create a minimal test case that fetches the secret from DynamoDB but does no decryption. Also turning on logging in the aws package might help to uncover more information.

@tmichel
Copy link

tmichel commented Dec 20, 2017

I'm closing this as there is no response and I could not reproduce the issue.

@tmichel tmichel closed this as completed Dec 20, 2017
@twglomski
Copy link

Because this got closed without resolution and we ran into something similar to this, turned out that there was a garbage invisible character somewhere in the particular secret we were retrieving with credstash. We stored a new version of the secret without the garbage invisible character and everything started working. Hopefully the next person to google this will find this and it will help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants