-
Notifications
You must be signed in to change notification settings - Fork 9.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
implements gzip interpolation func #3858
Conversation
In case you feel it can be merged, here it is the PR. You can take a look to these use cases: For etcd2: Ignore the old-fashioned |
Interesting... I think this would be the first time we explicitly encouraged having raw binary data interpolated in Terraform. Technically it has always been possible to read binary data using If it is always used inside |
I cannot figure out any use-case in which this can be used without base64ing or encoding in whatever form. At least I have not found in the position. Thanks for the quick response. |
@apparentlymart, I found that sooner or later #3846 & #3909 are going to be merged. If this is the direction Terraform is going to take regarding to local resource management, I can imagine another use case in which this would be useful. Imagine you want to save files in a storage service (s3, gcs or whatever) and you want to make them gzipped. Something like: current:
gzipped:
WDYT? |
@glerchundi I definitely agree that this is useful... I'd love to be able to upload gzipped stuff to S3 as you suggested! My concern is specifically with how well Terraform's internals will deal with having raw binary data in property values: Go strings are UTF-8 by convention but in practice can be any bytes, but JSON can't serialize raw bytes and so the This will probably work fine for attributes that have a I think it would be good to test some edge cases here to make sure that Terraform won't silently corrupt binary data. We could test some of this by just adding a couple extra unit tests that return raw binary data without base64-encoding it, but I'm not sure how best to test what happens when binary data gets written out into the JSON state file... that probably requires something like an acceptance test using the |
@apparentlymart thanks for this detailed explanation, I really appreciate it.
This seems reasonable. I think it would be possible to diverge paths depending on how do you what to save this in the state based on the value type. But anyway this is your concern and your internal&core headaches ;P.
Yep, absolutly needed! If I find a free slot I can work on this. And again, thanks for pointing me out with every detail it was really interesting. |
+1. I really need something like this. |
+1 This would be extremely helpful. |
@apparentlymart rebased this, any chance to make this merged into the master branch? It seems like it's not just me ;) If not, please tell us something so that we can start thinking in other approaches. Thanks. |
+1 I really need this right now :) |
+1 |
Running in to the same issues with large cloud inits. Coreos doesn't support multipart, but it does support base64+gzip content of files. This would be a big win. |
Hello, I also need it for coreos ! |
I also really need this to use terraform with coreos and CloudFormation. This PR looks good to be merged. If you need help with anything related to adding this feature, please let me know. |
thanks for looking into this, much needed feature! |
Any updates on this? This would be really helpful when using some rather large user data scripts (I just hit this problem). @glerchundi would you mind rebasing the branch with latest master so the conflict is resolved? |
@glerchundi does it make sense for you too? |
I've been using this patch in a custom built Terraform binary and haven't seen this corrupt gzip data issue, this is worrying, thanks for bringing this up. My use case does not allow the base64 wrapping though, since CoreOS cloud init implementation does not support it (not sure if this changed recently). For reference, this is the way I've been using it in my resources:
|
If passing base64 directly to |
Coreos support gzip+base64 encoded
But we have so much certificates so we had to pass them as gzip since we exceeded 16K limit. If |
@apparentlymart, @glerchundi I have rebased and added necessary implementations at here which is mostly derived from @glerchundi's work. But expected result at my test is different than @glerchundi. He has used base64 as well but I am not sure why the result is different. Would you review this please guys? |
I think it is related to Go version the time first PR is submitted. Since there are changes at 1.8 related gzip the output might change. |
@cemo definitely, it's related to gzip, specifically in DEFLATE implementation changes. We already faced this issue in other projects, same input different results depending on the runtime you were using: https://golang.org/doc/go1.8#compress_flate @apparentlymart Ok, seems reasonable. That being said, and knowing there were precedents in place ( UPDATE: @cemo i missed you already pointed me to the golang reference... |
4aebc83
to
f273344
Compare
Just to put some info on this thread, So it seems this PR applies. |
@apparentlymart I think that you can merge this issue since all your concerns are addressed. This PR also closes #3407. |
Hi, This seems to be only one way, right? No way to get back the original using any other function? I happened to stumble upon this while searching for my requirement: Thanks and Regards, |
@cemo where's the PR with your implementation? seems like we could close this PR if yours gets merged. |
@josephholsten I had implemented but did not create a PR since @glerchundi was the original author. He also updated to address @apparentlymart's concerns. Now it is safe to merge this PR. |
Hi everyone! Sorry for the silence here. This is on my list to review soon, just getting delayed by some legwork going on for the 0.10 release. I will look at this as soon as I can. |
This would be incredibly useful. It would be great if we could simply do
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems straightforward, needs user doc
I rebased this, added some docs, and merged it as a303817. Thanks for working on this, @glerchundi, and for your patience as we worked through the design constraints. Sorry this one took so long! At present this function will likely be of limited use due to the need for other resources to be updated to expect base64 data as an option, but this is a good first step. We'd be happy to review PRs for adding base64 variants of attributes that are often used with the |
Don't worry @apparentlymart and thanks this is going to be a huge win for us. |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
fixes #3407
Signed-off-by: Gorka Lerchundi Osa glertxundi@gmail.com