Added json unmarshal for potentially encoded JSON string values #95

sgettys · 2022-09-09T20:40:29Z

Signed-off-by: Steven Gettys s.gettys@f5.com

Closes #94
Terraform outputs from porter included escaped html characters for things like connection strings. This is a regression due to switching from -raw to -json. The reason the switch occurred is because -raw only supports number, boolean, and string outputs from terraform.

The mixin will decode all of the json strings and remove the html escapes. This works for complex nested objects like arrays and maps as well as strings

Signed-off-by: Steven Gettys <s.gettys@f5.com>

sgettys · 2022-09-09T20:40:45Z

Need to add a couple more json encoding scenarios to the tests to validate

Signed-off-by: Steven Gettys <s.gettys@f5.com>

VinozzZ · 2022-09-13T18:43:12Z

Hey @sgettys , I added below test in the test-cli.sh and the test failed. It looks like we need to always parse the output from terraform -json through a json parser. Do you mind to update the code to handle it?

${PORTER_HOME}/porter install --verbosity=debug --param file_contents='foo!' --param map_var='{"foo": "bar"}' --param array_var='["mylist", "https://ml.azure.com/?wsid=/subscriptions/zzzz/resourceGroups/rg-scctre-ws-935d/providers/Microsoft.MachineLearningServices/workspaces/ml-scctre-ws-935d-svc-0cdf&tid=zzzzz"]' --param boolean_var=true --param number_var=1 --force

echo "Verifying installation output after install"
verify-output "file_contents" 'foo!'
verify-output "map_var" '{"foo":"bar"}'
verify-output "array_var" '["mylist", "https://ml.azure.com/?wsid=/subscriptions/zzzz/resourceGroups/rg-scctre-ws-935d/providers/Microsoft.MachineLearningServices/workspaces/ml-scctre-ws-935d-svc-0cdf&tid=zzzzz"]'

In addition, we would like to have all PRs to link to the issue that's trying to fix and have an explanation of the change. If the issue is not clear, it's best to summarize the issue in the PR description so that reviewers can easily catch up

VinozzZ · 2022-09-13T18:48:11Z

build/testdata/bundles/terraform/terraform/outputs.tf

@@ -18,3 +18,7 @@ output "boolean_var" {
 output "number_var" {
  value = var.number_var
 }
+
+output "json_encode_html" {
+  value = "hello&world?@test#$><"


Can we follow the existing pattern for the other test cases, defining the value as a variable. Also it seems like the variables value are not currently used in the test-cli.sh. It feels confusing that's not used. If we are going to define it, I think we should use it in our tests.

sgettys · 2022-09-13T19:13:19Z

Hey @sgettys , I added below test in the test-cli.sh and the test failed. It looks like we need to always parse the output from terraform -json through a json parser. Do you mind to update the code to handle it?
${PORTER_HOME}/porter install --verbosity=debug --param file_contents='foo!' --param map_var='{"foo": "bar"}' --param array_var='["mylist", "https://ml.azure.com/?wsid=/subscriptions/zzzz/resourceGroups/rg-scctre-ws-935d/providers/Microsoft.MachineLearningServices/workspaces/ml-scctre-ws-935d-svc-0cdf&tid=zzzzz"]' --param boolean_var=true --param number_var=1 --force

echo "Verifying installation output after install"
verify-output "file_contents" 'foo!'
verify-output "map_var" '{"foo":"bar"}'
verify-output "array_var" '["mylist", "https://ml.azure.com/?wsid=/subscriptions/zzzz/resourceGroups/rg-scctre-ws-935d/providers/Microsoft.MachineLearningServices/workspaces/ml-scctre-ws-935d-svc-0cdf&tid=zzzzz"]'
In addition, we would like to have all PRs to link to the issue that's trying to fix and have an explanation of the change. If the issue is not clear, it's best to summarize the issue in the PR description so that reviewers can easily catch up

Great test case, thanks for finding that! I'll address that test case and update the PR, thanks!

sgettys · 2022-09-13T20:04:04Z

Hey @sgettys , I added below test in the test-cli.sh and the test failed. It looks like we need to always parse the output from terraform -json through a json parser. Do you mind to update the code to handle it?
${PORTER_HOME}/porter install --verbosity=debug --param file_contents='foo!' --param map_var='{"foo": "bar"}' --param array_var='["mylist", "https://ml.azure.com/?wsid=/subscriptions/zzzz/resourceGroups/rg-scctre-ws-935d/providers/Microsoft.MachineLearningServices/workspaces/ml-scctre-ws-935d-svc-0cdf&tid=zzzzz"]' --param boolean_var=true --param number_var=1 --force

echo "Verifying installation output after install"
verify-output "file_contents" 'foo!'
verify-output "map_var" '{"foo":"bar"}'
verify-output "array_var" '["mylist", "https://ml.azure.com/?wsid=/subscriptions/zzzz/resourceGroups/rg-scctre-ws-935d/providers/Microsoft.MachineLearningServices/workspaces/ml-scctre-ws-935d-svc-0cdf&tid=zzzzz"]'
In addition, we would like to have all PRs to link to the issue that's trying to fix and have an explanation of the change. If the issue is not clear, it's best to summarize the issue in the PR description so that reviewers can easily catch up

Might need to think about how to solve more complex nested json encoded values. Maybe should define some behaviors for this. Theoretically we could have n-nested json objects as outputs from terraform that could potentially have encoded JSON at any level. Let me dig into this a little more, might have to do a custom unmarshaler to solve this.

…ight have escaped html Signed-off-by: Steven Gettys <s.gettys@f5.com>

sgettys · 2022-09-13T21:47:16Z

Closes #94

Signed-off-by: Steven Gettys <s.gettys@f5.com>

sgettys · 2022-09-13T21:53:50Z

Reworked to support decoding nested values by unmarshaling into empty interface and then re-encoding with a custom json encoder that skips the html encoding.

VinozzZ

Thanks for the quick update. I left some questions on the PR but overall it looks good to me

build/testdata/bundles/terraform/terraform/variables.tf

VinozzZ · 2022-09-13T23:36:32Z

pkg/terraform/terraform.go

+	// string so that the outer quotes are stripped, otherwise return the raw
+	// json syntax directly.
+	var outString string
+	if err = json.Unmarshal(outJSON, &outString); err == nil {


Do we still need to unmarshal the entire output json in order to remove the quotes here?

I could also do type assertion and skip then re-encode for string types then we wouldn't need this second unmarshal

Signed-off-by: Steven Gettys <s.gettys@f5.com>

VinozzZ · 2022-09-14T16:33:02Z

pkg/terraform/terraform.go

-	}
-	// For all other JSON types we'll return the raw JSON syntax directly.
-	return outJSON, nil
+	return bytes.TrimRight(buffer.Bytes(), "\n"), nil


Does this mean we don't need to trim the new line character when the output is a string?

No since we're not re-encoding it. The json.Encode is what puts the newline on the end:
From the docs - Encode writes the JSON encoding of v to the stream, followed by a newline character.

Based on this comment, it looks like the original code is for removing it from Terraform output?
https://github.com/getporter/terraform-mixin/pull/95/files#diff-282757e263fe02ec96e35fd1120b9263db1ca315d75cea25d3ccf68239bd7472L59

I did some testing. It looks like json.Unmarshal() removes the new line character. We should be good. Thanks for answering my questions :)

Yep that's what I was in the middle of validating! Beat me to it :)

Added json unmarshal for potentially encoded JSON string values

21da72a

Signed-off-by: Steven Gettys <s.gettys@f5.com>

getporterbot added this to In Progress in Porter and Mixins Sep 9, 2022

added more special characters to test the unmarshaling of json string

240e5ae

Signed-off-by: Steven Gettys <s.gettys@f5.com>

sgettys changed the title ~~WIP: Added json unmarshal for potentially encoded JSON string values~~ Added json unmarshal for potentially encoded JSON string values Sep 12, 2022

sgettys requested review from bdegeeter, carolynvs and VinozzZ September 12, 2022 15:56

carolynvs assigned carolynvs and VinozzZ Sep 13, 2022

VinozzZ reviewed Sep 13, 2022

View reviewed changes

Implemented recursing json encoder for handling nested objects that m…

dab8e64

…ight have escaped html Signed-off-by: Steven Gettys <s.gettys@f5.com>

sgettys linked an issue Sep 13, 2022 that may be closed by this pull request

Ampersands in Terraform Outputs are being Encoded #94

Closed

added json encoded html string var back

3f91163

Signed-off-by: Steven Gettys <s.gettys@f5.com>

sgettys requested a review from VinozzZ September 13, 2022 21:52

VinozzZ reviewed Sep 13, 2022

View reviewed changes

Addressed review comments

c6bc6cf

Signed-off-by: Steven Gettys <s.gettys@f5.com>

sgettys requested a review from VinozzZ September 14, 2022 16:22

VinozzZ reviewed Sep 14, 2022

View reviewed changes

VinozzZ approved these changes Sep 14, 2022

View reviewed changes

VinozzZ merged commit 6199f64 into getporter:main Sep 14, 2022

Porter and Mixins automation moved this from In Progress to Done Sep 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added json unmarshal for potentially encoded JSON string values #95

Added json unmarshal for potentially encoded JSON string values #95

sgettys commented Sep 9, 2022 •

edited

Loading

sgettys commented Sep 9, 2022

VinozzZ commented Sep 13, 2022 •

edited

Loading

VinozzZ Sep 13, 2022

sgettys commented Sep 13, 2022

sgettys commented Sep 13, 2022

sgettys commented Sep 13, 2022

sgettys commented Sep 13, 2022

VinozzZ left a comment

VinozzZ Sep 13, 2022

sgettys Sep 14, 2022

VinozzZ Sep 14, 2022

sgettys Sep 14, 2022

VinozzZ Sep 14, 2022

VinozzZ Sep 14, 2022

sgettys Sep 14, 2022

Added json unmarshal for potentially encoded JSON string values #95

Added json unmarshal for potentially encoded JSON string values #95

Conversation

sgettys commented Sep 9, 2022 • edited Loading

sgettys commented Sep 9, 2022

VinozzZ commented Sep 13, 2022 • edited Loading

Choose a reason for hiding this comment

sgettys commented Sep 13, 2022

sgettys commented Sep 13, 2022

sgettys commented Sep 13, 2022

sgettys commented Sep 13, 2022

VinozzZ left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgettys commented Sep 9, 2022 •

edited

Loading

VinozzZ commented Sep 13, 2022 •

edited

Loading