Fix parsing of environment variables to prevent object parsing. #132 #139

blakerouse · 2020-01-16T15:21:56Z

Currently internal/parse handles the values from the environment variable just like values from the configuration file. This PR changes the parser to limit environment variable parsing to not parse object notation.

This is a breaking change because if any environment relies on this behavior it would break them. I question if this is correct way to solve the bug.

Really the bug could be solved by ensuring that the environment variable is single quoted:

export ENV_VAR="'{user}'"

I would be okay with closing this PR and closing the bug with this comment.

urso

Personally for environment variables I would not expect them to support the object syntax. But I'm not sure if we are facing a bug that has turned into a feature :)

For arrays we have to support top-level arrays. There are definitely users configuring multiple hosts by separating them via commas.

Really the bug could be solved by ensuring that the environment variable is single quoted:

Good to know we have a workaround.

@ph WDYT?

urso · 2020-01-16T16:07:15Z

internal/parse/parse.go

+//
+// The parser implements a superset of JSON, but only a subset of YAML by
+// allowing for arrays and objects having a trailing comma. In addition 3
+// strings types are supported:


s/strings/string

urso · 2020-01-16T16:09:01Z

internal/parse/parse.go

+//
+// 1. single quoted string (no unescaping of any characters)
+// 2. double quoted strings (characters are escaped)
+// 3. strings without quotes. String parsing stops in


but shouldn't it be stops at or stops at (non-native english speaker here :) )

urso · 2020-01-16T16:17:12Z

internal/parse/parse.go

@@ -99,13 +142,25 @@ func (p *flagParser) parseValue(stopSet string) (interface{}, error) {

 	switch in[0] {
 	case '[':
-		return p.parseArray()
+		if p.cfg.Array {


even if p.cfg.Array is false we might still parse top-level arrays if users have an environment variable like 1,2,3.

For the Env case p.cfg.Array is still true which allows the flowing to be in the environment variables:

{`[]`, nil}, { `a,b,c`, []interface{}{"a", "b", "c"}, }, { `C:\Windows\Path1,C:\Windows\Path2`, []interface{}{ `C:\Windows\Path1`, `C:\Windows\Path2`, }, }, { `[array, 1, true, "abc"]`, []interface{}{"array", uint64(1), true, "abc"}, }, { `[test, [1,2,3], on]`, []interface{}{ "test", []interface{}{uint64(1), uint64(2), uint64(3)}, true, }, }, { `[host1:1234, host2:1234]`, []interface{}{ "host1:1234", "host2:1234", }, },

urso · 2020-01-16T16:18:51Z

internal/parse/parse.go

+		if p.cfg.Array {
+			return p.parseArray()
+		}
+		return p.parsePrimitive(stopSet)


Should we adapt the stop set based on the config? e.g. if Array and Object are false and the string does not start with quotes we could return the original string as is.

In the case that Array and Object are false the stopSet will just be "," (the topLevelStopSet) and it will not change unless the parser hits a double or single quoted string (when enabled).

Being that we want top level array parsing to still be allowed when p.cfg.Array is false, I believe this path is correct. I am going to add more tests for all the combinations of configs.

Oh, I see. Top-level array parsing can not be disabled and is done independent of the cfg.Array setting. My impression was that Array parsing also disables the top-level array support.

I think that makes sense. Was just not what I expected. Maybe we should document that top-level array splitting is active always.

urso · 2020-01-16T16:27:27Z

types.go

@@ -533,7 +533,7 @@ func (r *refDynValue) getValue(
 		}
 		return nil, err
 	}
-	return parseValue(p, opts, str)
+	return parseValue(p, opts, str, parse.EnvParserConfig)


this will affect parsing for every single resolver. go-ucfg has 3 resolver options by default: ResolveEnv, ResolveNOOP, and Resolve. The last one accepts a custom function.

I think it makes sense to have the resolver also configure the parsing mode.

There is also a Keystore resolver in libbeat.

ph · 2020-01-16T18:59:27Z

@urso @blakerouse Even if this is breaking change I doubt it is really used in the wild (the parsing). But as you have said, this might be a bug turned into a feature. Even I believe this behavior leads to confusion and it is a usability problem.

ph · 2020-01-16T19:00:59Z

Also, I don't think the object reference used in the variables is a well-understood behavior, I wish we had stats to understand how people are actually using it.

blakerouse · 2020-01-17T14:39:39Z

@urso this is ready for another review. It became a much bigger change with adding the config to the resolvers but I like the result.

codecov-io · 2020-01-17T15:08:42Z

Codecov Report

Merging #139 into master will increase coverage by 0.14%.
The diff coverage is 92.1%.

@@            Coverage Diff             @@
##           master     #139      +/-   ##
==========================================
+ Coverage    77.8%   77.94%   +0.14%     
==========================================
  Files          23       23              
  Lines        2595     2612      +17     
==========================================
+ Hits         2019     2036      +17     
  Misses        425      425              
  Partials      151      151

Impacted Files	Coverage Δ
flag/value.go	`68.18% <ø> (ø)`	⬆️
types.go	`74.23% <100%> (ø)`	⬆️
parse/parse.go	`95.74% <100%> (ø)`
variables.go	`80.28% <100%> (+0.07%)`	⬆️
opts.go	`64.7% <50%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5059399...30da7a8. Read the comment docs.

…ic#132

…more tests.

…ust Config, so import is parse.Config. Add parse.Config to return of resolvers.

urso · 2020-01-17T15:52:06Z

Although it is a breaking change, I like it. I think we should go forward with it. In case users have problems in Beats, we can still point them towards the -E setting. The -E setting still accepts the object syntax.

Please add changelog entries. Besides fixing #132, the change breaks API + removes object syntax for env variables.

blakerouse · 2020-01-17T17:02:41Z

@urso Added changelog entries.

ph

LGTM, Lets also make it obvious in the beats changelog when this change goes in 7.7

blakerouse added the review label Jan 16, 2020

blakerouse requested a review from urso January 16, 2020 15:21

blakerouse added the [zube]: In Review label Jan 16, 2020

urso requested a review from ph January 16, 2020 16:04

urso reviewed Jan 16, 2020

View reviewed changes

blakerouse requested a review from urso January 17, 2020 14:39

blakerouse added 4 commits January 17, 2020 10:23

Fix parsing of environment variables to prevent object parsing. elast…

d48f299

…ic#132

Add validation of config to prevent Object true and Array false. Add …

d0afc42

…more tests.

Move parse from interal to importable module. Rename ParseConfig to j…

5007476

…ust Config, so import is parse.Config. Add parse.Config to return of resolvers.

Add test for parse config validation.

fdebcfd

blakerouse force-pushed the fix-env-parsing branch from 245ea61 to fdebcfd Compare January 17, 2020 15:23

Add changelog entry.

30da7a8

urso approved these changes Jan 17, 2020

View reviewed changes

ph approved these changes Jan 17, 2020

View reviewed changes

blakerouse merged commit f677390 into elastic:master Jan 17, 2020

blakerouse deleted the fix-env-parsing branch January 17, 2020 18:08

zube bot added [zube]: Done and removed [zube]: In Review labels Jan 17, 2020

blakerouse mentioned this pull request Jan 17, 2020

Environment variable expansion in string context triggers JSON parser #132

Closed

andresrc removed the [zube]: Done label Jan 20, 2020

blakerouse mentioned this pull request Jan 31, 2020

Upgrade go-ucfg to v0.8.1 release elastic/beats#15937

Merged

6 tasks

blakerouse mentioned this pull request Feb 11, 2020

Cherry-pick #15937 to 7.x: Upgrade go-ucfg to v0.8.1 release elastic/beats#16259

Merged

6 tasks

urso mentioned this pull request Apr 15, 2020

[docs] Make editorial changes to the relnotes elastic/beats#17718

Merged

orhantoy mentioned this pull request Jun 18, 2020

Is there an alternative to env. variables object substitution? elastic/beats#19272

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix parsing of environment variables to prevent object parsing. #132 #139

Fix parsing of environment variables to prevent object parsing. #132 #139

blakerouse commented Jan 16, 2020

urso left a comment

urso Jan 16, 2020

urso Jan 16, 2020

urso Jan 16, 2020

blakerouse Jan 17, 2020

urso Jan 16, 2020

blakerouse Jan 17, 2020

urso Jan 17, 2020

urso Jan 16, 2020

ph Jan 16, 2020

ph commented Jan 16, 2020

ph commented Jan 16, 2020

blakerouse commented Jan 17, 2020

codecov-io commented Jan 17, 2020 •

edited

Loading

urso commented Jan 17, 2020

blakerouse commented Jan 17, 2020

ph left a comment

Fix parsing of environment variables to prevent object parsing. #132 #139

Fix parsing of environment variables to prevent object parsing. #132 #139

Conversation

blakerouse commented Jan 16, 2020

urso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ph commented Jan 16, 2020

ph commented Jan 16, 2020

blakerouse commented Jan 17, 2020

codecov-io commented Jan 17, 2020 • edited Loading

Codecov Report

urso commented Jan 17, 2020

blakerouse commented Jan 17, 2020

ph left a comment

Choose a reason for hiding this comment

codecov-io commented Jan 17, 2020 •

edited

Loading