Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

opa test results are nondeterministic with errors – the results change from run to run #1620

Closed
mikol opened this issue Aug 9, 2019 · 0 comments · Fixed by #1641
Closed

opa test results are nondeterministic with errors – the results change from run to run #1620

mikol opened this issue Aug 9, 2019 · 0 comments · Fixed by #1641
Assignees
@tsandall
Labels
bug

Comments

@mikol
Copy link
Contributor

mikol commented Aug 9, 2019 •
edited
Loading

Expected Behavior

opa test ... should produce the same output every time it is run if no modules have changed.

Actual Behavior

opa test ... may produce different results from run to run. For example:

. 2019-08-09 12:22:29
. ~/tmp/join
. opa test .
9 errors occurred:
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:111: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:121: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:69: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:79: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:10: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:11: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
utils/object/test_v1/test_v1.rego:11: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
utils/object/test_v1/test_v1.rego:6: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
utils/object/test_v1/test_v1.rego:16: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
. 2019-08-09 12:22:35
. ~/tmp/join
. opa test .
10 errors occurred:
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:111: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:121: rego_type_error: match error
	left  : string
	right : boolean
utils/object/v1/v1.rego:13: rego_type_error: undefined function data.library.v1.utils.object.v1.get
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:79: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:69: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:10: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:11: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
utils/object/test_v1/test_v1.rego:11: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
utils/object/test_v1/test_v1.rego:16: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
utils/object/test_v1/test_v1.rego:6: rego_type_error: undefined function data.library.v1.utils.object.v1.merge
. 2019-08-09 12:22:39
. ~/tmp/join
. opa test .
4 errors occurred:
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:79: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:111: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:121: rego_type_error: match error
	left  : string
	right : boolean
stacks/resolutions/prioritized/allow_then_deny/v1/v1.rego:69: rego_type_error: match error
	left  : string
	right : boolean
. 2019-08-09 12:22:40
. ~/tmp/join
.

Steps to Reproduce the Problem

  1. Run opa test ... more than once with tests/modules that are expected to produce errors
@tsandall tsandall added the bug label Aug 12, 2019
@tsandall tsandall self-assigned this Aug 12, 2019
tsandall added a commit to tsandall/opa that referenced this issue Aug 15, 2019
@tsandall
ast: Fix ordering of rule type checking errors
2b565c5 
Previously the type checker would not sort errors before returning
them. As a result, the ordering of errors would depend on the order of
rules passed to the type checker (which is non-deterministic due to Go
map iteration order.)

These changes update the type checker to sort errors by location
before before returning them. This way the errors reported by opa test
and opa check and everything else are consistent across runs.

Fixes open-policy-agent#1620

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
@tsandall tsandall mentioned this issue Aug 15, 2019
Merged
tsandall added a commit that referenced this issue Aug 16, 2019
@tsandall
ast: Fix ordering of rule type checking errors
2b77240 
Previously the type checker would not sort errors before returning
them. As a result, the ordering of errors would depend on the order of
rules passed to the type checker (which is non-deterministic due to Go
map iteration order.)

These changes update the type checker to sort errors by location
before before returning them. This way the errors reported by opa test
and opa check and everything else are consistent across runs.

Fixes #1620

Signed-off-by: Torin Sandall <torinsandall@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tsandall tsandall

Labels
bug
Projects
Open Policy Agent
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

ast: Fix ordering of rule type checking errors tsandall/opa
2 participants
@tsandall @mikol