Add simple rule indexing implementation #319
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously the query compiler checked whether input was defined to catch two cases: 1) Input was required by query or transitive dependencies of query. Input was required if the input document was referenced at all. Motivation for this was to produce "correct" results when negation is used. In practice, this never proved to be very helpful. 2) Input document was specified multiple times, causing a conflict. Again, in practice, this never proved to be very helpful. In both cases, if the policy decision is *incorrect* someone has to look at (i) the input (ii) the data and (iii) the policy to understand why. Ultimately we expect users to push schema information into OPA so that we can validate inputs and data conform to those schema. In that case, if an input was not specified (or conflicting); the type checking should catch it.
These changes add basic indexing support to the compiler. Rules can be indexed based on their contents. In this case, the first indexing strategy we support is based on expressions of the form: eq(term, base_doc_ref) where the term doesn't require evaluation (except vars). In these cases, we construct a trie that allows us to efficiently search for rules that must be evaluated (rules that do not have to be evaluated are excluded by the search.) As part of these changes the virtual doc benchmark has been updated to include rules that are hit and missed by the rule indexer.
tsandall
force-pushed
the
http-api-optimization
branch
from
e305736 to
fec318b
Compare
timothyhinrichs
approved these changes
May 9, 2017
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These changes add a simple rule indexing implementation to OPA. In the future, we can add more sophisticated strategies and perform analysis on queries to determine which indices ought to be used.
The first indexing strategy supports expressions of form eq(ref, term) (or vice versa) where ref is a ground, non-nested reference to a base document (input or data) and term is scalar, non-nested array, or var. The implementation builds a trie from the expressions in rule sets that match this pattern. The trie shrinks the search space required at evaluation time.