Implement query validation endpoint

[lbschanno]

Implement the query/{logicName}/validate endpoint. This feature supports the ability to configure validation rules that will validate LUCENE and JEXL queries against a number of criteria and provide meaningful feedback to customers.

Closes #2585

Each validation criteria is implemented via a corresponding datawave.query.rules.QueryRule implementation.

Rule implementations:

datawave.query.rules.AmbiguousNotRule

Checks LUCENE queries for usage of NOT without using parens (NOT takes precedence so the resulting plan could be quite incorrect):

• FIELD1:abc OR FIELD2:def NOT FIELD3:123 -> should be (FIELD1:abc OR FIELD2:def) NOT FIELD3:123

Sample bean config:

<bean id="AmbiguousNotRule" scope="prototype" class="datawave.query.rules.AmbiguousNotRule">  
    <property name="name" value="Check for Ambiguous Usage of NOT"/>  
</bean>

Sample message for query FIELD1:abc FIELD2:def NOT FIELD:ghi:

Ambiguous usage of NOT detected with multiple unwrapped preceding terms: "FIELD1:abc AND FIELD2:def NOT" should be "(FIELD1:abc AND FIELD2:def) NOT".

datawave.query.rules.AmbiguousOrPhrasesRule

Checks LUCENE queries for any parens before field or completely missing parens when using ORs:

• FIELD:1234 OR 5678 -> should be FIELD:(1234 OR 5678)
• (FIELD:1234 OR 5678) -> should be FIELD:(1243 OR 5687)

Sample bean config:

<bean id="AmbiguousOrPhrasesRule" scope="prototype" class="datawave.query.rules.AmbiguousOrPhrasesRule">  
    <property name="name" value="Check for Unfielded Terms That Could Be Wrapped"/>  
</bean>

Sample message for query FOO:abc OR def OR efg:

Ambiguous unfielded terms OR'd with fielded term detected: FOO:abc OR def OR efg Recommended: FOO:(abc OR def OR efg)

datawave.query.rules.AmbiguousUnquotedPhrasesRule

Checks LUCENE queries for any phrases not quoted (where we would automatically AND) anywhere that the boolean operator is missing

• FIELD:term1 term2 -> should be FIELD:"term1 term2"

Sample bean config:

<bean id="AmbiguousUnquotedPhrasesRule" scope="prototype" class="datawave.query.rules.AmbiguousUnquotedPhrasesRule">  
    <property name="name" value="Check for Unfielded Terms That Could Be Quoted"/>  
</bean>

Sample message for query FOO:abc def ghi:

Ambiguous unfielded terms AND'd with fielded term detected: FOO:abc AND def AND ghi. Recommended: FOO:"abc def ghi"

datawave.query.rules.FieldExistenceRule

Checks the query for any fields that do not exist in the data dictionary. Special fields may be specified that will not be flagged if they do not exist in the data dictionary.
Sample bean config:

<bean id="FieldExistenceRule" scope="prototype" class="datawave.query.rules.FieldExistenceRule">  
    <property name="name" value="Check Field Existence"/>  
    <property name="specialFields">  
        <list value-type="java.lang.String">   
            <value>_ANYFIELD_</value>  
            <value>_NOFIELD_</value>  
        </list>  
    </property>  
</bean>

Sample message for query FOO:abc AND MISSING1:def AND MISSING2:efg:

Fields not found in data dictionary: MISSING1, MISSING2

datawave.query.rules.FieldPatternPresenceRule

Returns configured messages if we see field X or Y.
Sample bean config:

<bean id="FieldPatternPresenceRule" scope="prototype" class="datawave.query.rules.FieldPatternPresenceRule">  
    <property name="name" value="Check Presence of Field or Pattern"/>  
    <property name="fieldMessages">  
        <map key-type="java.lang.String" value-type="java.lang.String">  
            <entry key="_ANYFIELD_" value="Detected presence of _ANYFIELD_"/>  
        </map>  
    </property>  
    <property name="patternMessages">  
        <map key-type="java.lang.String" value-type="java.lang.String">  
            <entry key=".*" value="Detected pattern '.*' that will match everything"/>  
        </map>  
    </property>  
</bean>

Sample messages for query _ANYFIELD_:abc || FOO:.*

Detected presence of _ANYFIELD_
Detected pattern '.*' that will match everything

datawave.query.rules.IncludeExcludeArgsRule

Checks LUCENE queries for any #INCLUDE or #EXCLUDE functions with an incorrect number of arguments, taking into account whether a boolean was given as the first argument.
Sample bean config:

<bean id="IncludeExcludeArgsRule" scope="prototype" class="datawave.query.rules.IncludeExcludeArgsRule">  
    <property name="name" value="Validate Args of #INCLUDE and #EXCLUDE"/>  
</bean>

Sample message for query: #INCLUDE(OR, 'value')

Function #INCLUDE supplied with uneven number of arguments after the first boolean arg OR. Must supply field/value after the boolean, e.g. #INCLUDE(OR, FIELD, 'value') or #INCLUDE(OR, FIELD1, 'value1',' FIELD2, 'value2').

datawave.query.rules.IncludeExcludeIndexFieldsRule

Checks queries for any usage of #INCLUDE or #EXCLUDE with index fields in them.
Sample bean config:

<bean id="IncludeExcludeIndexFieldsRule" scope="prototype" class="datawave.query.rules.IncludeExcludeIndexFieldsRule">  
    <property name="name" value="Check #INCLUDE and #EXCLUDE for Indexed Fields"/>  
</bean>

Sample message for query: FOO:abc AND #INCLUDE(INDEXED1, 'abc'):

Indexed fields found within the function filter:includeRegex: INDEXED1

datawave.query.rules.InvalidQuoteRule

Checks LUCENE queries for the usage of the quote ` instead of '.
Sample bean config:

<bean id="InvalidQuoteRule" scope="prototype" class="datawave.query.rules.InvalidQuoteRule">  
    <property name="name" value="Check for Invalid Quote"/>  
</bean>

Sample message for query FOO:'abc' AND BAR:`def` :

Invalid quote ` used in phrase "BAR:`abc`". Use ' instead.

datawave.query.rules.MinimumSlopProximityRule

Checks queries for any cases where the proximity is smaller than the number of terms.

• FIELD:"term1 term2 term3"~1 -> the 1 should be 3 or greater.

Sample bean config:

<bean id="MinimumSlopProximityRule" scope="prototype" class="datawave.query.rules.MinimumSlopProximityRule">  
    <property name="name" value="Validate Slop Proximity"/>  
</bean>

Sample message for query FOO:"term1 term2 term3"~2:

Invalid slop proximity, the 2 should be 3 or greater: FOO:"term1 term2 term3"~2

datawave.query.rules.NumericValueRule

Checks queries for any cases where a numeric value is used against a non-numeric field.
Sample bean config:

<bean id="NumericValueRule" scope="prototype" class="datawave.query.rules.NumericValueRule">  
    <property name="name" value="Validate Numeric Values Only Given for Numeric Fields"/>  
</bean>

Sample message for query AGE:1 AND NON_NUMERIC_FIELD:4

Numeric values supplied for non-numeric field(s): NON_NUMERIC_FIELD

datawave.query.rules.TimeFunctionRule

Checks queries for any usage of #TIME_FUNCTION with any non-date fields.
Sample bean config:

<bean id="TimeFunctionRule" scope="prototype" class="datawave.query.rules.TimeFunctionRule">  
    <property name="name" value="Validate #TIME_FUNCTION has Date Fields"/>  
</bean>

Sample message for query filter:timeFunction(DATE1,NON_DATE2,'-','>',2522880000000L):

Function #TIME_FUNCTION (filter:timeFunction) found with fields that are not date types: NON_DATE2

datawave.query.rules.UnescapedSpecialCharsRule

Checks queries for fields and patterns with unescaped special characters. Characters that should be considered exceptions to the rule may be configured for literals and patterns. The regex-reserved characters . + * ? ^ $ ( ) [ ] { } | \ will always be treated as exceptions for patterns. Whether whitespace characters should be considered special characters is also configurable.
Sample bean config:

<bean id="UnescapedSpecialCharsRule" scope="prototype" class="datawave.query.rules.UnescapedSpecialCharsRule">  
    <property name="name" value="Check for Unescaped Special Characters"/>  
    <property name="literalExceptions">  
        <list value-type="java.lang.Character">  
            <value>?</value>  
        </list>  
    </property>  
    <property name="patternExceptions">  
        <list value-type="java.lang.Character">  
            <value>_</value>  
        </list>  
    </property>  
    <property name="escapedWhitespaceRequiredForLiterals" value="false"/>  
    <property name="escapedWhitespaceRequiredForPatterns" value="false"/>  
</bean>

Sample message for query FOO == 'ab?c' || FOO =~ 'ab_cd':

Literal string "ab?c" has the following unescaped special character(s): '?'");  
Regex pattern "ab_cd" has the following unescaped special character(s): '_'"

datawave.query.rules.UnescapedWildcardsInPhrasesRule

Checks LUCENE queries for any quoted phrases that have an unescaped wildcard in them.
Sample bean config:

<bean id="UnescapedWildcardsInPhrasesRule" scope="prototype" class="datawave.query.rules.UnescapedWildcardsInPhrasesRule">  
    <property name="name" value="Check Quoted Phrases for Unescaped Wildcard"/>  
</bean>

Sample message for FOO:"ab*":

Unescaped wildcard found in phrase FOO:"ab*". Wildcard is incorrect, or phrase should be FOO:/ab*/

datawave.query.rules.UnfieldedTermsRule

Checks LUCENE queries for any unfielded terms.
Sample bean config:

<bean id="UnfieldedTermsRule" scope="prototype" class="datawave.query.rules.UnfieldedTermsRule">  
    <property name="name" value="Check for Unfielded Terms"/>  
</bean>

Sample message for FOO:abc def "efg"

Unfielded term def found.
Unfielded term "efg" found.

Activating Rules

Rules must be configured in the QueryLogicFactory.xml and added to the list of rules configured for the validationRules property of a ShardQueryLogic.
Sample bean config:

	<!-- Rules added to validationRules via referencing bean definition samples. -->
    <property name="docAggregationThresholdMs" value="-1" />  
    <property name="tfAggregationThresholdMs" value="-1" />  
    <property name="validationRules">  
        <list value-type="datawave.query.rules.QueryRule">  
            <ref bean="InvalidQuoteRule"/>  
            <ref bean="UnfieldedTermsRule"/>  
            <ref bean="UnescapedWildcardsInPhrasesRule"/>  
            <ref bean="AmbiguousNotRule"/>  
            <ref bean="AmbiguousOrPhrasesRule"/>  
            <ref bean="AmbiguousUnquotedPhrasesRule"/>  
            <ref bean="MinimumSlopProximityRule"/>  
            <ref bean="IncludeExcludeArgsRule"/>  
            <ref bean="FieldExistenceRule"/>  
            <ref bean="UnescapedSpecialCharsRule"/>  
            <ref bean="FieldPatternPresenceRule"/>  
            <ref bean="IncludeExcludeIndexFieldsRule"/>  
            <ref bean="NumericValueRule"/>  
            <ref bean="TimeFunctionRule"/>  
        </list>  
    </property>  
</bean>

Some notes on rules:

• All rules may be configured with a name. Names are not required to be unique. If a name is not specified, the name will be the name of the rule class.
• Rules will be executed in the order that they are listed in the validationRules property.
• Rules will never be executed twice for a validation operation.
• Some rules will only execute against LUCENE queries. The rest will execute on the resulting parsed JEXL query. LUCENE-only rules:

AmbiguousNotRule
AmbiguousOrRule
AmbiguousUnquotedPhrasesRule
IncludeExcludeArgsRule
InvalidQuoteRule
MinimumSlopProximityRule
UnescapedWildcardsInPhrasesRule
UnfieldedTermsRule

REST Response Schema

The REST response schema will be as follows:

{
	// A list of names of rules that successfully executed 
	// against the query, in the order that they executed.
	"ExecutedRules": [
		"<RuleName>",
		// ...
	],
	// Whether any issues were flagged.
	"HasResults": true,
	"LogicName": "<logicName>",
	"OperationTimeMS": 10,
	"QueryId": "<uuid>",
	// The flagged issues in the order that they were flagged.
	"Results": [
		{
			"RuleName":"<RuleName>",
			"Messages":[
				"A message about a flagged issue",
				// ...
			],
			// This is populated only if an exception occurred 
			// while a rule was attempting to validate a query.
			"Exception": {
				"Cause": "",
				"Message":""
			}
		},
		// ...
	],
	"Exceptions": [
		{
			"Cause":"",
			"Code":"",
			"Message":""
		},
		// ...
	]
}

[ivakegg]

I am not seeing this class being used anywhere, what am I missing. Also what functionality does this provide that the LuceneQueryStringBuildingVisitor does not already supply?

[lbschanno]

This class is analogous to the PrintingVisitor for JEXL queries. It prints the overall structure of LUCENE query trees so that we can the hierarchy of nodes and their details. We used it extensively when examining trees while writing tests. It is currently now used in at least one place to log the LUCENE query after the initial parse in ShardQueryLogic.validateQuery().

The LuceneQueryStringBuildingVisitor is analogous to JexlStringBuildingVisitor. It returns a LUCENE query tree as a query string, but does not show the actual structure of the tree.

[ivakegg]

A few comments to handle, but all in all this looks pretty good! Created #2656 with some of the suggested changes.