SOLR-17567: Improve Stream CLI implementation

[epugh]

https://issues.apache.org/jira/browse/SOLR-17567

This is to improve StreamTool based on some assessments by @malliaridis!

• Use of raw strings in Option#getOptionValue()
• Overlapping options (-f and -e)
• Multiple type suppressions that can be avoided
• A bug that causes specific bats tests to fail. done in seperate PR by @malliaridis !
• StreamTool.LocalCatStream overrides CatStream btu does not properly set commaDelimitedFilepaths (this may be a bug)
• Obsolete LetStream#getLetParams() was introduced
• Function StreamTool.constructStream seems obsolete and occurrences can be replaced with implementation
• Implementation of StreamTool.readExpression does not take leading spaces infront of comment lines into account
• Implementation of StreamTool.readExpression does not support */ at the end of line

[epugh]

@malliaridis I've marked off the items I took care of. The remaining ones I need more detail to more forward on.

Regardless, it's nice to get a bit more polish!

[epugh]

I've also polished up the docs a bit.

[epugh]

Finally! All the junit tests related to StreamTool are passing!

[malliaridis]

Wow, did not have SolrOnHdfsSnapshotsTool under my radar. Did you find it throught the forbidden-api rules?

Additionally to the forbidden-api rules I have commented below, I would probably add one small "feature" to complete the comments "parsing", that would be trailing line comments like in

search(
collection,
q=*:* # this is a trailing comment
)

// or like in 

/*search(
collection,
q=*:*
)*/

It is hard to know all edge-cases in advance when it comes to comment stripping.

And thanks for responding so quick to the update requests. :D

[malliaridis]

I would extend the following as well (from CommandLine):

org.apache.commons.cli.CommandLine#hasOption(char)
org.apache.commons.cli.CommandLine#getOptionValue(char)
org.apache.commons.cli.CommandLine#getOptionValue(char, java.lang.String)
org.apache.commons.cli.CommandLine#getOptionValue(char, Supplier<String>)
org.apache.commons.cli.CommandLine#getOptionValues(char)
org.apache.commons.cli.CommandLine#getOptionValues(java.lang.String)
org.apache.commons.cli.CommandLine#getParsedOptionValue(char)
org.apache.commons.cli.CommandLine#getParsedOptionValue(char, Supplier<T>)
org.apache.commons.cli.CommandLine#getParsedOptionValue(char, java.lang.Object)
org.apache.commons.cli.CommandLine#getParsedOptionValue(java.lang.String)
org.apache.commons.cli.CommandLine#getParsedOptionValue(String, Supplier<T>)

I haven't written any rules before, so I am not sure if the above have the correct format (especially Supplier<T> is likely not correct).

[epugh]

Wow, did not have SolrOnHdfsSnapshotsTool under my radar. Did you find it throught the forbidden-api rules?

I did!!!

[malliaridis]

About the remaining TODOs:

Multiple type suppressions that can be avoided

Suppressions like @SuppressWarnings({"rawtypes"}) for classes like List can usually be fixed by simply using List<?> or using the known generic type if any present.

StreamTool.LocalCatStream overrides CatStream btu does not properly set commaDelimitedFilepaths (this may be a bug)

In CatStream commaDelimitedFilepaths is a class property that is also used in CatStream#toExpression, which is called by other classes. By passing passing the property as parameter in validateAndSetFilepathsInSandbox and overriding the method does not update the class property, resulting to potential inconsistent behavior / values in other classes.

Instead, it is possible to simply remove the property from the parameters of validateAndSetFilepathsInSandbox (like it was before), and make the property commaDelimitedFilepaths visible (protected modifier). That way, overriding validateAndSetFilepathsInSandbox will be possible without changing the method signature.

If the description is a bit irritating, I can create a separate PR for that. :)

[epugh]

Okay, this I think is almost ready to merge!

[epugh]

Since this is a straight up follow on from the introduction of the StreamTool in main I am not adding CHANGES.txt entry.