Turning off parquet logging in SparkFunSuite tests

[tdanford]

(I'm going to experiment with opening this as an issue, so that it's on our collective minds and maybe someone will remember how to track it down?)

If you're using the SparkFunSuite for Spark-based tests, with sparkTest, sparkBefore, and sparkAfter, there's a silenceSpark option that allows you to turn off Spark logging. However, if you're using adamSave as part of your spark test, even silenceSpark=true won't silence the parquet.hadoop INFO level logging coming out of (I presume) Parquet.

I tried adding "parquet" and "parquet.hadoop" to the hard-coded list of silenced packages in the SparkLogUtils, and that didn't work for me either.

If someone can figure out where the Parquet logging is enabled (my guess: a log4j.xml embedded in the Parquet dependency?) and whether it could be modified at runtime, that would take us a long way towards figuring out how to turn it off during tests. It's making the test outcome output nearly unreadable.

Actually, generalizing that list of hard-coded silenced packages (or allowing the user to add new packages to it "on the fly" from the test as an argument) might be another reasonable change in the future too.

[massie]

ADAM has a utility function, e.g. ParquetLogger.hadoopLoggerLevel(Level.SEVERE). While Spark uses SFL4J and a log4j backend, Parquet uses Java Logging.