Qualification tool: Parse expressions in ProjectExec #5952
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
|
build |
|
If you haven't yet we should test against the NDS event logs |
tgravescs
reviewed
Jul 6, 2022
| @@ -268,6 +268,25 @@ object SQLPlanParser extends Logging { | |||
| funcName | |||
| } | |||
|
|
|||
| def parseProjectExpressions(exprStr: String): Array[String] = { | |||
| val parsedExpressions = ArrayBuffer[String]() | |||
| // remove the alias names before parsing | |||
There was a problem hiding this comment.
can you add an example in the comment
| val pattern = """(AS) ([(\w# )]+)""".r | ||
| // This is to split multiple column names in Project. Project may have a function on a column. | ||
| // This will contain array of columns names specified in ProjectExec. | ||
| val paranRemoved = pattern.replaceAllIn(exprStr.replace("),", "::"), "") |
There was a problem hiding this comment.
lets start putting more description on these so when we come back to them we can understand.... put example as well.
| "ProjectExprsSupported") { spark => | ||
| import spark.implicits._ | ||
| val df1 = Seq(9.9, 10.2, 11.6, 12.5).toDF("value") | ||
| df1.write.parquet(s"$outputLoc/testtext") |
There was a problem hiding this comment.
put parquet in name of directory
| "ProjectExprsNotSupported") { spark => | ||
| import spark.implicits._ | ||
| val df1 = spark.sparkContext.parallelize(List(10, 20, 30, 40)).toDF | ||
| df1.write.parquet(s"$outputLoc/testtext") |
There was a problem hiding this comment.
same here put parquet inname for outputloc
12 tasks
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
I tested this against NDS event logs and surprisingly we don't have any unsupported exprs in any of the ProjectExec. I created some custom eventlogs with unsupported exprs to be sure that the testing is done correctly. |
|
build |
tgravescs
approved these changes
Jul 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This contributes to #5617 .
In this PR we parse expressions which are in ProjectExec. Project may have aliases so made sure we are taking that into account. Tested with several expressions locally but added only 1 test each for supporting and non-supporting case. Supporting case pretty much covers all cases where the ProjectExec string needs to be parsed.
There was a bug in FilterExec parser where we are not taking into account if the predicate was
+. Updated that as well.