[spark] Support show columns in spark #3888
Conversation
|
|
||
| assertDoesNotThrow( | ||
| () => { | ||
| spark.sql("SHOW COLUMNS FROM T") |
There was a problem hiding this comment.
Why not check its result?
There was a problem hiding this comment.
UT had added for check its result and it like picture in discription. @JingsongLi
xuzifu666
changed the title
xuzifu666
changed the title
| import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeReference} | ||
| import org.apache.spark.sql.types.{BinaryType, Metadata, StringType} | ||
|
|
||
| case class PaimonShowColumnsCommand(v2Table: SparkTable) |
There was a problem hiding this comment.
this can extend PaimonLeafRunnableCommand directly, then we don't need PaimonShowColumnsExec.
There was a problem hiding this comment.
Thanks，had change it as your suggestion. @YannByron
| |""".stripMargin | ||
| ) | ||
|
|
||
| assertDoesNotThrow( |
There was a problem hiding this comment.
if check its result below, we don't need this assert.
There was a problem hiding this comment.
OK，had check result，I remove the assert @YannByron
| case class PaimonShowColumnsCommand(v2Table: SparkTable) | ||
| extends PaimonLeafRunnableCommand | ||
| with WithFileStoreTable { | ||
| override def table: FileStoreTable = v2Table.getTable.asInstanceOf[FileStoreTable] |
There was a problem hiding this comment.
maybe this command can also apply to the system table, not just FileStoreTable. if yes, modify here and add a UT for that.
There was a problem hiding this comment.
Thanks for comments，I had a try but seems not find a way to compatible with system table and FileStoreTable due to their not use same interface. Maybe we can just support FileStoreTable firstly keep the same with other Command？
Purpose
Support show columns syntax in paimon，currently spark not support v2 in show columns(fix in spark apache/spark#47568 ) but paimon side not support yet，would error out like：
spark-sql> show COLUMNS from bdsp_test.paimon_mi_31;
Error in query: SHOW COLUMNS is not supported for v2 tables.
Add spark analyzer extensions to capture it，after fix:
Linked issue: #3889
Tests
API and Format
Documentation