Skip to content

Feature Support Detail

shiyuhang0 edited this page Jul 8, 2022 · 13 revisions

Main Features

Feature Support TiSpark 2.4.x TiSpark 2.5.x TiSpark 3.0.x TiSpark master
SQL select without tidb_catalog
SQL select with tidb_catalog
DataFrame append
DataFrame reads
SQL show databases
SQL show tables
SQL auth
SQL delete from with tidb_catalog
TLS
DataFrame auth

Read

you can read with both SQL select and DataFrame reads

Push down

TiSpark can push down some operators to TiKV

  • predicate
  • aggregate

Aggregate

The following aggregate can be pushed down

  • Min
  • Max
  • Count
  • Sum
  • Avg
push down types comment
sum smallint,bigint,decimal,mediumint,real,tinyint,int,double,year Spark JDBC does not support sum(year)
count only set can not be pushed down Count(col1,col2,...) will not be pushed down
avg smallint,bigint,decimal,mediumint,real,tinyint,int,double,year Spark JDBC does not support avg(year)
  • Sum(float) can't be pushed down because Spark will cast float to double, but cast can't be pushed down in TiSpark.
  • Count(set) can't be pushed down because TiKV does not support set type.
  • Count(col1,col2,...) can't be pushed down because TiDB/TiKV does not support Count(col1,col2,...).
  • Avg can be pushed down when both the sum and count can be pushed down.

Table Partition

TiSpark currently supports Range Partition and Hash Partition. Users can select data from the Range Partition table and the Hash Partition table through TiSpark.

In most cases, TiSpark use a full table scan on partition tables. Only in certain cases, TiSpark applies partition pruning. For more details, see here.

Expression Index

tidb-5.0 supports Expression Index.

TiSpark currently supports retrieving data from table with Expression Index, but the Expression Index will not be used by the planner of TiSpark.

TiFlash

TiSpark can read from TiFlash with the configuration spark.tispark.isolation_read_engines

Write

You can only write to TiDB with DataFrame append now

Not supported

  • write to the partition table

Feature List