Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support count on Struct of Map(String,String) #5056

Closed
viadea opened this issue Mar 25, 2022 · 1 comment
Closed

[FEA] Support count on Struct of Map(String,String) #5056

viadea opened this issue Mar 25, 2022 · 1 comment
Labels
duplicate This issue or pull request already exists

Comments

@viadea
Copy link
Collaborator

viadea commented Mar 25, 2022

I wish we can support count on Struct of Map(String,String).

For example:

import org.apache.spark.sql.Row
import org.apache.spark.sql.types._

val data = Seq(
    Row("Adam",Map("hair"->"black","eye"->"black"), Row("address" , Map("state"->"CA"),Map("city"->"santa clara"))),
    Row("Bob",Map("hair"->"red","eye"->"red"),Row("address", Map("state"->"GA"),Map("city"->"abc"))),
    Row("Cathy",Map("hair"->"blue","eye"->"blue"),Row("address",Map("state"->"NC"),Map("city"->"xyz")))
)

val mapType  = DataTypes.createMapType(StringType,StringType)

val schema = new StructType().add("name",StringType).add("properties", mapType).add("prop2", new StructType().add("propname",StringType).add("address", mapType).add("address2", mapType))

val mapTypeDF = spark.createDataFrame(spark.sparkContext.parallelize(data),schema)
mapTypeDF.write.format("parquet").mode("overwrite").save("/tmp/testparquet")
val df1 = spark.read.parquet("/tmp/testparquet")
df1.createOrReplaceTempView("df1")
df1.printSchema

spark.sql("SELECT count(prop2) FROM df1").collect()

Not-supported messages:

!Expression <Count> count(prop2#228) cannot run on GPU because input expression AttributeReference prop2#228 (child MapType(StringType,StringType,true) is not supported, child MapType(StringType,StringType,true) is not supported)

@viadea viadea added feature request New feature or request ? - Needs Triage Need team to review and classify labels Mar 25, 2022
@sameerz sameerz added duplicate This issue or pull request already exists and removed feature request New feature or request ? - Needs Triage Need team to review and classify labels Mar 29, 2022
@sameerz
Copy link
Collaborator

sameerz commented Mar 29, 2022

This is a duplicate of #5060

@sameerz sameerz closed this as completed Mar 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
duplicate This issue or pull request already exists
Projects
None yet
Development

No branches or pull requests

2 participants