Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-2179][SQL] Public API for DataTypes and Schema #1346

Closed
wants to merge 47 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
16be3e5
This commit contains three changes:
yhuai Jul 9, 2014
3fa0df5
Provide easier ways to construct a StructType.
yhuai Jul 10, 2014
90460ac
Infer the Catalyst data type from an object and cast a data value to …
yhuai Jul 10, 2014
03eec4c
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 10, 2014
0266761
Format
yhuai Jul 10, 2014
43a45e1
Remove sql.util.package introduced in a previous commit.
yhuai Jul 11, 2014
7a6a7e5
Fix bug introduced by the change made on SQLContext.inferSchema.
yhuai Jul 11, 2014
949d6bb
When creating a SchemaRDD for a JSON dataset, users can apply an exis…
yhuai Jul 11, 2014
eca7d04
Add two apply methods which will be used to extract StructField(s) fr…
yhuai Jul 11, 2014
fc649d7
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 11, 2014
9168b83
Update comments.
yhuai Jul 11, 2014
dcaf22f
Add a field containsNull to ArrayType to indicate if an array can con…
yhuai Jul 12, 2014
3209108
Add unit tests.
yhuai Jul 16, 2014
68525a2
Update JSON unit test.
yhuai Jul 16, 2014
b8b7db4
1. Move sql package object and package-info to sql-core.
yhuai Jul 16, 2014
2e58dbd
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 16, 2014
c3f4a02
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 16, 2014
42d47a3
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 16, 2014
e495e4e
More comments.
yhuai Jul 16, 2014
85e9b51
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 16, 2014
1d9c13a
Update applySchema API.
yhuai Jul 23, 2014
9c99bc0
Several minor updates.
yhuai Jul 23, 2014
8da1a17
Add Row.fromSeq.
yhuai Jul 23, 2014
aa92e84
Update data type tests.
yhuai Jul 23, 2014
624765c
Tests for applySchema.
yhuai Jul 23, 2014
1c9f33c
Java APIs for DataTypes and Row.
yhuai Jul 24, 2014
b9f3071
Java API for applySchema.
yhuai Jul 24, 2014
33c4fec
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 24, 2014
d48fc7b
Minor updates.
yhuai Jul 25, 2014
1db9531
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 25, 2014
246da96
Add java data type APIs to javadoc index.
yhuai Jul 25, 2014
1d93395
Python APIs.
yhuai Jul 28, 2014
692c0b9
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 28, 2014
3edb3ae
Python doc.
yhuai Jul 28, 2014
1cb35fe
Add "valueContainsNull" to MapType.
yhuai Jul 28, 2014
991f860
Move "asJavaDataType" and "asScalaDataType" to DataTypeConversions.sc…
yhuai Jul 28, 2014
bd40a33
Address comments.
yhuai Jul 29, 2014
fc2bed1
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 29, 2014
ab71f21
Format.
yhuai Jul 29, 2014
2476ed0
Minor updates.
yhuai Jul 29, 2014
03bfd95
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 29, 2014
122d1e7
Address comments.
yhuai Jul 29, 2014
e5f8df5
Scaladoc.
yhuai Jul 29, 2014
4ceeb66
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 29, 2014
c712fbf
Converts types of values based on defined schema.
yhuai Jul 30, 2014
a6e08b4
Merge remote-tracking branch 'upstream/master' into dataTypeAndSchema
yhuai Jul 30, 2014
1d45977
Clean up.
yhuai Jul 30, 2014
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -544,7 +544,8 @@ private[spark] object PythonRDD extends Logging {
}

/**
* Convert an RDD of serialized Python dictionaries to Scala Maps
* Convert an RDD of serialized Python dictionaries to Scala Maps (no recursive conversions).
* It is only used by pyspark.sql.
* TODO: Support more Python types.
*/
def pythonToJavaMap(pyRDD: JavaRDD[Array[Byte]]): JavaRDD[Map[String, _]] = {
Expand Down
2 changes: 1 addition & 1 deletion project/SparkBuild.scala
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,7 @@ object Unidoc {
"mllib.regression", "mllib.stat", "mllib.tree", "mllib.tree.configuration",
"mllib.tree.impurity", "mllib.tree.model", "mllib.util"
),
"-group", "Spark SQL", packageList("sql.api.java", "sql.hive.api.java"),
"-group", "Spark SQL", packageList("sql.api.java", "sql.api.java.types", "sql.hive.api.java"),
"-noqualifier", "java.lang"
)
)
Expand Down
Loading