Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-8356][SQL] Reconcile callUDF and callUdf #6902

Closed
wants to merge 3 commits into from
Closed

[SPARK-8356][SQL] Reconcile callUDF and callUdf #6902

wants to merge 3 commits into from

Conversation

BenFradet
Copy link
Contributor

Deprecates callUdf in favor of callUDF.

@SparkQA
Copy link

SparkQA commented Jun 19, 2015

Test build #35270 has finished for PR 6902 at commit 9b1de4d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

@rxin is there any reason to not deprecate the callUDF functions (other than the string one that has been renamed here? They seem redundant with val myUDF = udf(...) and they don't handle input coercion.

@BenFradet BenFradet mentioned this pull request Jun 20, 2015
Closed
@BenFradet
Copy link
Contributor Author

pinging @marmbrus and @rxin

@marmbrus
Copy link
Contributor

Can we also please deprecate the def callUDF(f: FunctionN[_]... functions with a message that says to use udf(...) instead? Thanks!

@BenFradet
Copy link
Contributor Author

Will do, glad to hear this is sorted.

@SparkQA
Copy link

SparkQA commented Jun 22, 2015

Test build #35478 has finished for PR 6902 at commit ef4e9d8.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

Thanks! Merging to master.

@asfgit asfgit closed this in 50d3242 Jun 22, 2015
@BenFradet
Copy link
Contributor Author

Thank you for helping me out.

@marmbrus
Copy link
Contributor

Mind fixing these? I think its okay to drop the deprecated test in order to make the build clean, given the function is deprecated (and I think there is no way to avoid the warning here). The other test can probably just be updated to use the new function.

[warn] /home/michael/spark/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala:309: method callUdf in object functions is deprecated: Use callUDF
[warn]       df.select($"id", callUdf("simpleUdf", $"value")),
[warn]                        ^
[warn] /home/michael/spark/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala:146: method callUDF in object functions is deprecated: Use udf
[warn]     val dfWithId = df.withColumn("id", callUDF(id, StringType))
[warn]                                        ^

@BenFradet
Copy link
Contributor Author

Will do.

@BenFradet
Copy link
Contributor Author

Should I replace calls to the old callUDF by udf as well?
There are some in mllib such as this one.

@marmbrus
Copy link
Contributor

Yeah, that would be awesome where possible. Unfortunately, I think we may also need to add back in a version of callUDF that takes in DataTypes for its input and output parameters. In some places like this no TypeTags are available for udf to use for inference.

/cc @mengxr

@BenFradet
Copy link
Contributor Author

Would it be possible to reopen the PR or should I open another one?

@marmbrus
Copy link
Contributor

Open a new JIRA and PR please.

@BenFradet BenFradet mentioned this pull request Jun 24, 2015
Closed
animeshbaranawal pushed a commit to animeshbaranawal/spark that referenced this pull request Jun 25, 2015
@BenFradet
[SPARK-8356] [SQL] Reconcile callUDF and callUdf
04db247 
Deprecates ```callUdf``` in favor of ```callUDF```.

Author: BenFradet <benjamin.fradet@gmail.com>

Closes apache#6902 from BenFradet/SPARK-8356 and squashes the following commits:

ef4e9d8 [BenFradet] deprecated callUDF, use udf instead
9b1de4d [BenFradet] reinstated unit test for the deprecated callUdf
cbd80a5 [BenFradet] deprecated callUdf in favor of callUDF
asfgit pushed a commit that referenced this pull request Jun 29, 2015
@BenFradet @mengxr
[SPARK-8575] [SQL] Deprecate callUDF in favor of udf
0b10662 
Follow up of [SPARK-8356](https://issues.apache.org/jira/browse/SPARK-8356) and #6902.
Removes the unit test for the now deprecated ```callUdf```
Unit test in SQLQuerySuite now uses ```udf``` instead of ```callUDF```
Replaced ```callUDF``` by ```udf``` where possible in mllib

Author: BenFradet <benjamin.fradet@gmail.com>

Closes #6993 from BenFradet/SPARK-8575 and squashes the following commits:

26f5a7a [BenFradet] 2 spaces instead of 1
1ddb452 [BenFradet] renamed initUDF in order to be consistent in OneVsRest
48ca15e [BenFradet] used vector type tag for udf call in VectorIndexer
0ebd0da [BenFradet] replace the now deprecated callUDF by udf in VectorIndexer
8013409 [BenFradet] replaced the now deprecated callUDF by udf in Predictor
94345b5 [BenFradet] unifomized udf calls in ProbabilisticClassifier
1305492 [BenFradet] uniformized udf calls in Classifier
a672228 [BenFradet] uniformized udf calls in OneVsRest
49e4904 [BenFradet] Revert "removal of the unit test for the now deprecated callUdf"
bbdeaf3 [BenFradet] fixed syntax for init udf in OneVsRest
fe2a10b [BenFradet] callUDF => udf in ProbabilisticClassifier
0ea30b3 [BenFradet] callUDF => udf in Classifier where possible
197ec82 [BenFradet] callUDF => udf in OneVsRest
84d6780 [BenFradet] modified unit test in SQLQuerySuite to use udf instead of callUDF
477709f [BenFradet] removal of the unit test for the now deprecated callUdf
asfgit pushed a commit that referenced this pull request Jun 29, 2015
@BenFradet @marmbrus
[SPARK-8478] [SQL] Harmonize UDF-related code to use uniformly UDF in…
931da5c 
…stead of Udf

Follow-up of #6902 for being coherent between ```Udf``` and ```UDF```

Author: BenFradet <benjamin.fradet@gmail.com>

Closes #6920 from BenFradet/SPARK-8478 and squashes the following commits:

c500f29 [BenFradet] renamed a few variables in functions to use UDF
8ab0f2d [BenFradet] renamed idUdf to idUDF in SQLQuerySuite
98696c2 [BenFradet] renamed originalUdfs in TestHive to originalUDFs
7738f74 [BenFradet] modified HiveUDFSuite to use only UDF
c52608d [BenFradet] renamed HiveUdfSuite to HiveUDFSuite
e51b9ac [BenFradet] renamed ExtractPythonUdfs to ExtractPythonUDFs
8c756f1 [BenFradet] renamed Hive UDF related code
2a1ca76 [BenFradet] renamed pythonUdfs to pythonUDFs
261e6fb [BenFradet] renamed ScalaUdf to ScalaUDF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@BenFradet @SparkQA @marmbrus