-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-11244][SPARKR] sparkR.stop() should remove SQLContext #9205
Conversation
Thanks @saurfang -- Test is fine in the current location. Just curious what are the lint-r errors you saw ? |
Most of them are "style: Commented code should be removed." For example:
Looks like to be a new feature introduced at: r-lib/lintr#83 |
Ah I see. Hmm we use cc @yu-iskw |
Test build #44086 has finished for PR 9205 at commit
|
I see. Thanks for the information. I thought the convention in R package is to expose as much documentation as possible but don't export functions that is unstable or developer only. If users choose to use them, they acknowledge the risk by using |
is it possible for lin-r to skip # comment with code outside a function body? |
I still advocate making developer API documentation public. However I think one workaround is to make them roxygen doc again with |
@saurfang Can you open a new JIRA for these style failures ? We can discuss the points there. LGTM. Merging this in master and branch-1.5 |
cc @olarayej |
SparkR should remove `.sparkRSQLsc` and `.sparkRHivesc` when `sparkR.stop()` is called. Otherwise even when SparkContext is reinitialized, `sparkRSQL.init` returns the stale copy of the object and complains: ```r sc <- sparkR.init("local") sqlContext <- sparkRSQL.init(sc) sparkR.stop() sc <- sparkR.init("local") sqlContext <- sparkRSQL.init(sc) sqlContext ``` producing ```r Error in callJMethod(x, "getClass") : Invalid jobj 1. If SparkR was restarted, Spark operations need to be re-executed. ``` I have added the check and removal only when SparkContext itself is initialized. I have also added corresponding test for this fix. Let me know if you want me to move the test to SQL test suite instead. p.s. I tried lint-r but ended up a lots of errors on existing code. Author: Forest Fang <forest.fang@outlook.com> Closes #9205 from saurfang/sparkR.stop. (cherry picked from commit 94e2064) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
Thanks @shivaram! Opened https://issues.apache.org/jira/browse/SPARK-11263 for further discussion. |
SparkR should remove
.sparkRSQLsc
and.sparkRHivesc
whensparkR.stop()
is called. Otherwise even when SparkContext is reinitialized,sparkRSQL.init
returns the stale copy of the object and complains:producing
I have added the check and removal only when SparkContext itself is initialized. I have also added corresponding test for this fix. Let me know if you want me to move the test to SQL test suite instead.
p.s. I tried lint-r but ended up a lots of errors on existing code.