Update slaves.sh #238
Closed
Update slaves.sh #238
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
update the comment
|
Can one of the admins verify this patch? |
| @@ -43,7 +43,7 @@ sbin=`cd "$sbin"; pwd` | |||
|
|
|||
| # If the slaves file is specified in the command line, | |||
| # then it takes precedence over the definition in | |||
| # spark-env.sh. Save it here. | |||
| # load-spark-env.sh. Save it here. | |||
There was a problem hiding this comment.
I'm not sure this change is correct. We still want users defining variables inside of spark-env.sh right?
|
yeah ,get it. thank you@pwendell. close it |
andrewor14
pushed a commit
to andrewor14/spark
that referenced
this pull request
Apr 7, 2014
upgrade Netty from 4.0.0.Beta2 to 4.0.13.Final the changes are listed at https://github.com/netty/netty/wiki/New-and-noteworthy
davies
pushed a commit
to davies/spark
that referenced
this pull request
Apr 14, 2015
[SPARKR-154] Phase 3: implement intersection().
asfgit
pushed a commit
that referenced
this pull request
Apr 17, 2015
This PR pulls in recent changes in SparkR-pkg, including cartesian, intersection, sampleByKey, subtract, subtractByKey, except, and some API for StructType and StructField. Author: cafreeman <cfreeman@alteryx.com> Author: Davies Liu <davies@databricks.com> Author: Zongheng Yang <zongheng.y@gmail.com> Author: Shivaram Venkataraman <shivaram.venkataraman@gmail.com> Author: Shivaram Venkataraman <shivaram@cs.berkeley.edu> Author: Sun Rui <rui.sun@intel.com> Closes #5436 from davies/R3 and squashes the following commits: c2b09be [Davies Liu] SQLTypes -> schema a5a02f2 [Davies Liu] Merge branch 'master' of github.com:apache/spark into R3 168b7fe [Davies Liu] sort generics b1fe460 [Davies Liu] fix conflict in README.md e74c04e [Davies Liu] fix schema.R 4f5ac09 [Davies Liu] Merge branch 'master' of github.com:apache/spark into R5 41f8184 [Davies Liu] rm man ae78312 [Davies Liu] Merge pull request #237 from sun-rui/SPARKR-154_3 1bdcb63 [Zongheng Yang] Updates to README.md. 5a553e7 [cafreeman] Use object attribute instead of argument 71372d9 [cafreeman] Update docs and examples 8526d2e [cafreeman] Remove `tojson` functions 6ef5f2d [cafreeman] Fix spacing 7741d66 [cafreeman] Rename the SQL DataType function 141efd8 [Shivaram Venkataraman] Merge pull request #245 from hqzizania/upstream 9387402 [Davies Liu] fix style 40199eb [Shivaram Venkataraman] Move except into sorted position 07d0dbc [Sun Rui] [SPARKR-244] Fix test failure after integration of subtract() and subtractByKey() for RDD. 7e8caa3 [Shivaram Venkataraman] Merge pull request #246 from hlin09/fixCombineByKey ed66c81 [cafreeman] Update `subtract` to work with `generics.R` f3ba785 [cafreeman] Fixed duplicate export 275deb4 [cafreeman] Update `NAMESPACE` and tests 1a3b63d [cafreeman] new version of `CreateDF` 836c4bf [cafreeman] Update `createDataFrame` and `toDF` be5d5c1 [cafreeman] refactor schema functions 40338a4 [Zongheng Yang] Merge pull request #244 from sun-rui/SPARKR-154_5 20b97a6 [Zongheng Yang] Merge pull request #234 from hqzizania/assist ba54e34 [Shivaram Venkataraman] Merge pull request #238 from sun-rui/SPARKR-154_4 c9497a3 [Shivaram Venkataraman] Merge pull request #208 from lythesia/master b317aa7 [Zongheng Yang] Merge pull request #243 from hqzizania/master 136a07e [Zongheng Yang] Merge pull request #242 from hqzizania/stats cd66603 [cafreeman] new line at EOF 8b76e81 [Shivaram Venkataraman] Merge pull request #233 from redbaron/fail-early-on-missing-dep 7dd81b7 [cafreeman] Documentation 0e2a94f [cafreeman] Define functions for schema and fields
jamesrgrinter
pushed a commit
to jamesrgrinter/spark
that referenced
this pull request
Apr 22, 2018
Igosuki
pushed a commit
to Adikteev/spark
that referenced
this pull request
Jul 31, 2018
…sions of DC/OS. Includes all tests except HDFS ones. (apache#238)
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
Fix neutron-lbaas service launching error
arjunshroff
pushed a commit
to arjunshroff/spark
that referenced
this pull request
Nov 24, 2020
cloud-fan
added a commit
that referenced
this pull request
Jul 20, 2023
…n't be optimized
### What changes were proposed in this pull request?
Eg:
```scala
sql("create view t(c1, c2) as values (0, 1), (0, 2), (1, 2)")
sql("select c1, c2, (select count(*) cnt from t t2 where t1.c1 = t2.c1 " +
"having cnt = 0) from t t1").show()
```
The error will throw:
```
[PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery in batch Operator Optimization before Inferring Filters generated an invalid plan: The plan becomes unresolved: 'Project [toprettystring(c1#224, Some(America/Los_Angeles)) AS toprettystring(c1)#238, toprettystring(c2#225, Some(America/Los_Angeles)) AS toprettystring(c2)#239, toprettystring(cnt#246L, Some(America/Los_Angeles)) AS toprettystring(scalarsubquery(c1))#240]
+- 'Project [c1#224, c2#225, CASE WHEN isnull(alwaysTrue#245) THEN 0 WHEN NOT (cnt#222L = 0) THEN null ELSE cnt#222L END AS cnt#246L]
+- 'Join LeftOuter, (c1#224 = c1#224#244)
:- Project [col1#226 AS c1#224, col2#227 AS c2#225]
: +- LocalRelation [col1#226, col2#227]
+- Project [cnt#222L, c1#224#244, cnt#222L, c1#224, true AS alwaysTrue#245]
+- Project [cnt#222L, c1#224 AS c1#224#244, cnt#222L, c1#224]
+- Aggregate [c1#224], [count(1) AS cnt#222L, c1#224]
+- Project [col1#228 AS c1#224]
+- LocalRelation [col1#228, col2#229]The previous plan: Project [toprettystring(c1#224, Some(America/Los_Angeles)) AS toprettystring(c1)#238, toprettystring(c2#225, Some(America/Los_Angeles)) AS toprettystring(c2)#239, toprettystring(scalar-subquery#223 [c1#224 && (c1#224 = c1#224#244)], Some(America/Los_Angeles)) AS toprettystring(scalarsubquery(c1))#240]
: +- Project [cnt#222L, c1#224 AS c1#224#244]
: +- Filter (cnt#222L = 0)
: +- Aggregate [c1#224], [count(1) AS cnt#222L, c1#224]
: +- Project [col1#228 AS c1#224]
: +- LocalRelation [col1#228, col2#229]
+- Project [col1#226 AS c1#224, col2#227 AS c2#225]
+- LocalRelation [col1#226, col2#227]
```
The reason of error is the unresolved expression in `Join` node which generate by subquery decorrelation. The `duplicateResolved` in `Join` node are false. That's meaning the `Join` left and right have same `Attribute`, in this eg is `c1#224`. The right `c1#224` `Attribute` generated by having Inputs, because there are wrong having Inputs.
This problem only occurs when there contain having clause.
also do some code format fix.
### Why are the changes needed?
Fix subquery bug on single table when use having clause
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add new test
Closes #41347 from Hisoka-X/SPARK-43838_subquery_having.
Lead-authored-by: Jia Fan <fanjiaeminem@qq.com>
Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
ragnarok56
pushed a commit
to ragnarok56/spark
that referenced
this pull request
Mar 2, 2024
…n't be optimized
### What changes were proposed in this pull request?
Eg:
```scala
sql("create view t(c1, c2) as values (0, 1), (0, 2), (1, 2)")
sql("select c1, c2, (select count(*) cnt from t t2 where t1.c1 = t2.c1 " +
"having cnt = 0) from t t1").show()
```
The error will throw:
```
[PLAN_VALIDATION_FAILED_RULE_IN_BATCH] Rule org.apache.spark.sql.catalyst.optimizer.RewriteCorrelatedScalarSubquery in batch Operator Optimization before Inferring Filters generated an invalid plan: The plan becomes unresolved: 'Project [toprettystring(c1#224, Some(America/Los_Angeles)) AS toprettystring(c1)apache#238, toprettystring(c2#225, Some(America/Los_Angeles)) AS toprettystring(c2)apache#239, toprettystring(cnt#246L, Some(America/Los_Angeles)) AS toprettystring(scalarsubquery(c1))apache#240]
+- 'Project [c1#224, c2#225, CASE WHEN isnull(alwaysTrue#245) THEN 0 WHEN NOT (cnt#222L = 0) THEN null ELSE cnt#222L END AS cnt#246L]
+- 'Join LeftOuter, (c1#224 = c1#224#244)
:- Project [col1#226 AS c1#224, col2#227 AS c2#225]
: +- LocalRelation [col1#226, col2#227]
+- Project [cnt#222L, c1#224#244, cnt#222L, c1#224, true AS alwaysTrue#245]
+- Project [cnt#222L, c1#224 AS c1#224#244, cnt#222L, c1#224]
+- Aggregate [c1#224], [count(1) AS cnt#222L, c1#224]
+- Project [col1#228 AS c1#224]
+- LocalRelation [col1#228, col2#229]The previous plan: Project [toprettystring(c1#224, Some(America/Los_Angeles)) AS toprettystring(c1)apache#238, toprettystring(c2#225, Some(America/Los_Angeles)) AS toprettystring(c2)apache#239, toprettystring(scalar-subquery#223 [c1#224 && (c1#224 = c1#224#244)], Some(America/Los_Angeles)) AS toprettystring(scalarsubquery(c1))apache#240]
: +- Project [cnt#222L, c1#224 AS c1#224#244]
: +- Filter (cnt#222L = 0)
: +- Aggregate [c1#224], [count(1) AS cnt#222L, c1#224]
: +- Project [col1#228 AS c1#224]
: +- LocalRelation [col1#228, col2#229]
+- Project [col1#226 AS c1#224, col2#227 AS c2#225]
+- LocalRelation [col1#226, col2#227]
```
The reason of error is the unresolved expression in `Join` node which generate by subquery decorrelation. The `duplicateResolved` in `Join` node are false. That's meaning the `Join` left and right have same `Attribute`, in this eg is `c1#224`. The right `c1#224` `Attribute` generated by having Inputs, because there are wrong having Inputs.
This problem only occurs when there contain having clause.
also do some code format fix.
### Why are the changes needed?
Fix subquery bug on single table when use having clause
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Add new test
Closes apache#41347 from Hisoka-X/SPARK-43838_subquery_having.
Lead-authored-by: Jia Fan <fanjiaeminem@qq.com>
Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
update the comment