Updates to shell globbing in run-example and spark-class #449
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
|
Looks like Travis failed by exceeding 50 minutes, is that the norm? I don't see any other error. |
Conflicts: bin/run-example
pwendell
pushed a commit
to pwendell/spark
that referenced
this pull request
May 12, 2014
SPARK-1028 : fix "set MASTER automatically fails" bug. spark-shell intends to set MASTER automatically if we do not provide the option when we start the shell , but there's a problem. The condition is "if [[ "x" != "x$SPARK_MASTER_IP" && "y" != "y$SPARK_MASTER_PORT" ]];" we sure will set SPARK_MASTER_IP explicitly, the SPARK_MASTER_PORT option, however, we probably do not set just using spark default port 7077. So if we do not set SPARK_MASTER_PORT, the condition will never be true. We should just use default port if users do not set port explicitly I think.
Conflicts: bin/run-example bin/spark-class
|
Jenkins, test this please |
|
Build triggered. |
|
Build started. |
|
Build finished. |
|
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15365/ |
|
So now the failure is because your branch can't be merged automatically with master -- do you mind rebasing it on top of master? Before that 50 minute timeout was probably an issue with Jenkins. |
|
Can one of the admins verify this patch? |
gzm55
pushed a commit
to MediaV/spark
that referenced
this pull request
Jul 17, 2014
SPARK-1028 : fix "set MASTER automatically fails" bug. spark-shell intends to set MASTER automatically if we do not provide the option when we start the shell , but there's a problem. The condition is "if [[ "x" != "x$SPARK_MASTER_IP" && "y" != "y$SPARK_MASTER_PORT" ]];" we sure will set SPARK_MASTER_IP explicitly, the SPARK_MASTER_PORT option, however, we probably do not set just using spark default port 7077. So if we do not set SPARK_MASTER_PORT, the condition will never be true. We should just use default port if users do not set port explicitly I think. (cherry picked from commit 6b4eed7) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
|
Hi @kenahoo do you mind creating an associated JIRA and adding the number to the title? This will allow us to track what code changes correspond to what issues more easily. Also, could you rebase this on top of master once you have a chance? There have been many changes in these scripts in the recent past. |
|
Can one of the admins verify this patch? |
| export SPARK_EXAMPLES_JAR=`ls "$EXAMPLES_DIR"/target/scala-$SCALA_VERSION/spark-examples-*hadoop*.jar` | ||
| fi | ||
| . $FWDIR/bin/load-spark-env.sh | ||
| . $FWDIR/bin/sh-funcs.sh |
There was a problem hiding this comment.
We generally quote string like above as "$FWDIR"/bin/sh-funcs.sh and so on.
|
This is stale so let's close this issue. |
andrewor14
pushed a commit
to andrewor14/spark
that referenced
this pull request
Jan 8, 2015
SPARK-1028 : fix "set MASTER automatically fails" bug. spark-shell intends to set MASTER automatically if we do not provide the option when we start the shell , but there's a problem. The condition is "if [[ "x" != "x$SPARK_MASTER_IP" && "y" != "y$SPARK_MASTER_PORT" ]];" we sure will set SPARK_MASTER_IP explicitly, the SPARK_MASTER_PORT option, however, we probably do not set just using spark default port 7077. So if we do not set SPARK_MASTER_PORT, the condition will never be true. We should just use default port if users do not set port explicitly I think. (cherry picked from commit 6b4eed7) Signed-off-by: Patrick Wendell <pwendell@gmail.com>
j-esse
pushed a commit
to j-esse/spark
that referenced
this pull request
Jan 24, 2019
###### _excavator_ is a bot for automating changes across repositories.
Changes produced by the roomba/latest-gradle-wrapper-oss check.
{runtimeCheckDesc}
To enable or disable this check, please contact the maintainers of Excavator.
bzhaoopenstack
pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
Update the pipeline and jobs to add 1.14 K8s
arjunshroff
pushed a commit
to arjunshroff/spark
that referenced
this pull request
Nov 24, 2020
RolatZhang
pushed a commit
to RolatZhang/spark
that referenced
this pull request
Aug 15, 2022
…e's eventlog name (apache#449) * AL-5705 update pom version * AL-5705 when taskId is not empty, add taskId subStr(0,8) in local mode's eventlog name * AL-5705 add UT Co-authored-by: longfei.jiang <longfei.jiang@kyligence.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I was running into some shell problems & warnings that I couldn't figure out. It turned out I had a few jars left over from previous builds.
There were a few inconsistencies in how shell globbing was done, and some places where the code assumed only one file could match a glob, with unsightly consequences if more than one file matched. I wrote a couple functions that handle globbing in a more consistent way, and use them in the workhorse scripts. With these changes it should be more obvious to someone like me what the problem is.
I'm not a Bash expert, BTW. Much of Bash's syntax looks screwy to me, if there's some better way to do what I did, by all means modify.