Make sed do -i '' on OSX

[nicklan]

I don't have access to an OSX machine, so if someone could test this that would be great.

[AmplabJenkins]

Merged build triggered. One or more automated tests failed

[AmplabJenkins]

Merged build started. One or more automated tests failed

[mateiz]

Jenkins, test this please

[AmplabJenkins]

Merged build triggered. One or more automated tests failed

[mateiz]

@nicklan I actually see some other errors due to pushd and popd now:

./make-distribution.sh: line 133: pushd: no other directory
Fetchting tachyon tgz
--2014-03-27 11:31:11--  https://github.com/amplab/tachyon/releases/download/v0.4.1/tachyon-0.4.1-bin.tar.gz
Resolving github.com... 192.30.252.131
Connecting to github.com|192.30.252.131|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3.amazonaws.com/github-cloud/releases/7276954/dedb4c68-a574-11e3-960d-341edab9bc3c.gz?response-content-disposition=attachment%3B%20filename%3Dtachyon-0.4.1-bin.tar.gz&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1395945132&Signature=3%2FZTB8tWWf89B8pc2duZ6gl6n%2F4%3D [following]
--2014-03-27 11:31:12--  https://s3.amazonaws.com/github-cloud/releases/7276954/dedb4c68-a574-11e3-960d-341edab9bc3c.gz?response-content-disposition=attachment%3B%20filename%3Dtachyon-0.4.1-bin.tar.gz&AWSAccessKeyId=AKIAISTNZFOVBIJMK3TQ&Expires=1395945132&Signature=3%2FZTB8tWWf89B8pc2duZ6gl6n%2F4%3D
Resolving s3.amazonaws.com... 207.171.189.80
Connecting to s3.amazonaws.com|207.171.189.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22751619 (22M) [application/octet-stream]
Saving to: 'tachyon-0.4.1-bin.tar.gz'

100%[======================================>] 22,751,619  3.79MB/s   in 9.5s   

2014-03-27 11:31:23 (2.28 MB/s) - 'tachyon-0.4.1-bin.tar.gz' saved [22751619/22751619]

./make-distribution.sh: line 151: popd: directory stack empty

Haven't looked into it much, see if you can get someone with a Mac to work with.

[AmplabJenkins]

Merged build started. One or more automated tests failed

[nicklan]

@mateiz huh, that's odd, it suggests that mktemp -d returned an empty string. Were there any other errors right before the pushd one? I also checked and that should work on OSX. I really have no one around with a Mac today, but I'll try and get this sorted.

[AmplabJenkins]

Merged build finished. All automated tests passed.

[AmplabJenkins]

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13523/

[nicklan]

@mateiz okay, I think i figured out the mktemp problem. if you could test it once more that would be great. i'll also try and test it on a mac later today if you're too busy at the moment.

[AmplabJenkins]

Merged build triggered. One or more automated tests failed

[AmplabJenkins]

Merged build started. One or more automated tests failed

[AmplabJenkins]

Merged build finished. One or more automated tests failed

[AmplabJenkins]

One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13525/

[AmplabJenkins]

Merged build finished. One or more automated tests failed

[AmplabJenkins]

One or more automated tests failed
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13526/

[mateiz]

So it seems that on OS X, mktemp needs a template: https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/mktemp.1.html. Just doing mktemp -d would fail. The best way to run it there will be mktemp -d -t spark, which will put a file in TMPDIR with "spark" in its name. If that also works on GNU mktemp, we can go wit that.

The pushd/popd errors are because mktemp failed, but that was earlier in the log.

[nicklan]

Yeah, have you tried with my latests PR where I added the -t?

On Thu, Mar 27, 2014 at 6:08 PM, Matei Zaharia notifications@git.luolix.topwrote:

So it seems that on OS X, mktemp needs a template:
https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/mktemp.1.html.
Just doing mktemp -d would fail. The best way to run it there will be mktemp
-d -t spark, which will put a file in TMPDIR with "spark" in its name. If
that also works on GNU mktemp, we can go wit that.

The pushd/popd errors are because mktemp failed, but that was earlier in
the log.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/258#issuecomment-38879651
.

[mateiz]

Actually nm, saw you fixed it. This fixes make-distribution.sh, although unfortunately Tachyon as packaged here doesn't properly deploy on OS X. If I go into dist, add a conf/slaves file, and run sbin/start-all.sh --with-tachyon, I get:

starting org.apache.spark.deploy.master.Master, logging to /Users/matei/workspace/apache-spark/dist/sbin/../logs/spark-matei-org.apache.spark.deploy.master.Master-1-mbp-3.out
sed: 1: "/Users/matei/workspace/ ...": invalid command code m
awk: can't open file /proc/meminfo
 source line number 1
sed: 1: "/Users/matei/workspace/ ...": invalid command code m
localhost: Formatting Tachyon Worker @ mbp-3
localhost: Error: Could not find or load main class tachyon.Format
Formatting Tachyon Master @ localhost
Error: Could not find or load main class tachyon.Format

However we can fix this in another JIRA, so I'll merge this PR as is.

[mateiz]

(BTW if I run this twice, the "sed" error goes away but I still get "Error: Could not find or load main class tachyon.Format").

[mateiz]

Oh one other thing, make an account on our JIRA and I can assign https://spark-project.atlassian.net/browse/SPARK-1326 to you.

[nicklan]

Okay will do, and I can look at these other errors tomorrow
On Mar 27, 2014 6:20 PM, "Matei Zaharia" notifications@github.com wrote:

Oh one other thing, make an account on our JIRA and I can assign
https://spark-project.atlassian.net/browse/SPARK-1326 to you.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/258#issuecomment-38880262
.

[mateiz]

Thanks, I've now merged this.

[nicklan]

Some of this is unfortunately a bug in the tachyon scripts. I'll submit a
PR there to get them fixed. Not sure it not being able to find the main
class however, that seems bad. I'll dig a bit.

On Thu, Mar 27, 2014 at 6:17 PM, Matei Zaharia notifications@git.luolix.topwrote:

Actually nm, saw you fixed it. This fixes make-distribution.sh, although
unfortunately Tachyon as packaged here doesn't properly deploy on OS X. If
I go into dist, add a conf/slaves file, and run sbin/start-all.sh
--with-tachyon, I get:

starting org.apache.spark.deploy.master.Master, logging to /Users/matei/workspace/apache-spark/dist/sbin/../logs/spark-matei-org.apache.spark.deploy.master.Master-1-mbp-3.out
sed: 1: "/Users/matei/workspace/ ...": invalid command code m
awk: can't open file /proc/meminfo
source line number 1
sed: 1: "/Users/matei/workspace/ ...": invalid command code m
localhost: Formatting Tachyon Worker @ mbp-3
localhost: Error: Could not find or load main class tachyon.Format
Formatting Tachyon Master @ localhost
Error: Could not find or load main class tachyon.Format

However we can fix this in another JIRA, so I'll merge this PR as is.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/258#issuecomment-38880156
.