make tachyon version configurable

[ghost]

This patch comes from the tachyon community to resolve issue:
https://tachyon.atlassian.net/browse/TACHYON-11

there will be an accompanying patch to spark/ec2 in apache/spark too.

[shivaram]

Thanks - One more issue we've seen is that Tachyon doesn't work when spark-ec2 sets the Hadoop version to 2 in the environment. It'll be great if you could file that as an issue in Tachyon

[nchammas]

LGTM

[nchammas]

Yeah, we have two other Tachyon/spark-ec2 issues on the Spark JIRA: SPARK-3185 and SPARK-5331

[shivaram]

@uronce-cc One thing that I'm concerned about is that the Spark 1.3 cut has already happened so the spark_ec2.py change will most likely be in Spark 1.4 -- In that case we should probably merge this in the branch-1.4 and not branch-1.3

[haoyuan]

@shivaram In Spark 1.2/1.3, it requires Tachyon 0.5.0. This fix should fix the issue with the current released spark work.

When Spark 1.3 is released, is the plan to create a new spark-ec2 branch based on the current branch-1.3?

[ghost]

@haoyuan When I build the docker image, you told me to support hadoop 1, have hadoop 2 supported yet?

[shivaram]

Ah I see -- Thanks @haoyuan . In that case @uronce-cc could you break up the PR into two parts ? In this PR lets just have Tachyon 0.5, 0.6 added. Once the spark_ec2.py change goes in you can send another PR adding the environment variable.

Does that sound good ?

[haoyuan]

@shivaram Which Hadoop version does Spark run against by default? The current Tachyon bin tar is built against Hadoop 1.0.4. It is simple to have tar ball against other Hadoop version.

[shivaram]

Spark by default is built with 1.0.4 -- You can add another switch statement based on HADOOP_MAJOR_VERSION as it is done in https://github.com/mesos/spark-ec2/blob/branch-1.3/spark/init.sh#L113

P.S: Note that Spark for Hadoop Major Version 2 is actually built with CDH 4 (I think CDH 4.2.0)

[haoyuan]

Thanks @shivaram

@uronce-cc

https://s3.amazonaws.com/Tachyon/tachyon-0.5.0-cdh4-bin.tar.gz
https://s3.amazonaws.com/Tachyon/tachyon-0.6.0-cdh4-bin.tar.gz

[ghost]

@shivaram ok, I'll do the spark_ec2.py task first.

[ghost]

@shivaram You've meant that this PR should be merged to branch-1.4, but there is no branch-1.4 yet, where shall I submit to?

[shivaram]

We don't have a branch-1.4 yet as Spark's master branch version hasn't been updated to 1.4 yet (https://github.com/apache/spark/blob/master/pom.xml#L29) . I think this will happen soon as a release vote is underway -- If not I'll create a branch-1.4 in spark-ec2 in the next couple of days

[ghost]

@shivaram Could you open a branch1.4 in spark-ec2 ?

[shivaram]

Created branch-1.4. I don't have permission to set it as the default branch though
@pwendell could probably help with that