SPARK-1478

[tmalaska]

SPARK-1478

[AmplabJenkins]

Can one of the admins verify this patch?

[pwendell]

Jenkins, test this please. @tmalaska mind updating the title of the PR to include the title of the JIRA? It makes it easier when scanning the (long list) of active pull requests.

[AmplabJenkins]

Merged build triggered.

[AmplabJenkins]

Merged build started.

[AmplabJenkins]

Merged build finished.

[AmplabJenkins]

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14118/

[tdas]

@tmalaska I did a cursory pass, this looks good. I will do a more detailed pass soon. However, there something you should know. I am in the middle of a PR ( #300 ) that tweaks the receiver API a little bit for greater stability and so a bit of your code will have a to change a little. This should go in pretty soon (couple of days, max). The PR has the changes necessary for the current FlumeReceiver.

[tmalaska]

Yeah no problem. Thanks for taking the time to review my code. This is my first time committing with Scala :)

Just let me know when ( #300 ) is done and I will re check out. Also when you have time I would love to know how else I could help.

I was thinking of adding :

• encryption to the Flume Stream as is in Flume 1.4.0.
• Fail recover support when a Flume Stream host goes down and Spark starts up the Flume Stream on another node.

[tdas]

@tmalaska You probably missed Patrick's comment about updating the title of the PR. :) Please do update it with the title in the JIRA, so that its easier to identify in git's logs.

Also, please update your branch by merging the master branch. I still see flume version 1.3 in the diff, which i expected to be 1.4.

[tdas]

Still 1.3? Have you merged master branch with your branch?

[tdas]

Aaah, crap, I did not realize that you submitted a new PR #566 for SPARK-1478!
Can you please close this PR?

[tmalaska]

As soon as I figure out how. I will look into it after work.