Examples for the Learning Spark book. These examples require a number of libraries and as such have long build files. We have also added a stand alone example with minimal dependcies and a small build file in the mini-complete-example directory.
- JDK 1.7 or higher
- Scala 2.10.3
- scala-lang.org
- Spark 1.0 sanp shot
- You can checkout spark from https://github.com/apache/spark and then run "sbt/sbt publish-local"
- Protobuf compiler
- On debian you can install with sudo apt-get install protobuf-compiler
- R & the CRAN package Imap are required for the ChapterSixExample
- The Python examples require urllib3
From spark just run ./bin/pyspark ./src/python/[example]
You can also create an assembly jar with all of the dependcies for running either the java or scala versions of the code and run the job with the spark-submit script
./sbt/sbt assembly OR mvn package cd $SPARK_HOME; ./bin/spark-submit --class com.oreilly.learningsparkexamples.[lang].[example] ../learning-spark-examples/target/scala-2.10/learning-spark-examples-assembly-0.0.1.jar
