[spark] installation and test on Ubuntu, docker, OS X

• java
  • https://www.digitalocean.com/community/tutorials/how-to-install-java-on-ubuntu-with-apt-get
  • version : java 7
• scala
  • http://www.scala-lang.org/download/2.10.4.html
  • version : scala 2.10.4
• hadoop
  • http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
    • single node에서 가상으로 distributed cluster를 구성
  • 모든 설정이 끝난 이후, 다시 docker run

  docker run -i -t -p 9001:9001 -p 50070:50070 -p 9000:9000 -p 8088:8088 -p 8089:8089 image_name
  /etc/init.d/ssh start
  /root/hadoop-2.7.2/sbin/start-dfs.sh
  /root/hadoop-2.7.2/sbin/start-yarn.sh
  

  • dfs namenode interface : http://192.168.99.100:50070/
  • yarn resource manager interface : http://192.168.99.100:8088/
  • virtual box에서 port forwarding
    • 이걸 설정해야 OS X host ip로 web interface에 접근이 가능하다.
  • 실제 cluster 전체 설치가 필요한 경우 : cluster setup
• spark
  • http://why-not-learn-something.blogspot.kr/2015/06/spark-installation-pseudo.html
  • http://mbonaci.github.io/mbo-spark/
  • http://spark.apache.org/docs/latest/spark-standalone.html

  /root/spark-1.6.1-bin-without-hadoop/sbin/start-master.sh
  /root/spark-1.6.1-bin-without-hadoop/sbin/start-slave.sh spark://localhost:7077
  

  • spark master interface : http://192.168.99.100:8089/
  • wordcount example

  hadoop fs -put hadoop-2.7.2/etc/hadoop/ ./
  hadoop fs -move hadoop input
  spark-submit --master spark://localhost:7077 --total-executor-cores 2 --executor-memory 512M  wordcount.py -f input
  hadoop fs -ls ./output