Skip to content

phamthuonghai/ansible-spark-ec2

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Ansible: Spark standalone cluster on Amazon EC2

How to use

  • Install Ansible
sudo apt-get install software-properties-common
sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible==1.9.4
  • Install python boto package
sudo apt-get install python-pip
sudo pip install boto
  • Put your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to ~/.boto file:
[profile DSPlatform]
aws_access_key_id = ABCDEFGHIJKLMNOPQRST
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • Check your access to AWS
./inventory/ec2.py --list
region: us-west-2
instance_type: m4.large
slave_count: 2
boto_profile: DSPlatform
remote_user: ubuntu

spark_download_url: http://d3kbcqa49mib13.cloudfront.net/spark-1.5.2-bin-hadoop2.6.tgz
spark_version: 1.5.2-bin-hadoop2.6
spark_root: /opt/spark

hadoop_download_url: http://www-us.apache.org/dist/hadoop/common/hadoop-2.6.4/hadoop-2.6.4.tar.gz
hadoop_version: 2.6.4
hadoop_root: /opt/hadoop
  • Run the playbook
ansible-playbook ds_platform.yaml 

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published