-
Notifications
You must be signed in to change notification settings - Fork 10
Ansible server provisioning and deployment
Software requirements for the developer machine:
- Ubuntu 16.04
- Python>=3.5
- boto3>=1.5
- boto>=2.48
- pyyaml
- Ansible==2.8.6
git clone -b release https://github.com/metaspace2020/metaspace.git
cd metaspace/ansible/aws
git clone https://github.com/metaspace2020/metaspace-ansible-config.git
mv metaspace-ansible-config env
Put the master password into <METASPACE_HOME>/ansible/aws/vault_pass.txt
.
- Create IAM user with admin rights for SM engine.
- Create two buckets: one for the engine and one for the web application uploads. For the web application bucket make sure to allow PUT, POST, and DELETE requests from all hosts in the CORS configuration.
- Create a separate IAM user for the web application. Make sure this user have full access to the web application bucket.
- Create a pair of public/private keys with
ssh-keygen
. Upload the public key to the AWS Console. It will be used to ssh to instances. - Create AWS configuration and credential files for the admin IAM user on the developer manchine.
You will need at least three: main instance for the web app, database, RabbitMQ, and Elasticsearch; Spark master and Spark slave instances.
ansible-playbook -i env/dev aws_start.yml -e "components=all"
Provision the web services and Spark cluster instances
ansible-playbook -i env/dev provision/web.yml
ansible-playbook -i env/dev provision/spark.yml
ansible-playbook -i env/dev create_ami.yml
This step will take a while. Once the playbook is finished replace AMI ids for master ans slave instances with new ones in the dev/group_vars/all/vars.yml. New AMI ids can be found in the AWS Console.
After AMIs were successfully created stop Spark instances. They will be started automatically from new AMIs after a dataset is uploaded.
ansible-playbook -i env/dev aws_stop.yml -e "components=master,slave"
ansible-playbook -i env/dev deploy/web.yml
Start all instances
ansible-playbook -i env/dev aws_start.yml -e "components=all"
ansible-playbook -i env/dev aws_cluster_setup.yml
Deploy and start the web application and other services
ansible-playbook -i env/dev deploy/web.yml
To Stop all SM platform instances execute
ansible-playbook -i env/dev aws_stop.yml -e "components=all"