METRON-1509 Docs - metron-deployment/other-examples - Adding 0.4.1 AWS EC2 Single Node/Vagrant Method

metron-contrib/Metron-041SingleNodeEC2Vagrant/README.md

@@ -0,0 +1,165 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+Metron 0.4.1 - AWS EC2 Single Node Deployment with Vagrant 
+==================
+
+This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized node in AWS EC2.  
+
+#### What is this good for?
+
+* If you are new to Metron and want to explore the functionality that it offers, this is good place to start.  
+
+* If you are a developer contributing to the Apache Metron project, this is also a great way to test your changes.  
+
+* The single node will survive a reboot.
+
+* The single node can use a pre-existing AWS elastic ip, security group id, and subnet id.
+
+#### What is this **not** good for?
+
+* This single node is **not** intended for processing anything beyond the most basic, low volume work loads. For high volumes your experience might be poor you are not using least an m5.4xlarge instance for a single node.
+
+* Additional services should **not** be installed along side Metron in this VM.
+
+* This single node should **not** be used to run a proof-of-concept for Apache Metron within your organization.
+
+* You might need to run the Vagrant file in Mac OS and have install prerequisites installed properly 
+
+Running Metron within the resource constraints of a single VM is incredibly challenging. Failing to respect this warning, will cause various services to fail mysteriously as the system runs into memory and processing limits.
+
+
+Getting Started
+---------------
+### Amazon Web Services
+
+If you already have an Amazon Web Services account that you have used to deploy EC2 hosts, then you should be able to skip the next few steps.
+
+1. Head over to [Amazon Web Services](http://aws.amazon.com/) and create an account.  As part of the account creation process you will need to provide a credit card to cover any charges that may apply.
+
+2. Create a set of user credentials through [Amazon's Identity and Access Management (IAM) ](https://console.aws.amazon.com/iam/) dashboard.  On the IAM dashboard menu click "Users" and then "Create New User". Provide a name and ensure that "Generate an access key for each user" remains checked.  Download the credentials and keep them for later use.
+
+3.  While still in [Amazon's Identity and Access Management (IAM) ](https://console.aws.amazon.com/iam/) dashboard, click on the user that was previously created.  Click the "Permissions" tab and then the "Attach Policy" button.  Attach the following policies to the user.
+
+  - AmazonEC2FullAccess
+  - AmazonVPCFullAccess
+
+
+
+### Prerequisites
+Apache Metron uses the [official, open source CentOS 6](https://aws.amazon.com/marketplace/pp/B00NQAYLWO) Amazon Machine Image (AMI).  If you have never used this AMI before then you will need to accept Amazon's terms and conditions. Navigate to the [web page for this AMI](https://aws.amazon.com/marketplace/pp/B00NQAYLWO) and "Accept Software Terms" for the "Manual Launch" tab.
+
+The computer used to deploy Apache Metron will need to have the following components installed.
+
+ - [Ansible](https://github.com/ansible/ansible) (2.2.2.0)
+ - [Docker](https://www.docker.com/community-edition)
+ - [Vagrant](https://www.vagrantup.com) 1.8+
+ - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager)
+ - [Virtualbox](https://virtualbox.org) 5.0+
+ - Python 2.7
+ - Maven 3.3.9
+ - C++11 compliant compiler, like [GCC](https://gcc.gnu.org/projects/cxx-status.html#cxx11)
+
+
+Running the following script can help validate whether you have all the prerequisites installed and running correctly.
+
+  ```
+  metron-deployment/scripts/platform-info.sh
+  ```
+
+#### How do I install these on MacOS?
+
+1. Download the [apache-metron-0.4.2.tar.gz](https://archive.apache.org/dist/metron/0.4.2/) and decompress the file. 
+
+2. Copy this custom [Vagrantfile](./Vagrantfile) file to the directory below (replace the existing Vagrantfile that is in the directory)
+```
+/../metron-deployment/vagrant/full-dev-platform/
+```
+
+3. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/).
+
+4. Run the following commands in a terminal to install all of the required tools.
+
+    ```  
+    brew cask install vagrant virtualbox docker
+    brew cask install caskroom/versions/java8
+    brew install maven@3.3 git
+    pip install ansible==2.2.2.0
+    vagrant plugin install vagrant-hostmanager
+    vagrant plugin install vagrant-aws
+    vagrant plugin install vagrant-reload
+    pip install --upgrade setuptools --user python
+    open /Applications/Docker.app
+    ```
+5. In your AWS console you need to reserve/create an AWS Elastic IP, a Subnet id, Security Group id, and a key pair (key pair name & *.pem file[remember to set permissions to chmod 400]). Take a note of these. These will be used later. 
+
+6. Update the macOS /etc/hosts file to map node1 to the AWS Elastic IP that was defined in step 5. 
+```
+127.0.0.1       localhost
+.
+.
+.
+<AWS_ELASTIC_IP>       node1
+```
+
+7. The following will clear existing Vagrant, Docker & Maven builds and deployments. WARNING - THESE STEPS WILL DISTROY ALL LOCAL DOCKER CONTAINERS AND VAGRANT BOXES. It will also create a new Vagrant box.
+```
+vagrant halt node1 -f
+vagrant halt default -f
+vagrant destroy node1 -f
+vagrant destroy default -f
+for i in `vagrant global-status | grep virtualbox | awk '{print $1 }'` ; do vagrant destroy $i  ; done
+vagrant global-status --prune
+docker rm $(docker ps -aq)
+osascript -e 'quit app "Docker"'
+open -a Docker
+rm -rf /../.m2/repository/*
+rm -rf /../.vagrant.d/boxes/*
+vagrant box add dummy --force https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
+```
+8. Associate your AWS ids and key information to the following enviroment variables (enter values inside the quotes).
+```
+export AWS_ACCESS_KEY_ID=''
+export AWS_SECRET_ACCESS_KEY=''
+export AWS_KEYNAME=''
+export AWS_KEYPATH='../..*.pem'
+export AWS_ELASTIC_IP=""
+export AWS_SECURITYGROUP_ID=""
+export AWS_SUBNET_ID=""
+```
+9. Update your local /etc/hosts file to include AWS_ELASTIC_IP which will be tied to "node1" (do only once).
+```
+sed -i "$AWS_ELASTIC_IP  node1" /etc/hosts
+```
+
+### Deploy Metron
+
+1. Ensure that the Docker service is running.
+
+2. Deploy Metron
+
+    ```
+    cd metron-0.4.1/metron-deployment/vagrant/full-dev-platform
+    vagrant up --provider=aws
+    ```
+
+### Explore Metron
+
+Navigate to the following resources to explore your newly minted Apache Metron environment.
+
+* [Ambari] http://node1:8080
+* [Kibana] http://node1:5000

metron-contrib/Metron-041SingleNodeEC2Vagrant/Vagrantfile

@@ -0,0 +1,195 @@
+#Global Cybersecurity Resource 2017-2018 (cugcr.ca)
+#
+##########################################################################
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#a  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+##########################################################################
+#ABOUT:
+#-------------------
+#This Vagrant file will deploy Apache Metron as a single node instance (node1) to AWS EC2.
+#
+#This file was originally adapted from the Metron Single Node Install Vagrant file located here:
+#https://cwiki.apache.org/confluence/display/METRON/Dev+VM+Install
+#https://github.com/apache/metron/blob/master/metron-deployment/vagrant/full-dev-platform/Vagrantfile
+##########################################################################
+#LOCATION OF FILE:
+#-------------------
+#This file should be located in the following directory:
+#/../metron-deployment/vagrant/full-dev-platform
+##########################################################################
+#REQUIREMENTS(What is needed to run this file):
+#-------------------
+#install: vagrant plugin install vagrant-aws
+#install: vagrant plugin install vagrant-reload
+#install: vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
+#install: pip install --upgrade setuptools --user python
+#
+#You will also need to reserve an AWS Elastic IP and have a Subnet id and Security Group id
+#In the code below you will see "<_comment_>" letting you know where to place the ids
+#You will also need to update the macOS /etc/hosts file to map node1 to the AWS Elastic IP (i.e. node1     x.x.x.x)
+##########################################################################
+#BEFORE RUNNING:
+#Recommended steps before running file (clearing vagrant, docker, ext..):
+#-------------------
+#vagrant halt node1 -f
+#vagrant halt default -f
+#vagrant destroy node1 -f
+#vagrant destroy default -f
+#for i in `vagrant global-status | grep virtualbox | awk '{print $1 }'` ; do vagrant destroy $i  ; done
+#vagrant global-status --prune
+#docker rm $(docker ps -aq)
+#osascript -e 'quit app "Docker"'
+#open -a Docker
+#rm -rf /../.m2/repository/*
+#rm -rf /../.vagrant.d/boxes/*
+#vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
+##########################################################################
+#HOW TO EXECUTE THIS FILE:
+#-------------------
+#Set Environment Variables & Run Vagrant
+#-------------------
+#export AWS_ACCESS_KEY_ID=''
+#export AWS_SECRET_ACCESS_KEY=''
+#export AWS_KEYNAME=''
+#export AWS_KEYPATH='../..*.pem'
+#
+#export AWS_ELASTICIP=""
+#export AWS_SECURITYGROUPID=""
+#export AWS_SUBNETID=""
+#
+#vagrant up --provider=aws
+##########################################################################
+#OTHER NOTES:
+#-------------------
+#How to find Canadian Centos7 AMI ids:
+#run: pip install awscli
+#run: aws --region ca-central-1 ec2 describe-images --owners aws-marketplace --filters Name=product-code,Values=aw0evgkw8e5c1q413zgy5pjce
+#imageIDs: ami-28823e4c, ami-af62d0cb, ami-cbd069af
+
+require 'getoptlong'
+require 'vagrant-aws'
+require 'vagrant-reload'
+
+ansibleTags=''
+ansibleSkipTags='sensors,quick_dev'
+
+begin
+   opts = GetoptLong.new(
+     [ '--ansible-tags', GetoptLong::OPTIONAL_ARGUMENT ],
+     [ '--ansible-skip-tags', GetoptLong::OPTIONAL_ARGUMENT ]
+   )
+
+   opts.quiet = TRUE
+
+   opts.each do |opt, arg|
+     case opt
+       when '--ansible-tags'
+         ansibleTags=arg
+       when '--ansible-skip-tags'
+         ansibleSkipTags=arg
+     end
+   end
+rescue Exception => ignored
+#Ignore to allow other opts to be passed to Vagrant
+end
+
+puts " Running with ansible-tags: " + ansibleTags.split(",").to_s if ansibleTags != ''
+puts " Running with ansible-skip-tags: " + ansibleSkipTags.split(",").to_s if ansibleSkipTags != ''
+
+hosts = [{
+    hostname: "node1",
+    ip: ENV['AWS_ELASTICIP'],
+    memory: "6000",
+    cpus: 4,
+    promisc: 2  # enables promisc on the 'Nth' network interface
+}]
+
+Vagrant.configure(2) do |config|
+ config.vm.provider :aws do |aws, override|
+   aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']
+   aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
+   aws.keypair_name = ENV['AWS_KEYNAME']
+
+   #Below is a CentOS ami id available in Canada
+   aws.ami = "ami-00e45864"
+   aws.region = "ca-central-1"
+   aws.instance_type = "t2.2xlarge"
+
+   #Example of a Security Group id: sg-1193xxxx
+   aws.security_groups = ENV['AWS_SECURITYGROUPID']
+
+   #Example of Subnet id: subnet-fbc3exxxx
+   aws.subnet_id = ENV['AWS_SUBNETID']
+   aws.elastic_ip = ENV['AWS_ELASTICIP']
+   override.vm.box = "dummy"
+   override.ssh.username = "centos"
+   override.ssh.private_key_path = ENV['AWS_KEYPATH']
+
+  #config.vm.provider "aws" do |aws, override|
+   override.nfs.functional = false
+   aws.tags = {
+         'Name' => 'Phase1.5-MetronA-TESTING',
+         'Organization' => 'GCR',
+         'Developer' => 'GCR Dev Team',
+         'Function' => 'MetronInstall'
+   }
+   aws.block_device_mapping = [{'DeviceName' =>'/dev/sda1', 'Ebs.VolumeSize' => 100, 'Ebs.DeleteOnTermination' => true}]
+ end
+
+ #The following will install rsyslog, change hostname to node1, resize the disk partition, and reboot
+ config.vm.provision "shell", inline: <<-SHELL
+   yum install -y wget
+   yum install yum-utils
+   #wget http://rpms.adiscon.com/v8-stable/rsyslog.repo
+   #mv rsyslog.repo /etc/yum.repos.d/rsyslog.repo
+   #yum info rsyslog --skip-broken
+   #yum install -y rsyslog
+   yum-config-manager --add-repo http://rpms.adiscon.com/v8-stable/epel-6/x86_64
+   yum install --nogpg -y rsyslog rsyslog-kafka
+   rm /etc/rsyslog.d/listen.conf
+   rsyslogd -N1 && rsyslogd
+   hostname node1
+   sed -i "s/^HOSTNAME=.*/HOSTNAME=node1/g" /etc/sysconfig/network
+   echo -e "u s\nd 1\nn\np\n1\n2048\n\na\n1\nw\n" | fdisk /dev/xvda
+   ip link set eth0 promisc on
+   ifconfig eth0 promisc
+   #(echo u s; echo d 1; echo n; echo p; echo 1; echo 2048 ; echo ;echo a; echo 1; echo w) | fdisk {{ vol_src_1 }} || true
+   #
+   #INSTALL NIFI
+   #-------------------
+   #sudo -i
+   #cd /opt
+   #wget https://archive.apache.org/dist/nifi/1.2.0/nifi-1.2.0-bin.tar.gz
+   #tar xf nifi-1.2.0-bin.tar.gz
+   #note: modify nifi-1.1.2/conf/nifi.properties to change "nifi.web.http.port" to port 8089
+   #sed -i "s/^nifi\.web\.http\.port=.*/nifi\.web\.http\.port=8089/g" /opt/nifi-1.2.0/conf/nifi.properties
+   #/opt/nifi-1.2.0/bin/nifi.sh install
+   #/opt/nifi-1.2.0/bin/nifi.sh stop
+   #/opt/nifi-1.2.0/bin/nifi.sh start
+   shutdown -r now
+ SHELL
+
+ config.vm.provision :reload
+
+ config.vm.provision :ansible do |ansible|
+   ansible.playbook = "../../playbooks/metron_full_install.yml"
+   ansible.host_key_checking = false
+   ansible.limit = 'all'
+   ansible.sudo = true
+   ansible.tags = ansibleTags.split(",") if ansibleTags != ''
+   ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
+   ansible.inventory_path = "../../inventory/full-dev-platform"
+ end
+end

metron-deployment/other-examples/README.md

@@ -16,6 +16,15 @@ See the License for the specific language governing permissions and
 limitations under the License.
 -->
 # Other Example Deployments
-This folder contains other example deployments which are not covered under the automatic installs.
-These examples have been vetted and tested by the Metron community at the time of entry. However, development moves fast and things may change. If there are problems with these steps, please check the mailing lists first or log a Jira ticket.
-- [Metron 0.4.0 on CentOS 6 Manual Install](manual-install/Manual_Install_CentOS6.md) (July 2017)
+This page contains a list other example deployments methods that might not be actively maintained in the most recent build. Keep in mind that development moves fast and things may change.
+
+## Maintained by Apache Metron
+These examples have been vetted and tested by the Metron community at the time of entry. If there are problems with these steps, please check the mailing lists first or log a Jira ticket.
+
+[Metron 0.4.0 on CentOS 6 Manual Install](manual-install/Manual_Install_CentOS6.md) (July 2017)
+
+## External Resources
+The following are external community contributions that are not actively maintained by the Apache Metron project. However, you might find them helpful. If there are any issues do **NOT** log a JIRA. Reach out to the mailing list or the contributor first.
+
+
+[Metron 0.4.1 - AWS EC2 Single Node Deployment with Vagrant](/metron-contrib/Metron-041SingleNodeEC2Vagrant)     |  Global Cybersecurity Resource (March 2018)