Consolidate host files

[fullyint]

Trellis could use a single host file instead of separate host files per environment. A single file may be cleaner conceptually and technically. It would also accommodate drop-in playbooks that need to access hosts from multiple environments at the same time (e.g., a playbook that syncs the db from a staging host to a production host).

Example Host File

	Web hosts	DB hosts
Staging	webhost1	dbhost1
Production	webhost2, webhost3	dbhost2

The hosts above could translate to the host file below.

[staging]
webhost1
dbhost1

[production]
webhost2
webhost3
dbhost2

[web]
webhost1
webhost2
webhost3

[db]
dbhost1
dbhost2

Revised Commands

• ansible-playbook server.yml -e env=staging (or env=production)
• ansible-playbook dev.yml (see note about "Development Host File" below)

Changes that Users would Face

• Transfer hosts to new host file, a one-time task per Trellis install.
• Remember to use revised commands
• Can run ansible-playbook dev.yml if desired over vagrant up or vagrant provision. This makes it easier to add the --tags flag to run only portions of the playbook. (see note about "Development Host File" below)

Edits to Trellis

ansible.cfg

• Add inventory = hosts to the [defaults]. This hosts is a directory like the current hosts directory.

hosts directory

• (Edit: This hosts directory idea is revised in later comments.)
• The hosts directory would contain an inventory file named non-dev (or something). The file content would be like the example host file above.
• The hosts directory could contain something to make Ansible aware of the development host defined at .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory. This "something" could be a relative symlink, a copy of the vagrant host file, etc. (see note about "Development Host File" below)

hosts parameter in playbooks

• hosts: web:&development in dev.yml
• hosts: web:&{{ env }} in server.yml and deploy.yml

If a user were to run the command ansible-playbook server.yml -e env=staging

This would mean the hosts must be in the group webservers and the host must also be in the group staging

• http://docs.ansible.com/ansible/intro_patterns.html

deploy.sh

• adjust to accommodate env extra-var

Development Host File

A primary benefit of consolidating host files is for Ansible to be simultaneously aware of the development host and other hosts (e.g., for a playbook syncing the DB between development and staging/production). However, a challenge arises due to the fact that vagrant auto-corrects port collisions for the dev machine.

For example, after vagrant provision, you might see this:
==> default: Fixed port collision for 22 => 2222. Now on port 2200.
This means that the development host can't be manually "hard-coded" like the staging and production hosts (if your dev host in the host file had ansible_ssh_port=2222, it now needs ansible_ssh_port=2200 after the collision).

I could use some advice on how to get the host file to pick up on this dynamic port value.

• (Edit: Later comments propose a better idea of tapping into vagrant ssh-config.)
• Relative symlink committed to repo. We could commit a relative symlink to the Trellis repo, such that hosts/vagrant_ansible_inventory is a symlink pointing to .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory. Presumably the symlink will work for users whose OSes handle symlinks. Windows users should be ok because they'd be running Trellis on their vagrant/ubuntu VM. But there may be perils with committing symlinks to a repo. I just don't know.
• Users manually create symlink if they want it. Maybe we should not commit the symlink to the repo so as to avoid asserting that the symlink will work for everyone. Instead, we could just mention that users could create a symlink if they wish.
• Users copy dev host file into hosts directory if they want it. This option seems inferior to a symlink because the ansible_ssh_port could change, requiring re-copy.

Hopefully someone out there has a better idea. I wish the main host file could use an include directive like
include = .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory

Feedback Requested

• Would consolidating host files be worth the "Changes that Users would Face" and "Edits to Trellis"?
• (Edit: Later comments address how development hosts could be included automatically.) Would it still be worth it if the consolidation doesn't automatically include the vagrant development host (i.e., if users have to manually create a symlink to the "Development Host File", etc.)?
• (Edit: Later comments show how using vagrant ssh-config is likely superior to using a symlink.) Any better ideas than a symlink for how to simultaneously define the non-dev hosts and the dev host with its dynamic ansible_ssh_port?

[nathanielks]

Could you outline a few more benefits?

[nathanielks]

And/or, more specifically, what problem(s) does this solve?

[louim]

Totally in for this! @nathanielks, as @fullyint said, the main benefit is that it will be easy to write cross environment scripts, like database and file sync as we were able to do with capistranowp-cli. Even this alone is enough to warrant the change IMO. Other benefits include shortened ansible-playbook command, the possibility of using options like -vvvv without having to change the vagrantfile, and running specific roles (via --tags) on the VM.

@fullyint have you looked into http://docs.vagrantup.com/v2/provisioning/ansible.html, specifically the Static Inventory section? If it work (I think we will lose the ansible_ssh_port dynamic feature), it would allow to use only one inventory file, because with 2 inventory file, we are still unable to run playbook from live environment to the VM.

If it's impossible to have only one inventory file, would go for the symlink. Git supports it without problem, and I can't imagine an OS where symlinks wouldn't work. I would name the files local and remote in that case. However, I think it doesn't do much more than the original 3 separates inventory files.

[louim]

I checked a couple of things to better understand what you wanted to do @fullyint. Here are my opinion (after research).

Things I like in this:

• Using the host directory and putting in inside the ansible.cfg
• The possibility of running cross environments playbooks (more on that later)
• refactor the host grouping

Things i would leave like they are:

• Keeping one file per environment. I think it makes it easier to understand and more structured.

I tested the symlink locally. The concept work, but other files will need tweaking to make it functional.

In the remote-user role, inventory_hostname is used. This will fail currently because the VM has the inventory hostname of default. see below for result of the command:

ansible-playbook server.yml -i hosts/development -v                                                                                                     ⏎ master ✱

PLAY [Determine Remote User] **************************************************

TASK: [remote-user | Determine whether to connect as root or admin_user] ******
ok: [default -> 127.0.0.1] => {"changed": false, "cmd": "ssh -o PasswordAuthentication=no root@default \"echo root\" || echo admin", "delta": "0:00:00.011059", "end": "2015-08-04 00:46:52.094944", "rc": 0, "start": "2015-08-04 00:46:52.083885", "stderr": "ssh: Could not resolve hostname default: nodename nor servname provided, or not known", "stdout": "admin", "stdout_lines": ["admin"], "warnings": []}

TASK: [remote-user | Set remote user for each host] ***************************
ok: [default] => {"ansible_facts": {"ansible_ssh_user": "admin"}}

PLAY [WordPress Server - Install LEMP Stack with PHP 5.6 and MariaDB MySQL] ***

GATHERING FACTS ***************************************************************
fatal: [default] => SSH Error: Permission denied (publickey).
    while connecting to 127.0.0.1:2222
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.

TASK: [common | Validate Ansible version] *************************************
FATAL: no hosts matched or all hosts have already failed -- aborting


PLAY RECAP ********************************************************************
           to retry, use: --limit @/Users/louim/server.retry

default                    : ok=2    changed=0    unreachable=1    failed=0

admin_user: vagrant should also be set in the development file and the ansible.group set correctly to to make this work. (the example below is using the current syntax).

       ansible.playbook = File.join(ANSIBLE_PATH, 'dev.yml')
       ansible.groups = {
         'web' => ['default'],
-        'development' => ['default']
+        'development:children' => ['web']
       }

@fullyint check https://github.com/louim/bedrock-ansible/commit/4b4ff3365b254eaa941c37febf3fbc673f8b5c51 to see what I had to change. Still needed to implement is the groups refactor that will be needed to use a host folder.

[fullyint]

@nathanielks Thank you for looking this over and helping me think it through.

@louim Thank you so much for reviewing this and for explaining the benefits so well. I'll emphasize one you already mentioned. Currently, users can only access development hosts via vagrant commands, or by creating/editing host files. The proposed changes make it possible to access any host, using the default host file and Ansible commands.

Abandoned Approaches

Symlinks. When I vagrant up two different Vagrant VMs from different Trellis projects, each has the Trellis default IP 192.168.50.5. The first VM has port 22 => 2222 and the second has port 22 => 2200. The .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory shows ansible_ssh_port=2222 for both VMs, so a symlinked host file won't resolve the dynamic port issue. One inventory should show port 2200, but shows incorrect port 2222. Edit: When vagrant runs the Ansible provisioner, it will update the port values in the inventory file. The problem is this: If vm A is running, and you vagrant up a halted vm B, this vm B may have its port reassigned. However, vm B's port won't be updated in the inventory file because vagrant doesn't run the Ansible provisioner on a vagrant up for a machine that has already been provisioned. So, we can't just rely on the vagrant_ansible_inventory. We must go get the latest ssh-config info each time we want to run ansible-playbook on a dev vm directly (vs indirectly via vagrant provision).

Omitting port from host file. @nathanielks and @austinpray helped me understand that if/when there is only one VM, there is no need to specify the port. An SSH connection is still possible to example.dev because the /etc/hosts connects this domain to 192.168.50.5 and the default port 22 will forward fine. In my basic testing, the SSH connection became unreliable when I added a second VM on the same IP, with different domain name and port forwarding. We could tell users who want multiple VMs to change the config.vm.network :private_network, ip: '192.168.50.5' in the Vagrantfile, but I'd rather not require users to change anything.

New Approach: Create SSH Config from vagrant ssh-config

I decided to grab all ssh config info provided by the command vagrant ssh-config (initial inspiration). In addition to capturing the correct dynamic port, it also includes other useful settings like UserKnownHostsFile /dev/null and StrictHostKeyChecking no, which seem acceptable for the local vm. Users wanting these will no longer have to add them manually to their ~/.ssh/config.

I made a preliminary consolidate-hosts branch I'll use for testing. It demonstrates

• a single host file named hosts
• modified hosts parameters for playbooks
• ansible.cfg specifying default hosts file
• ansible.cfg adds a ssh-config file for dev hosts (in ssh_args)
• new ssh-config-dev role populates ssh config file for dev hosts. It isn't meant to be touched by users, so it is tucked away at roles/ssh-config-dev/files/config (but whatever)

Alternatives I Turned Down

• Too bad it's not possible to specify ansible_ssh_config nor ssh_args by host or group.
• ANSIBLE_SSH_ARGS can only be set globally, not using the environment keyword.
• There are ideas out there to cat multiple ssh config files, ruby gems to assemble files in .ssh/config.d/, and ansible galaxy roles to build an ssh config file from your inventory, but I think the ssh-config-dev role requires less from users and doesn't mess with their existing ~/.ssh/config.

Review: How this New Approach Affects Users

• This approach has all the benefits described
• The negative is still that existing users have the one-time task of transfering hosts to new hosts file
• The slightly negative is that users have to remember to switch their server.yml command:

- ansible-playbook server.yml -i hosts/staging
+ ansible-playbook server.yml -e env=staging

Hosts File vs. Hosts Directory

My test branch uses a single file named hosts. There are few enough hosts listed that it still looks very manageable, and I like seeing all hosts at once. If prefered, hosts could be a directory, either with that single file, or separate files per environment.

@nathanielks pointed out to me that one traditional argument for separate host files per environment is that Ansible would only ever know about the hosts in the host file specified, avoiding risk of changes in one environment accidentally spilling over to another. This proposed consolidation makes all hosts available, if desired, but the existing playbooks isolate hosts via the hosts parameter in each playbook.

For example, dev.yml has hosts: web:&development indicating that it can only ever run on a host that appears in both the web and development groups (no environment spillage). Similarly, server.yml and deploy.yml have hosts: web:&{{ env }}, so they can only ever run on the env the user specifies. There is no default for env in this case, so if the user fails to specify env from the CLI, the playbook will error and halt, affecting zero environments.

Web servers vs. DB Servers

At present, the Trellis default has the web server and db server on a single box. So, for now, the host file could omit the [db] group in the example above, or just list dummy example hosts for [db] (like I did in my test branch). Although at present the hosts parameter in the playbooks could omit the web group (e.g., dev.yml could have hosts: development), keeping web in there (e.g., hosts: web:&development) could make Trellis more easily extensible for users wanting to incorporate db servers.

Other Notes

@louim thanks for pointing out the Ansible static inventory section in the vagrant docs. I hadn't remembered that. Perhaps it will come in handy in all of this.

I also liked your idea for naming the separate host files local and remote. Of course! Why didn't I think of that? In this case, I think we can get away with a single host file however. With these updated thoughts, do you have preference for a single file or separate files per environment?

A number of your comments addressed running server.yml on a development host. I would have only expected server.yml to be run on staging or production hosts, and only dev.yml on development hosts. Thank you for pointing out that if we decide to accommodate development hosts with server.yml, we may need to do things like

• accommodate port in "Determine whether to connect as root or admin_user" task
• add admin_user: vagrant to group_vars/development
• maybe more

[swalkinshaw]

Closed via #313