Enhancement: Test with TestInfra and Molecule

[samrocketman]

I recommend taking away the dependency of Ruby and testing this role using Molecule and TestInfra. Molecule has a docker driver which works with with Travis CI (just like Vagrant has a docker driver). Molecule uses TestInfra and the VirtualBox/Vagrant driver by default.

I maintain the GIMP project CI infrastructure and have documented how I get started with Molecule when creating new roles.

[samrocketman]

If the dev-sec team is open to this idea I'm willing to create a PR with suggested changes. This assumes #127 is an accepted idea because all of the technologies discussed in this issue are installed via Python pip.

[rndmh3ro]

We're using inspec for all our tests (see https://github.com/dev-sec/linux-baseline/) and since we use these for the chef, puppet and ansible roles, its unlikely that we will replace them with TestInfra.

However some (long) time ago I took a look at molecule and found it quite good. However there's no support for inspec so it's out of the question right now. However @chris-rock thought about writing a driver for molecule to support inspec.

So if that happens, we'll see gladly take a PR that implements molecule testing.

[chris-rock]

@samrocketman Thank you very much for your open feedback. I'd like to understand your concerns a little bit more about. Could you elaborate on:

• Why is Ruby as a dependency for testing an issue (it is not a runtime dependency)?
• How is the workflow of Molecule different from test-kitchen?
• Why do you think testinfra is better then InSpec?

[samrocketman]

Why is Ruby as a dependency for testing an issue (it is not a runtime dependency)?

It's not really a concern. It just simplifies setting up a development and test environment with fewer dependencies. For example, really the only dependencies you need installed is Python, pip, and virtualenv. The rest of the dependencies would get installed via requirements.txt (akin to Ruby Gemfile.lock).

How is the workflow of Molecule different from test-kitchen?

I have only a light familiarity with test kitchen. However, conceptually they're similar. The workflow should be somewhat the same.

1. Develop ansible task and the test.
2. Run molecule test which in one command: provisions, installs ansible if missing, runs the playbook, evaluates the result with testinfra tests, reports the results. I'm not sure if deprovisioning happens before or after reporting.

Why do you think testinfra is better then InSpec?

Because Ansible, molecule, and TestInfra are all tracked by python, it's easy to track them all for repeatability within requirements.txt.

I would say that's the primary advantage but it's not a big one. You can achieve the same level of repeatability with Gemfile.lock. It's just a fewer set of dependencies.

Keep in mind I'm not suggesting test kitchen and inspec are inferior. They both can achieve the same thing with success. I'm only suggesting simplifying your workflow with fewer dependencie and a tool specifically designed with ansible in mind.

[samrocketman]

Here's an example with the dependency chain I'm recommending. https://github.com/Comcast/ansible-sdkman/blob/master/.travis.yml

[elliotweiser]

Building on @samrocketman's point, a nice alternative testing framework (similar to testinfra or serverspec) is GOSS. It's super lightweight, tests are easy to develop (just a YAML file), and it's REALLY fast (much faster in my experience than both serverspec and testinfra).

[rndmh3ro]

Closing this for now. If anyone wants to support other testing-methods, feel free to reopen and provide a PR.