kitchen-in-travis Cookbook

Proof of concept cookbook to run test-kitchen inside Travis CI using kitchen-docker in User Mode Linux.

You can use this in your cookbook by using a .travis.yml file similar to the following:

rvm:
- 2.2

sudo: true

before_script:
- source <(curl -sL https://raw.githubusercontent.com/zuazo/kitchen-in-travis/0.5.0/scripts/start_docker.sh)

script:
# Run test-kitchen with docker driver, for example:
- KITCHEN_LOCAL_YAML=.kitchen.docker.yml bundle exec kitchen test

Look below for more complete examples.

The following files will help you understand how this works:

• .travis.yml
• scripts/start_docker.sh: Starts Docker Engine inside Travis CI.
• .kitchen.docker.yml
• Rakefile

This example cookbook only installs nginx. It also includes some Serverspec tests to check everything is working correctly.

Related Projects

• kitchen-in-travis-native: Runs test-kitchen inside Travis CI using the native Docker service and kitchen-docker. The builds are faster (~2 mins to start), but a little less customizable.
• kitchen-in-circleci: Runs test-kitchen inside CircleCI.

Install the Requirements

First you need to install Docker.

Then you can use bundler to install the required ruby gems:

$ gem install bundle
$ bundle install

Running the Tests in Your Workstation

$ bundle exec rake

This example will run kitchen with Vagrant in your workstation. You can use $ bundle exec rake integration:docker to run kitchen with Docker, as in Travis CI.

Available Rake Tasks

$ bundle exec rake -T
rake integration:docker   # Run integration tests with kitchen-docker
rake integration:vagrant  # Run integration tests with kitchen-vagrant

How to Implement This in My Cookbook

First, create a .kitchen.docker.yml file with the platforms you want to test:

---
driver:
  name: docker

platforms:
- name: centos-6.6
  run_list:
- name: ubuntu-14.04
  run_list:
  - recipe[apt]
# [...]

If not defined, it will get the platforms from the main .kitchen.yml by default.

You can get the list of the platforms officially supported by Docker here.

Then, I recommend you to create a task in your Rakefile:

# Rakefile
require 'bundler/setup'

# [...]

desc 'Run Test Kitchen integration tests'
namespace :integration do
  desc 'Run integration tests with kitchen-docker'
  task :docker do
    require 'kitchen'
    Kitchen.logger = Kitchen.default_file_logger
    @loader = Kitchen::Loader::YAML.new(local_config: '.kitchen.docker.yml')
    Kitchen::Config.new(loader: @loader).instances.each do |instance|
      instance.test(:always)
    end
  end
end

This will allow us to use $ bundle exec rake integration:docker to run the tests.

The .travis.yml file example:

rvm:
- 2.0.0
- 2.1
- 2.2

sudo: true

before_script:
- source <(curl -sL https://raw.githubusercontent.com/zuazo/kitchen-in-travis/0.5.0/scripts/start_docker.sh)

script:
- travis_retry bundle exec rake integration:docker

If you are using a Gemfile, you can add the following to it:

# Gemfile

group :integration do
  gem 'test-kitchen', '~> 1.2'
end

group :docker do
  gem 'kitchen-docker', '~> 2.1.0'
end

This will be enough if you want to test only 2 or 3 platforms. If you want more, continue reading:

How to Run Tests in Many Platforms

Travis CI has a build time limitation of 50 minutes. If you want to test many platforms, you will need to split up the tests in multiple Travis CI builds. For those cases, I recommend you to use a Rakefile Rake task similar to the following:

# Rakefile
require 'bundler/setup'

# [...]

desc 'Run Test Kitchen integration tests'
namespace :integration do
  # Gets a collection of instances.
  #
  # @param regexp [String] regular expression to match against instance names.
  # @param config [Hash] configuration values for the `Kitchen::Config` class.
  # @return [Collection<Instance>] all instances.
  def kitchen_instances(regexp, config)
    instances = Kitchen::Config.new(config).instances
    return instances if regexp.nil? || regexp == 'all'
    instances.get_all(Regexp.new(regexp))
  end

  # Runs a test kitchen action against some instances.
  #
  # @param action [String] kitchen action to run (defaults to `'test'`).
  # @param regexp [String] regular expression to match against instance names.
  # @param loader_config [Hash] loader configuration options.
  # @return void
  def run_kitchen(action, regexp, loader_config = {})
    action = 'test' if action.nil?
    require 'kitchen'
    Kitchen.logger = Kitchen.default_file_logger
    config = { loader: Kitchen::Loader::YAML.new(loader_config) }
    kitchen_instances(regexp, config).each { |i| i.send(action) }
  end

  desc 'Run integration tests with kitchen-vagrant'
  task :vagrant, [:regexp, :action] do |_t, args|
    run_kitchen(args.action, args.regexp)
  end

  desc 'Run integration tests with kitchen-docker'
  task :docker, [:regexp, :action] do |_t, args|
    run_kitchen(args.action, args.regexp, local_config: '.kitchen.docker.yml')
  end
end

This will allow us to run different kitchen tests using the $ rake integration:docker[REGEXP] command.

Then, you can use the following .travis.yml file:

rvm:
- 2.0.0
- 2.1
- 2.2

sudo: true

env:
  matrix:
# Split up the test-kitchen run to avoid exceeding 50 minutes:
  - KITCHEN_REGEXP=centos
  - KITCHEN_REGEXP=debian
  - KITCHEN_REGEXP=ubuntu

before_script:
- source <(curl -sL https://raw.githubusercontent.com/zuazo/kitchen-in-travis/0.5.0/scripts/start_docker.sh)

script:
- travis_retry bundle exec rake integration:docker[$KITCHEN_REGEXP]

Test multiple kitchen instances concurrently per build

Sometimes it may be quicker to test multiple kitchen-docker instances in each Travis build due to a limit of concurrent builds (due to total Travis system load), rather than split out to a separate build for every kitchen-docker instance. The key is to find the balance between the number of kitchen-docker instances per Travis build.

Rakefile.concurrency provides rake tasks that can run actions against multiple test-kitchen instances simultaneously:

$ bundle exec rake --rakefile Rakefile.concurrency --tasks
rake integration:docker[regexp,action,concurrency]   # Run integration tests with kitchen-docker
rake integration:vagrant[regexp,action,concurrency]  # Run integration tests with kitchen-vagrant

Test multiple versions of Chef

If you'd like to test multiple versions of Chef, you can use a second environment variable in .travis.yml combined with the require_chef_omnibus property in .kitchen.yml.

# .travis.yml
# ...

env:
- KITCHEN_REGEXP=centos
- KITCHEN_REGEXP=centos CHEF_OMNIBUS_VERSION=12.10
- KITCHEN_REGEXP=centos CHEF_OMNIBUS_VERSION=12.8
- KITCHEN_REGEXP=ubuntu
- KITCHEN_REGEXP=ubuntu CHEF_OMNIBUS_VERSION=12.10
- KITCHEN_REGEXP=ubuntu CHEF_OMNIBUS_VERSION=12.8

# ...

# .kitchen.yml
# ...

provisioner:
  name: chef_zero
  require_chef_omnibus: <%= ENV.fetch('CHEF_OMNIBUS_VERSION', true) %>

# ...

When require_chef_omnibus is set to true, test-kitchen will build with the latest version of chef. Specifying a version in the form x.y as shown above will get the latest patch version of the minor version specified (~> x.y.0).

Real-world Examples

• netstat cookbook (.travis.yml, .kitchen.docker.yml, Rakefile): Runs kitchen tests against many platforms. Includes a minimal Serverspec test.

• opendkim cookbook (.travis.yml, .kitchen.docker.yml, Rakefile): Runs kitchen tests in different Travis builds separated by platform. Includes Serverspec tests.

• dhcp cookbook (.travis.yml, .kitchen.docker.yml, Rakefile): Runs kitchen tests in different Travis builds separated by distribution. Includes Serverspec and bats tests.

• onddo_proftpd cookbook (.travis.yml, .kitchen.docker.yml, Rakefile): Runs kitchen tests in 9 different Travis builds. Includes Serverspec tests.

Known Issues

The Test Cannot Exceed 50 Minutes

Each test can not take more than 50 minutes to run within Travis CI. It's recommended to split the kitchen run in multiple builds using the Travis CI build matrix.

Look at the examples in this documentation to learn how to avoid this.

Official CentOS 7 and Fedora Images

Cookbooks requiring systemd may not work correctly on CentOS 7 and Fedora containers. See Systemd removed in CentOS 7.

You can use alternative images that include systemd. These containers must run in privileged mode:

# .kitchen.docker.yml

# Non-official images with systemd
- name: centos-7
  driver_config:
    # https://registry.hub.docker.com/u/milcom/centos7-systemd/dockerfile/
    image: milcom/centos7-systemd
    privileged: true
- name: fedora
  driver_config:
    image: fedora/systemd-systemd
    privileged: true

Problems with Upstart in Ubuntu

Some cookbooks requiring Ubuntu Upstart may not work correctly.

You can use the official Ubuntu images with Upstart enabled:

# .kichen.docker.yml

- name: ubuntu-14.10
  run_list: recipe[apt]
  driver_config:
    image: ubuntu-upstart:14.10

Install netstat Package

It's recommended to install net-tools on some containers if you want to test listening ports with Serverspec. This is because some images come without netstat installed.

This is required for example for the following Serverspec test:

# test/integration/default/serverspec/default_spec.rb
describe port(80) do
  it { should be_listening }
end

You can ensure that netstat is properly installed running the netstat cookbook:

# .kitchen.docker.yml

- name: debian-6
 run_list:
 - recipe[apt]
 - recipe[netstat]

Travis CI Error: SSH session could not be established

Sometimes kitchen exits with the following error:

>>>>>> Converge failed on instance <default-debian-7>.
>>>>>> Please see .kitchen/logs/default-debian-7.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: SSH session could not be established
>>>>>> --------

If you get this error on Travis CI, avoid passing the --concurrency option to test-kitchen. It does not work in some cases. I recommend using the Travis CI build matrix to run multiple tests concurrently.

Travis CI Error: No output has been received in the last 10 minutes

If a command can take a long time to run and is very quiet, you may need to run it with some flags to increase verbosity such as: --verbose, --debug, --l debug, ...

Travis CI Error: Waiting Docker to Start Timeout (No output has been received in the last 10 minutes)

UML does not seem to work properly on some projects. The Travis build output in these cases:

$ source <(curl -sL https://raw.githubusercontent.com/zuazo/kitchen-in-travis/0.5.0/scripts/start_docker.sh)
[...]
Starting Docker Engine
Waiting Docker to start

No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.

Try using kitchen-in-travis-native or kitchen-in-circleci if you encounter this problem.

Travis CI Error: Waiting Docker to Start Timeout Debug Information

On some project builds SLIRP seems not to work correctly. This is the exact error in the UML boot process:

$ ip link set eth0 up
RTNETLINK answers: No such file or directory

But the interface exists:

eth0      Link encap:Serial Line IP
          inet addr:10.1.1.1  Mask:255.255.255.0
          NOARP  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:5

I have not found a way to fix the problem. Please, let me know if you find a solution.

Feedback Is Welcome

Currently I'm using this for my own projects. It may not work correctly in many cases. If you use this or a similar approach successfully with other cookbooks, please open an issue and let me know about your experience. Problems, discussions and ideas for improvement, of course, are also welcome.

Acknowledgements

See here.

License and Author

Author:	Xabier de Zuazo (xabier@zuazo.org)
Contributor:	Jacob McCann
Contributor:	Pedro Salgado
Contributor:	Austin Heiman
Copyright:	Copyright (c) 2015-2016, Xabier de Zuazo
License:	Apache License, Version 2.0

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.