Create a test suite

[ryanlovett]

BCE needs to have an automated test suite to catch release bugs. It should focus on the functioning of important applications and the desktop environment.

[ck37]

Also we are getting a fair number of failed package installs it looks like (I'll submit em as separate issues). Confirmation that all desired packages were installed would be a helpful test case, or maybe separate cases for apt-get, R, and python.

[ck37]

Anyone know what framework we should use for EC2 vs. Virtualbox?

[ck37]

Looks like perhaps ServerSpec (http://serverspec.org/) may be the way to go:

• http://code.hootsuite.com/build-test-and-automate-server-image-creation/
• https://github.com/garethr/packer-serverspec-example

But maybe Test Kitchen (or both?):

• https://medium.com/@pereiragoncalo/continuos-integration-and-delivery-with-packer-chef-aws-d29453d93382
• http://kitchen.ci/docs/getting-started/writing-server-test
• http://www.techinsight.io/review/devops-and-automation/testing-infrastructure-with-test-kitchen/

[ck37]

Ok, in discussions with Ryan, Chris, & Aaron we are interested in:

• Adding a test directory that runs at the end of the image bootstrap.
• Test key packages (apt-get, R, python) and set build status to failed if tests fail.
• But don't terminate the packer build: create the image but let the builder know which tests failed.

[ck37]

Or maybe instead of a test directory/script, we simply integrate validation within each script in the bootstrap.d directory.