Initial integration testing environment

[ncoghlan]

• expected behaviours defined with behave and PyHamcrest
• configures local VMs with Vagrant and Ansible
• uses Leapp to run macrocontainer redeployments
  and query local IP addreses
• uses requests to check web service behaviour
• uses pipenv to manage the test environment itself

Known issues:

• VM IP addresses are not being retrieved correctly
• relies on the SSH kludge in leapp-tool to access the VMs
• even the simple baseline scenario takes ~5 minutes to run (~200s for VM setup, ~100s for the migration)

[ncoghlan]

The core idea here is that even though this is only a prototype, we still want to be able to clearly define:

• migration scenarios we already know work
• migration scenarios we know don't work
• migration scenarios we haven't even checked yet

That kind of scenario based testing is where BDD frameworks like behave excel (since they're designed as "scenario description" DSLs), and with Ansible already in the mix to configure the VMs after they're created from the base images, it makes sense to also rely on that to create the VMs in the first place.

[ncoghlan]

@shaded-enmity ^^^

[ncoghlan]

Updated the PR description to reflect today's updates - these are real tests now, although I had to modify the current SSH kludge in leapp-tool to get the migration step to work.

[ncoghlan]

The tests are currently failing, and I suspect the problem is that leapp-tool isn't handling the migration correctly, but is still reporting a return code of 0. This is based on setting up the VMs and then running the migration command directly:

$ sudo /usr/bin/python2 leapp-tool.py migrate-machine -t centos7-target centos6-guest-httpd
! looking up "centos6-guest-httpd" as source and "centos7-target" as target
! obtaining SSH keys
! copying over
! provisioning ...
Unable to find image 'centos:6' locally
Trying to pull repository docker.io/library/centos ... 
Pulling repository docker.io/library/centos
/usr/bin/docker-current: Network timed out while trying to connect to https://index.docker.io/v1/repositories/library/centos/images. You may want to check your internet connection or if you are behind a proxy..
See '/usr/bin/docker-current run --help'.
! starting services
Error response from daemon: No such container: container
! done

$ echo $?
0

[ncoghlan]

This also shows that getting to a point where we can keep the target VM around between runs should give a decent performance boost - not only will it save setup time on the VM itself, but we'll be able to retain a Docker image cache inside the VM.

[ncoghlan]

Current status:

• I realised the demo migration deletes the container on the target VM, so I dropped the ensure_fresh requirement for the target VM as well
• I adjusted the migration script to fail if the container on the target VM doesn't start
• port 80 is exposed on the target system instead of port 9000
• when testing manually, the source VM is giving me a 500 error rather than the expected test page
• on the target VM, curl localhost in the migrated container gives the same error page, but curl localhost at the VM layer gives connection refused, and so does attempting to access the page from the VM host running the migration script

So I'm guessing I broke something when attempting to simplify the test scenario back to just the default httpd test page...

[ncoghlan]

From further testing today it looks like the problems on Friday were just a race condition between Apache coming up in the macrocontainer and the test harness checking for a response - looping for up to 90 seconds waiting for a response from the target resolved that.

There are at least two remaining issues to resolve:

• accounting for the host IP address appearing in the default Apache response pages
• figuring out why both the original VM and the migrated macrocontainer are giving 500 errors rather than the Apache test page

[shaded-enmity]

Wow, that's plenty of nice changes, my finger is itching on the merge button :)

One thing I've been thinking about is how to run the test suite in e.g. Travis-CI, which led me to:

http://stackoverflow.com/questions/37307114/launching-a-vagrant-vm-inside-travis-ci

So if we "reduce" our source VM to 32bit we might be able to run on Travis (which leads me to another interesting question whether or not the 32bit kernel actually supports Docker).
We could probably facilitate the testing via nested-VM enabled Jenkins (seems like overkill), or something like the new testing pipelines, which allow spawning multiple sub-tasks, in an orderly fashion, as part of single test run.

[shaded-enmity]

I think that the 500's thrown by Apache are due to default misconfiguration, I'll take a look at that.

[shaded-enmity]

Hmm, for some reason I'm getting digest mismatch on the cffi package:

Installing dependencies from Pipfile.lock...
An error occured while installing!
THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    cffi==1.10.0 from https://pypi.python.org/packages/5b/b9/790f8eafcdab455bcd3bd908161f802c9ce5adbf702a83aa7712fcc345b7/cffi-1.10.0.tar.gz#md5=2b5fa41182ed0edaf929a789e602a070 (from -r /tmp/tmp8HwoNK-requirements.txt (line 7)):
        Expected sha256 4fc9c2ff7924b3a1fa326e1799e5dd58cac585d7fb25fe53ccaa1333b0453d65
             Got        b3b02911eb1f6ada203b0763ba924234629b51586f72a21faacc638269f4ced5

Investigating 🔍 🕵️ 🔎

[shaded-enmity]

Okay, so this might be something worth reporting to pipfile devs, I just downloaded bunch of "suspicious" versions of cffi:

 $ sha256sum ~/Downloads/cffi-1.10.0*
267dd2c66a5760c5f4d47e2ebcf8eeac7ef01e1ae6ae7a6d0d241a290068bc38  /home/podvody/Downloads/cffi-1.10.0-cp34-cp34m-manylinux1_x86_64.whl
4fc9c2ff7924b3a1fa326e1799e5dd58cac585d7fb25fe53ccaa1333b0453d65  /home/podvody/Downloads/cffi-1.10.0-cp35-cp35m-manylinux1_x86_64.whl
e7175287f7fe7b1cc203bb958b17db40abd732690c1e18e700f10e0843a58598  /home/podvody/Downloads/cffi-1.10.0-cp36-cp36m-manylinux1_x86_64.whl
b3b02911eb1f6ada203b0763ba924234629b51586f72a21faacc638269f4ced5  /home/podvody/Downloads/cffi-1.10.0.tar.gz

Ie. also include the version of the interpreter in the lockfile so that later on I can get more useful error message - instead of digest mismatch I'd be happier with something like "This pipfile was generated against a different version of Python interpreter, which may negatively impact/affect package resolution, proceed with caution".

[shaded-enmity]

Had to cd integration-tests/ before calling behave.

[shaded-enmity]

Hmm, for some reason running python --version in the pipenv still gives me Python 2.7.11. Probably something weird with my setup (Fedora 23, which still defaults to Python 2.X).

[shaded-enmity]

Ok, after some wrestling I was able to run the tests:

• Had to manually add the venv path to my PATH: export PATH=/home/podvody/.local/share/virtualenvs/leapp-proto-WxQciaax/bin:$PATH
• Run manually in the pip install pyhamcrest requests behave

After the initial test run I'll look into why I'm needed to run these two commands in the first place, must be something subtle specific to my box.

[shaded-enmity]

The 500's are thrown by Apache due to old config, all that is needed is to remove the line:

Require all granted

From roles/httpd/templates/default.tpl.

[ncoghlan]

I filed https://github.com/kennethreitz/pipenv/issues/295 to cover improving the hash mismatch error messages.

When setting up with pipenv, one thing to watch out for is that the --two and --three options are silently ignored when included with any other command. https://github.com/kennethreitz/pipenv/issues/286 covers being able to set a default in Pipfile, while https://github.com/kennethreitz/pipenv/issues/236 covers the fact that pipenv --three install doesn't work and you need to do pipenv --three && pipenv install instead.

[ncoghlan]

Conveniently, fixing the Apache configuration error also removed the server IP address from the HTTP response, which meant the only subsequent adjustment needed to the tests was to start checking for the specific expected HTTP status code, rather than just that the migrated server returned the same status code as the original.

Also, TIL that the Apache test page returns 403 rather than 200 :)

[ncoghlan]

Since this is passing the tests now, I went ahead and merged it so I could start working on removing the Vagrant SSH hacks: #9

As far as 32-bit arches go, you're not going find any official 32-bit CentOS vagrant boxes, so we'd have to create and maintain those ourselves. At that point, it would probably be simpler to ask the CentOS folks if we can set something 64-bit friendly up in the CentOS CI infrastructure rather than trying to get things working in Travis.