Testing

Installation

Make sure you have the correct versions of the testing software! Working versions of firefox, selenium and RSelenium are listed in testthat.R and the working version of phantomjs is listed in wercker.yml.

Quick start

animint provides three main functions for testing:

tests_init()
tests_run()
tests_exit()

tests_init() will start up a "remote-controlled" web browser and a local file server. By default, tests_init() will initialize phantomjs, but you should make sure tests pass on Firefox with tests_init("firefox"); tests_run() before pushing changes to GitHub.

Tests adhere to the testthat framework and are automagically performed via TravisCI and wercker everytime we push to GitHub. If you want to simulate those tests locally, simply run tests_init(); tests_run(). If, for some reason, your machine doesn't mimic wercker results, we can actually pull down wercker building environment and inspect it.

On Windows machines, these tests work better using Firefox with tests_init("firefox").

Testing philosophy

Tests are designed to verify that both the compiler and renderer work as intended. Testing is relatively easy on the compiler side as we simply check whether the list returned by animint2dir() contains what we expect. To test the renderer, we have to check whether the DOM contains what we expect.

Writing tests

Since 10 Oct 2015 tests are organized into the following categories. Each category is run in a separate entry of the Travis build matrix.

• test-compiler-*.R: do not need a headless browser, since they only test the JSON/CSV files.
• test-renderer1-*.R, etc: do need a headless browser since they test the renderer HTML. For some reason phantomjs stops if these two categories are combined, so the current workaround is to test them separately.
• test-shiny.R: does need a headless browser but for some reason does not work on travis/wercker. Make sure this passes on your local machine before merging any PRs.

Accessing the DOM

To access the DOM, simulate user behavior, and control the web browser that renders test plots, we reference a global object named remDr. This object is introduced to the global environment by tests_init() and is an instance of RSelenium's remoteDriver class. A good example of using remDr is the getHTML() function. In this function, we query the current page source and return it as an R object.

Reusable functions

We have a collection of functions that are reused across the testing suite. For example, animint2HTML() is a wrapper around animint2dir() that adds the rendered SVG/HTML to the function output. If you write tests, please try to understand and use these functions so we don't re-invent the wheel.

Debugging workflow 1

When writing a new test, say 'test-new.R', I suggest this workflow:

tests_init("firefox")
tests_run(filter = "new")

If the "new" tests fail, and you want to know why, inject a browser() at the point of interest in 'test-new.R', then do:

devtools::load_all()
tests_run(filter = "new")

This will give you access to the testing environment. If you are using phantomjs, it may be useful to see what's happening with: remDr$screenshot(display = TRUE)

Debugging workflow 2

Start R in the animint/tests directory, then do

library(testthat)
library(animint)
library(RSelenium)
tests_init("firefox")
setwd("testthat")
source("helper-functions.R")

then execute whatever code you want in the test-*.R files.

Inspect test builds locally with wercker

Getting started

wercker allows us to essentially pull down the testing environment and inspect it locally. Here I'll walk through how to pull down and inspect this build. First, you'll need to install the wercker command line tool and login to wercker.

wercker login

Next, pull download the build image.

wercker pull 5547df2c909f121c3a040943

Load the image into docker (this could take a while).

docker load -i repository.tar 

Check that your image is there.

docker images

REPOSITORY                       TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
build-5547df2c909f121c3a040943   r-travis            3ab52601609b        37 minutes ago      1.852 GB

For some reason, I haven't had any luck using the REPOSITORY name to run the image, but using the IMAGE ID seems to work:

docker run -it --rm 3ab52601609b /bin/bash

Debugging inside the container

Now we're inside the testing environment (in other words, we're now inside the docker container). Note you can exit this container at any time.

To ensure that tests run, you'll probably want to rename the package "source" to "animint" (otherwise animint tests might not work) and open R under animint's root path.

mv pipeline/source pipeline/animint
cd pipeline/animint
R

Next install animint's package dependencies, load animint, and run the tests. This should replicate the result you see on wercker (if not, I've made a mistake somewhere, so please let me know).

devtools::install_deps(dependencies = TRUE)
devtools::install_github("faizan-khan-iit/ggplot2@5fb99d0cece13239bbbc09c6b8a7da7f86ac58e2")
devtools::install()
setwd("tests"); source("testthat.R")

For debugging purposes, it's probably more convenient to use animint's test helper functions directly (instead of doing setwd("tests"); source("testthat.R")):

library("animint")
library("RSelenium")
library("testthat")
tests_init()
# use the filter argument to just run the broken tests
tests_run(filter = "shiny")

Cleaning up

After you've identified the issue, you'll want to exit the container, and stop it from running. After exiting, you can see any running containers with:

docker ps -a

  CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS                NAMES
  9c7b6a7fc235        3ab52601609b:latest   "/bin/bash"         3 hours ago         Up 3 hours          1410/tcp, 8787/tcp   cocky_bell  

Note you can return just the CONTAINER ID(s) with docker ps -aq. To stop all running containers do:

docker stop $(docker ps -aq)

Finally, if you want, you can remove the image with

docker rmi 3ab52601609b