Add visual regression test and code coverage report (#1323)

* Add WebDriver test

* Auto host web server

* Clean up

* Clean up

* Add visual regression test

* Clean up

* Clean up

* Add chrome-docker

* Run Chrome in Docker

* Add coveralls

* Run WebDriver in detached mode

* Comment out slow part

* Debug coverage

* Build instrumented build

* Build instrumented build separately

* Reduce concurrently

* Include both instrumented and production build in Docker

* Fix coverage reporting

* Add badge

* Cleanup

* Update CHANGELOG.md

* Clean up

* Typo

.babelrc

@@ -0,0 +1,12 @@
+{
+  "plugins": [
+    "@babel/plugin-proposal-object-rest-spread"
+  ],
+  "presets": [
+    ["@babel/preset-env", {
+      "forceAllTransforms": true,
+      "modules": "commonjs"
+    }],
+    "@babel/preset-react"
+  ]
+}

.gitignore

@@ -1,3 +1,9 @@
+/__tests__/__image_snapshots__/**/__diff_output__
+/coverage
+/debug.log
 /gh-pages
 /lerna-debug.log
 /node_modules
+
+# Do not commit binaries
+/chromedriver*

.travis.yml

@@ -19,10 +19,15 @@ install:
 - npm run bootstrap
 - npm run build
 - npm run prepublishOnly
-- docker build -t webchat.azurecr.io/playground .
+- docker build -f Dockerfile-playground -t webchat.azurecr.io/playground .
 
 script:
-- echo
+- docker-compose up --build -d
+- docker-compose exec webchat ls -la
+- npm test -- --coverageReporters=lcov --coverageReporters=text
+- docker-compose logs
+- docker-compose down --rmi all
+- npm run coveralls
 
 before_deploy:
 - git config --local user.name "Bot Framework"

CHANGELOG.md

@@ -6,6 +6,17 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
+### Added
+- Development build now include instrumentation code, updated build scripts
+   - `npm run build` will build for development with instrumentation code
+   - `npm run prepublishOnly` will build for production
+   - `npm run watch` will also run Webpack in watch loop
+- Automated testing using visual regression testing technique
+   - [Docker-based](https://github.com/SeleniumHQ/docker-selenium) automated testing using headless Chrome and [Web Driver](https://npmjs.com/packages/selenium-webdriver)
+   - Screenshot comparison using [`jest-image-snapshot`](https://npmjs.com/packages/jest-image-snapshot) and [`pixelmatch`](https://npmjs.com/package/pixelmatch)
+   - Code is instrumented using [`istanbul`](https://npmjs.com/package/istanbul)
+   - Test report is hosted on [Coveralls](https://coveralls.io/github/compulim/BotFramework-WebChat)
+
 ### Changed
 - Bump dependencies, in [#1303](https://github.com/Microsoft/BotFramework-WebChat/pull/1303)
    - `@babel`

Dockerfile-testharness

@@ -0,0 +1,14 @@
+FROM node:alpine
+
+RUN apk update && \
+    apk upgrade && \
+    apk add --no-cache bash git openssh
+
+ENV PORT=80
+EXPOSE 80
+RUN npm install serve@10.0.0 -g
+ENTRYPOINT ["npx", "--no-install", "serve", "-p", "80", "/web"]
+
+ADD __tests__/setup/web/ /web
+ADD packages/bundle/dist /web
+WORKDIR /web

README.md

@@ -9,6 +9,7 @@
 <p align="center">
   <a href="https://badge.fury.io/js/botframework-webchat"><img alt="npm version" src="https://badge.fury.io/js/botframework-webchat.svg" /></a>
   <a href="https://travis-ci.org/Microsoft/BotFramework-WebChat"><img alt="Build Status" src="https://travis-ci.org/Microsoft/BotFramework-WebChat.svg?branch=master" /></a>
+  <a href="https://coveralls.io/github/Microsoft/BotFramework-WebChat?branch=master"><img src="https://coveralls.io/repos/github/Microsoft/BotFramework-WebChat/badge.svg?branch=master" alt="Coverage Status" /></a>
 </p>
 
 # How to use
@@ -151,7 +152,15 @@ npm run bootstrap
 npm run build
 ```
 
-> Instead of `npm run build`, you can use `npm run watch` for watch mode.
+## Build tasks
+
+There are 3 types of build tasks in the build process.
+
+- `npm run build`: Build for development (instrumented code for code coverage)
+   - Will bundle as `webchat-instrumented*.js`
+- `npm run watch`: Build for development with watch mode loop
+- `npm run prepublishOnly`: Build for production
+   - Will bundle as `webchat*.js`
 
 ## Testing Web Chat for development purpose
 

__tests__/README.md

@@ -0,0 +1,27 @@
+## How to run tests
+
+Automated testing in Web Chat is using multiple open-source technologies.
+
+- [Travis CI](https://travis-ci.org/) for automatic testing
+- Test against [MockBot](https://github.com/compulim/BotFramework-MockBot)
+   - Try it out with this [live demo](https://microsoft.github.io/BotFramework-WebChat/full-bundle)
+- Visual regression test (a.k.a. compare screenshots)
+   - Generated on [Chrome on Docker](https://github.com/SeleniumHQ/docker-selenium)
+   - Compared using [`pixelmatch`](https://npmjs.com/package/pixelmatch) via [`jest-image-snapshot`](https://npmjs.com/package/jest-image-snapshot)
+- Run under [`Jest`](https://jestjs.io/)
+- [`Istanbul`](https://npmjs.com/package/istanbul) for code coverage
+   - [`Coveralls`](https://coveralls.io/) for test statistics
+
+### Running tests under Docker
+
+- Install Docker
+- On Windows, set environment variable `COMPOSE_CONVERT_WINDOWS_PATHS=1`
+- `docker-compose up --build`
+- `npm test`
+
+### Running tests under local box
+
+- Install latest Chrome
+- Download [ChromeDriver](https://sites.google.com/a/chromium.org/chromedriver/downloads) and extract to project root
+- Set environment variable `WEBCHAT_TEST_ENV=chrome-local`
+- `npm test`

__tests__/__image_snapshots__/.gitignore

@@ -0,0 +1,2 @@
+# Ignore baseline images from local build because they "Works on my machine <TM>"
+/chrome-local

__tests__/basic.js

@@ -0,0 +1,23 @@
+import { By, Key } from 'selenium-webdriver';
+
+function sleep(ms = 1000) {
+  return new Promise(resolve => setTimeout(resolve, ms));
+}
+
+// selenium-webdriver API doc:
+// https://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/index_exports_WebDriver.html
+
+test('setup', async () => {
+  const { driver } = await setupWebDriver();
+
+  await sleep(2000);
+
+  const input = await driver.findElement(By.tagName('input[type="text"]'));
+
+  await input.sendKeys('layout carousel', Key.RETURN);
+  await sleep(2000);
+
+  const base64PNG = await driver.takeScreenshot();
+
+  expect(base64PNG).toMatchImageSnapshot();
+}, 60000);

__tests__/setup/setupTestEnvironment.js

@@ -0,0 +1,25 @@
+import { Options } from 'selenium-webdriver/chrome';
+
+export default function (browserName, builder) {
+  switch (browserName) {
+    case 'chrome-local':
+      return {
+        baseURL: 'http://localhost:$PORT/index.html',
+        builder: builder.forBrowser('chrome').setChromeOptions(
+          (builder.getChromeOptions() || new Options())
+            .windowSize({ height: 640, width: 360 })
+        )
+      };
+
+    case 'chrome-docker':
+    default:
+      return {
+        baseURL: 'http://webchat/',
+        builder: builder.forBrowser('chrome').usingServer('http://localhost:4444/wd/hub').setChromeOptions(
+          (builder.getChromeOptions() || new Options())
+            .headless()
+            .windowSize({ height: 640, width: 360 })
+        )
+      };
+  }
+};

__tests__/setup/setupTestFramework.js

@@ -0,0 +1,100 @@
+import { Builder } from 'selenium-webdriver';
+import { createServer } from 'http';
+import { join } from 'path';
+import { promisify } from 'util';
+import { configureToMatchImageSnapshot } from 'jest-image-snapshot';
+import getPort from 'get-port';
+import handler from 'serve-handler';
+
+import setupTestEnvironment from './setupTestEnvironment';
+
+const BROWSER_NAME = process.env.WEBCHAT_TEST_ENV || 'chrome-docker';
+// const BROWSER_NAME = 'chrome-docker';
+// const BROWSER_NAME = 'chrome-local';
+
+expect.extend({
+  toMatchImageSnapshot: configureToMatchImageSnapshot({
+    customSnapshotsDir: join(__dirname, '../__image_snapshots__', BROWSER_NAME)
+  })
+});
+
+let driverPromise;
+let serverPromise;
+
+global.setupWebDriver = async () => {
+  if (!driverPromise) {
+    driverPromise = (async () => {
+      let { baseURL, builder } = await setupTestEnvironment(BROWSER_NAME, new Builder());
+      const driver = builder.build();
+
+      // If the baseURL contains $PORT, it means it requires us to fill-in
+      if (/\$PORT/i.test(baseURL)) {
+        const { port } = await global.setupWebServer();
+
+        await driver.get(baseURL.replace(/\$PORT/ig, port));
+      } else {
+        await driver.get(baseURL);
+      }
+
+      return { driver };
+    })();
+  }
+
+  return await driverPromise;
+};
+
+global.setupWebServer = async () => {
+  if (!serverPromise) {
+    serverPromise = new Promise(async (resolve, reject) => {
+      const port = await getPort();
+      const httpServer = createServer((req, res) => handler(req, res, {
+        redirects: [
+          { source: '/', destination: '__tests__/setup/web/index.html' }
+        ],
+        rewrites: [
+          { source: '/webchat.js', destination: 'packages/bundle/dist/webchat.js' },
+          { source: '/webchat-es5.js', destination: 'packages/bundle/dist/webchat-es5.js' },
+          { source: '/webchat-minimal.js', destination: 'packages/bundle/dist/webchat-minimal.js' }
+        ],
+        public: join(__dirname, '../..'),
+      }));
+
+      httpServer.once('error', reject);
+
+      httpServer.listen(port, () => {
+        resolve({
+          close: promisify(httpServer.close.bind(httpServer)),
+          port
+        });
+      });
+    });
+  }
+
+  return await serverPromise;
+}
+
+afterEach(async () => {
+  if (driverPromise) {
+    const { driver } = await driverPromise;
+
+    try {
+      global.__coverage__ = await driver.executeScript(() => window.__coverage__);
+
+      ((await driver.executeScript(() => window.__console__)) || [])
+        .filter(([type]) => type !== 'info' && type !== 'log')
+        .forEach(([type, message]) => {
+          console.log(`${ type }: ${ message }`);
+        });
+    } finally {
+      await driver.quit();
+    }
+  }
+});
+
+afterAll(async () => {
+  if (serverPromise) {
+    const { close } = await serverPromise;
+
+    await close();
+  }
+});

__tests__/setup/web/index.html

@@ -0,0 +1,75 @@
+<!DOCTYPE html>
+<html lang="en-US">
+  <head>
+    <title>Web Chat: Automated test harness</title>
+    <script>
+    !function () {
+      'use strict';
+
+      const { console } = window;
+      const log = window.__console__ = [];
+      const push = (type, ...args) => {
+        log.push([type, args.join(' ')]);
+        console[type].apply(console, args);
+      };
+
+      window.console = {
+        error: push.bind(null, 'error'),
+        info: push.bind(null, 'info'),
+        log: push.bind(null, 'log'),
+        warn: push.bind(null, 'warn')
+      };
+
+      window.addEventListener('error', ({ colno, error, filename, lineno, message }) => {
+        push('onError', JSON.stringify({
+          colno,
+          error,
+          filename,
+          lineno,
+          message
+        }));
+      });
+    }();
+    </script>
+    <!--
+      For demonstration purpose, we are using development branch of Web Chat at "/master/webchat.js".
+      When you are using Web Chat for production, you should use the latest stable at "/latest/webchat.js".
+      Or locked down on a specific version "/4.1.0/webchat.js".
+    -->
+    <script src="/webchat-instrumented.js"></script>
+    <style>
+      html, body { height: 100% }
+      body { margin: 0 }
+
+      #webchat,
+      #webchat > * {
+        height: 100%;
+        width: 100%;
+      }
+    </style>
+  </head>
+  <body>
+    <div id="webchat"></div>
+    <script>
+      (async function () {
+        // In this demo, we are using Direct Line token from MockBot.
+        // To talk to your bot, you should use the token exchanged using your Direct Line secret.
+        // You should never put the Direct Line secret in the browser or client app.
+        // https://docs.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-direct-line-3-0-authentication
+
+        const res = await fetch('https://webchat-mockbot.azurewebsites.net/directline/token', { method: 'POST' });
+        const { token } = await res.json();
+
+        window.WebChat.renderWebChat({
+          // directLine: window.WebChat.createDirectLine({
+          //   domain: 'http://localhost:5000/v3/directline',
+          //   webSocket: false
+          // })
+          directLine: window.WebChat.createDirectLine({ token })
+        }, document.getElementById('webchat'));
+
+        document.querySelector('#webchat > *').focus();
+      })().catch(err => console.error(err));
+    </script>
+  </body>
+</html>

docker-compose.yml

@@ -0,0 +1,28 @@
+version: '3'
+
+services:
+  # On Windows, run with COMPOSE_CONVERT_WINDOWS_PATHS=1
+
+  chrome:
+    # https://github.com/SeleniumHQ/docker-selenium
+    # https://hub.docker.com/r/selenium/standalone-chrome/tags/
+    image: selenium/standalone-chrome:3.141.0-actinium
+    networks:
+    - selenium
+    depends_on:
+    - webchat
+    ports:
+    - "4444:4444"
+    volumes:
+    - /dev/shm:/dev/shm
+
+  webchat:
+    build:
+      context: ./
+      dockerfile: Dockerfile-testharness
+    networks:
+    - selenium
+
+networks:
+  selenium:
+    driver: bridge