Automated Storybook Setup
Simplify Storybook story creation and cross-browser image comparison testing
Setting up storybook and implement cross-browser image comparison testing on multiple projects is time consuming.
Instead of providing a wrapper on top of existing projects that will fall out of date, Picturebook lets you retain control of the Storybook and Nightwatch packages.
This project aims to provide utility methods to simplify Storybook, SauceLabs and Nightwatch configuration and screenshot comparison. Specifically:
- Creation of storybook stories: They are created based on your file system structure
- Saucelabs tunnel setup: Reduce SauceConnect config to the SauceConnect binary path and username / accessKey
- Screenshot: Take screenshots of every story on different browsers using SauceLabs and Nightwatch
- Image Comparison: Compare and update screenshots to baselines collocated with your stories.
- You will need a SauceLabs account. If you don't have one you can sign up for a trial here or request a free one for open source projects here.
- If you want to run tests against Safari and IE11 from localhost, add
localtest.dev 127.0.0.1to your/etc/hosts(the exact steps may vary depending on your platform, see here for more details). - Add picturebook and its peer dependencies:
yarn add --dev picturebook @storybook/react nightwatch reactReact is used in the examples but you can also use any flavor of storybook you want.
If you don't have Storybook set it up yet, follow these instructions first.
Some of Picturebook utility methods rely on the output of require.context. This is a webpack construct and it's not available in Node or test environments. There are multiple ways to mock it. If you are using babel you could add the require-context-hook plugin. For instance, if you want to enable it only for tests, you can do something like:
{
"env": {
"test": {
"plugins": ["require-context-hook"]
}
}
}and on your test setup:
require('babel-plugin-require-context-hook/register')()Alternatively, there's also the require-context npm package that will also emulate webpack's require.context.
The API can be split into 2 sections, storybook creation for finding files in the file system and creating storybook stories around them and storybook testing, to connect to SauceLabs and run cross-browser image comparison tests.
getFiles and loadStories have very similar APIs and output.
getFiles allows you to retrieve a list of related files. For instance if you have:
file1.js
file1.md
file2.png
file2.jsIt will provide an array of 2 elements (file1 and file2) with the different related files grouped.
These values are used to identify helper files for stories where *.md is understood to be documentation,.*(spec|test).js is a test file and an unprefixed *.js file is a story. The flattenFolders parameters allow grouping nested folders as well. For instance if flattenFolders: ['a'] file1.js and a/file1.png will be grouped together.
loadStories provide the same functionality than getFiles but it also loads and creates storybook stories. This is why it requires storybook's storiesOf method as a parameter. The response is also similar but it includes a main method with the loaded story.
You can find the exact APIs below:
type StoryPaths = {|
name: string,
parents: $ReadOnlyArray<string>,
title: string,
path: string,
screenshots: {|
[extension: string]: string,
|},
tests: {|
[extension: string]: string,
|},
doc: ?string,
url: ?string,
|}
// Retrieve and group the different stories based on their filenames
function getFiles({|
baseUrl?: string, // The url for the story, defaults to:
// 'http://localhost:6006'
flattenFolders?: Array<string> // Folders to flatten
// defaults to ['__snapshots__', '__screenshots__']
stories: any, // Output of require context with the stories
findKindAndStory?: ( // Customize "selectedKind" and "selectedStory" on the URL
story: StoryPaths, // this is not needed if stories are generated with "loadStories"
) => {|
selectedKind: string,
selectedStory: string
|},
filter?: {| // Filter functions to determine how different files are grouped
screenshots?: (file: string) => boolean,
tests?: (file: string) => boolean,
docs?: (file: string) => boolean,
story?: (file: string, target: string) => boolean,
|}
|}): Array<StoryPaths>
// Retrieve, group, load and create stories
function loadStories({|
baseUrl?: string, // The url for the story, defaults to:
// 'http://localhost:6006'
flattenFolders?: Array<string> // Folders to flatten
// defaults to ['__snapshots__', '__screenshots__']
stories: any, // Output of require context with the stories
decorators: $ReadOnlyArray<Function>,
storiesOf: any,
storyFiles: $ReadOnlyArray<string>,
|}): Array<StoryPaths & {|
main: () => any,
|}>Once stories are loaded, runTests will connect to SauceLabs (optionally through a SauceConnect tunnel) and return a Promise that includes the results of the test.
Since it uses nightwatch under the hood, you must have a valid nightwatch instance running. To assist with it, nightwatchConfig provides defaultValues
type Status = 'CREATED' | 'SUCCESS' | 'FAILED'
type ImgResults = {|
browser: string,
diffPath: ?string,
diffThreshold: number,
error: ?string,
name: string,
platform: string,
referencePath: ?string,
screenshotPath: ?string,
status: Status,
|}
type Tunnel = {|
id: string,
binaryPath: string,
|}
// Invoke nightwatch and run image comparison in SauceLabs
function runTests(options: {|
configPath: string, // absolute path to nightwatch config
files: Array<StoryPaths>, // output of getFiles()
overwrite: boolean, // true to replace failing or missing images
storyRoot: string, // absolute path to the stories
tunnel?: Tunnel, // if set, it will set up a SC tunnel with the id
outputPath?: string, // if set, where to store the test results as a
// json file matching the return of `runTests`
|}): Promise<{|
error: Error | null, // null if there were no exceptions during tests
results: Array<ImgResults>, // per browser / platform / story results
// An overall summary status. If any tests failed, it's FAILED, if any tests
// where created or updated it's CREATED, if all tests succeeded it's SUCCESS
// if no tests ran, it's EMPTY
status: Status | 'EMPTY',
counts: {| // counts for all tests
CREATED: number,
SUCCESS: number,
FAILED: number,
|},
version: string, // picturebook version used
date: string, // ISO date when was the test report created
|}>
// Nightwatch config generation helper
function nightwatchConfig(options: {|
desiredCapabilities?: {}, // nightwatch desired capabilities object
localOverrides: {}, // overrides set when browser.local flag is true
remoteOverrides: {}, // overrides when browser.local flag is falsy
files: Array<StoryPaths>, // output of getFiles()
username: string, // SauceLabs username
access_key: string, // SauceLabs accessKey
browsers?: { // List of browsers to test. Default list
// includes chrome, firefox, edge, ie11,
// iphone7 and safari but any valid SauceLabs
// config is valid
[browserName: string]: {
desiredCapabilities: {
platform: string,
version: string,
browserName: string,
screenResolution: string,
},
custom_vars: {| // Custom properties required per browser config:
local?: boolean, // optionally run this browser locally. Useful for
// a mix of remote and local browsers
name: string, // Must match "browserName"
platform: string, // "mobile" | "desktop" but any string is allowed
extract?: {| // If cropping the output, specify crop dimensions
top: number,
left: number,
width: number,
height: number,
|},
|},
},
},
// Alias for browsers where localhost cannot be accessed from a tunnel,
// defaults to "localtest.dev"
localhostAlias?: string,
// List of browsers that can't access localhost from a tunnel, defaults to:
// ["ie11", "safari"]
localhostAliasBrowsers?: Array<string>,
proxy?: {}, // Nightwatch proxy object
|}): ObjectThis project includes a sample app to demo the behavior. You can view it running yarn start and opening localhost:6006 on your browser.
To run the image comparison tests, keep yarn start running and call SAUCE_ACCESS_KEY={YOUR_ACCESS_KEY} SAUCE_USERNAME={YOUR_USERNAME} yarn test:app.
To take advantage of the automated storybook folder config (and testing) modify your storybook config.js to look like this:
import { storiesOf, configure } from '@storybook/react'
import * as picturebook from 'picturebook'
function loadStories() {
picturebook.loadStories({
storiesOf,
stories: require.context('../stories', true, /\.(js|md|png)/),
})
}
configure(loadStories, module)If you want to see a more advanced use case on how to use it with decorators and other plugins, check out the sample project.
To take screenshots, picturebook relies on nightwatch. With the exception of a few required fields, configuration is generated for you. To take advantage of it you should create a nightwatch.conf.js file on your root (or customize it following instructions here).
const { resolve } = require('path')
const { nightwatchConfig, getFiles } = require('picturebook')
const requireContext = require('require-context')
const config = nightwatchConfig({
username: process.env.SAUCE_USERNAME,
access_key: process.env.SAUCE_ACCESS_KEY,
files: getFiles({
baseUrl: 'http://localhost:6006',
stories: requireContext(resolve(__dirname, './stories'), true, /\.js/),
}),
desiredCapabilities: {
build: 'local',
'tunnel-identifier': 'picturebook-sample',
},
})
module.exports = configTo take screenshots and run image comparison, you can call:
runTests({
storyRoot,
files: getFiles({
stories: requireContext(storyRoot, true, /\.(js|png)/),
}),
overwrite,
tunnel,
configPath: resolve(__dirname, 'nightwatch.conf.js'),
})Note that if using a tunnel, tunnel-identifier on your nightwatch config must match the tunnel.id parameter passed to runTests.
This is a SauceLabs issue. To work around it, safari and edge browsers replace localhost from urls to localtest.dev as described here.
You can customize the target browsers and localhost alias with the localhostAliasBrowsers and localhostAlias, respectively.
Picturebook will only take screenshots of the files you tell it exists. You can filter the output from require.context by specifying a different regExp to filter by.
If you see the following error after a couple unsuccessful runs of SauceConnect:
events.js:167
throw er; // Unhandled 'error' event
^
Error: spawn ~/picturebook/node_modules/node-sauce-connect/lib/sc ENOENTSometimes it gets stuck. Try cleaning node_modules and re-running yarn:
rm -rf node_modules
yarnThis is a Nightwatch issue, addressed in 1.0.4 versions of Nightwatch and later as per this issue.
There's currently a bug in node-dir that fails on empty dirs.
Remove empty folders within your stories if you see the following error:
yourProject/node_modules/node-dir/lib/paths.js:92
results.files = results.files.concat(res.files);
^
TypeError: Cannot read property 'files' of undefined
at subloop (/Users/obartra/repos/gymnast/node_modules/node-dir/lib/paths.js:92:58)
at /Users/obartra/repos/gymnast/node_modules/node-dir/lib/paths.js:107:13
at onDirRead (/Users/obartra/repos/gymnast/node_modules/node-dir/lib/paths.js:139:35)
at onStat (/Users/obartra/repos/gymnast/node_modules/node-dir/lib/paths.js:154:14)You can do so with:
find /some/path -depth -type d -exec rmdir {} \;Selenium 3.0+ requires Java 8. Make sure it's installed if you get the following error:
Starting selenium server in parallel mode... There was an error while starting the Selenium server:
No Java runtime present, requesting install.For additional Selenium tips, check out selenium-standalone tips.