E2E testing enhancements (#2408)

* Local testing with docker

* Docs on e2e testing

* `bash` callback after docker

* Add a TL;DR in the CONTRIBUTING section

* Local e2e testing with modified files

* Remove the N.B. from the README

* Fixed a fixme in e2e doc

CONTRIBUTING.md

@@ -90,6 +90,12 @@ and then run `npm start` or `npm run build`.
 
 *Note: if you are using yarn, we suggest that you use `yarn install --no-lockfile` instead of the bare `yarn` or `yarn install` because we [intentionally](https://github.com/facebookincubator/create-react-app/pull/2014#issuecomment-300811661) do not ignore or add yarn.lock to our repo.*
 
+## Contributing to E2E (end to end) tests
+
+**TL;DR** use the command yarn e2e:docker to run unit and e2e tests.
+
+More detailed information are in the dedicated [README](/packages/react-scripts/fixtures/kitchensink/README.md).
+
 ## Cutting a Release
 
 1. Tag all merged pull requests that go into the release with the relevant milestone. Each merged PR should also be labeled with one of the [labels](https://github.com/facebookincubator/create-react-app/labels) named `tag: ...` to indicate what kind of change it is.

package.json

@@ -5,6 +5,7 @@
     "changelog": "lerna-changelog",
     "create-react-app": "tasks/cra.sh",
     "e2e": "tasks/e2e-simple.sh",
+    "e2e:docker": "tasks/local-test.sh",
     "postinstall": "lerna bootstrap && cd packages/react-error-overlay/ && npm run build:prod",
     "publish": "tasks/release.sh",
     "start": "node packages/react-scripts/scripts/start.js",

packages/react-scripts/fixtures/kitchensink/README.md

@@ -0,0 +1,52 @@
+# Contributing to Create React App's E2E tests
+
+This is an end to end kitchensink test suite, but has multiple usages in it.
+
+## Running the test suite
+
+Tests are automatically run by the CI tools.  
+In order to run them locally, without having to manually install and configure everything, the `yarn e2e:docker` CLI command can be used.
+
+This is a simple script that runs a **Docker** container, where the node version, git branch to clone, test suite, and whether to run it with `yarn` or `npm` can be chosen.  
+Simply run `yarn e2e:docker -- --help` to get additional info.
+
+## Writing tests
+
+Each time a new feature is added, it is advised to add at least one test covering it.
+
+Features are categorized by their scope:
+
+ - *env*, all those which deal with environment variables (e.g. `NODE_PATH`)
+
+ - *syntax*, all those which showcase a single EcmaScript syntax feature that is expected to be transpiled by **Babel**
+
+ - *webpack*, all those which make use of webpack settings, loaders or plugins
+
+### Using it as Unit Tests
+
+In it's most basic for this serve as a collection of unit tests on a single functionality.
+
+Unit tests are written in a `src/features/**/*.test.js` file located in the same folder as the feature they test, and usually consist of a simple `ReactDOM.render` call.
+
+These tests are run by **jest** and the environment is `test`, so that it resembles how a **Create React App** application is tested.
+
+### Using it as Integration Tests
+
+This suite tests how the single features as before behave while development and in production.  
+A local HTTP server is started, then every single feature is loaded, one by one, to be tested.
+
+Test are written in `integration/{env|syntax|webpack}.test.js`, depending on their scope.
+
+For every test case added there is just a little chore to do:
+
+ - a `case` statement must be added in `src/App.js`, which simply perform a dynamic `import()` of the feature
+
+ - add a test case in the appropriate integration test file, which calls and awaits `initDOM` with the previous `SwitchCase` string
+
+An usual flow for the test itself is something similar to:
+
+ - add an `id` attribute in a target HTML tag in the feature itself
+
+ - since `initDOM` returns a `Document` element, the previous `id` attribute is used to target the feature's DOM and `expect` accordingly
+
+These tests are run by **mocha** (why not **jest**? See [this issue](https://github.com/facebook/jest/issues/2288)) and the environments used are both `development` and `production`.

tasks/local-test.sh

@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+
+function print_help {
+  echo "Usage: ${0} [OPTIONS]"
+  echo ""
+  echo "OPTIONS:"
+  echo "  --node-version <version>  the node version to use while testing [6]"
+  echo "  --git-branch <branch>     the git branch to checkout for testing [the current one]"
+  echo "  --test-suite <suite>      which test suite to use ('simple', installs', 'kitchensink', 'all') ['all']"
+  echo "  --yarn                    if present, use yarn as the package manager"
+  echo "  --interactive             gain a bash shell after the test run"
+  echo "  --help                    print this message and exit"
+  echo ""
+}
+
+cd $(dirname $0)
+
+node_version=6
+current_git_branch=`git rev-parse --abbrev-ref HEAD`
+git_branch=${current_git_branch}
+use_yarn=no
+test_suite=all
+interactive=false
+
+while [ "$1" != "" ]; do
+  case $1 in
+    "--node-version")
+      shift
+      node_version=$1
+      ;;
+    "--git-branch")
+      shift
+      git_branch=$1
+      ;;
+    "--yarn")
+      use_yarn=yes
+      ;;
+    "--test-suite")
+      shift
+      test_suite=$1
+      ;;
+    "--interactive")
+      interactive=true
+      ;;
+    "--help")
+      print_help
+      exit 0
+      ;;
+  esac
+  shift
+done
+
+test_command="./tasks/e2e-simple.sh && ./tasks/e2e-kitchensink.sh && ./tasks/e2e-installs.sh"
+case ${test_suite} in
+  "all")
+    ;;
+  "simple")
+    test_command="./tasks/e2e-simple.sh"
+    ;;
+  "kitchensink")
+    test_command="./tasks/e2e-kitchensink.sh"
+    ;;
+  "installs")
+    test_command="./tasks/e2e-installs.sh"
+    ;;
+  *)
+    ;;
+esac
+
+read -r -d '' apply_changes <<- CMD
+cd /var/create-react-app
+git config --global user.name "Create React App"
+git config --global user.email "cra@email.com"
+git stash save -u
+git stash show -p > patch
+git diff 4b825dc642cb6eb9a060e54bf8d69288fbee4904 stash^3 >> patch
+git stash pop
+cd -
+mv /var/create-react-app/patch .
+git apply patch
+rm patch
+CMD
+
+if [ ${git_branch} != ${current_git_branch} ]; then
+  apply_changes=''
+fi
+
+read -r -d '' command <<- CMD
+echo "prefix=~/.npm" > ~/.npmrc
+mkdir ~/.npm
+export PATH=\$PATH:~/.npm/bin
+set -x
+git clone /var/create-react-app create-react-app --branch ${git_branch}
+cd create-react-app
+${apply_changes}
+node --version
+npm --version
+set +x
+${test_command} && echo -e "\n\e[1;32m✔ Job passed\e[0m" || echo -e "\n\e[1;31m✘ Job failes\e[0m"
+$([[ ${interactive} == 'true' ]] && echo 'bash')
+CMD
+
+docker run \
+  --env CI=true \
+  --env NPM_CONFIG_QUIET=true \
+  --env USE_YARN=${use_yarn} \
+  --tty \
+  --user node \
+  --volume ${PWD}/..:/var/create-react-app \
+  --workdir /home/node \
+  $([[ ${interactive} == 'true' ]] && echo '--interactive') \
+  node:${node_version} \
+  bash -c "${command}"