e2e: document e2e provisioning process

e2e/.gitignore

@@ -0,0 +1 @@
+provisioning.json

e2e/README.md

@@ -1,23 +1,90 @@
-End to End Tests
-================
+# End to End Tests
 
-This package contains integration tests. 
+This package contains integration tests.
 
-The `terraform` folder has provisioning code to spin up a Nomad cluster on AWS. The tests work with the `NOMAD_ADDR` environment variable which can be set either to a local dev Nomad agent or a Nomad client on AWS. 
+The `terraform` folder has provisioning code to spin up a Nomad cluster on AWS.
+The tests work with the `NOMAD_ADDR` environment variable which can be set
+either to a local dev Nomad agent or a Nomad client on AWS.
 
-Local Development
-=================
-The workflow when developing end to end tests locally is to run the provisioning step described below once, and then run the tests as described below.
-When making local changes, use `./bin/update $(which nomad) /usr/local/bin/nomad` and `./bin/run sudo systemctl restart nomad` to destructively modify the provisioned cluster.
+## Local Development
 
-Provisioning
-============
-You'll need AWS credentials (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`) to create the Nomad cluster. See the [README](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md) for details. The number of servers and clients is configurable, as is the configuration file for each client and server.
+The workflow when developing end to end tests locally is to run the
+provisioning step described below once, and then run the tests as described
+below.
 
-Running
-===========
-After completing the provisioning step above, you should see CLI output showing the IP addresses of Nomad client machines. To run the tests, set the NOMAD_ADDR variable to `http://[client IP]:4646/`
+When making local changes, use `./bin/update $(which nomad) /usr/local/bin/nomad`
+and `./bin/run sudo systemctl restart nomad` to destructively modify the
+provisioned cluster.
 
+## Provisioning Test Infrastructure on AWS
+
+You'll need Terraform and AWS credentials (`AWS_ACCESS_KEY_ID` and
+`AWS_SECRET_ACCESS_KEY`) to setup AWS instances on which e2e tests
+will run. See the [README](https://github.com/hashicorp/nomad/blob/master/e2e/terraform/README.md)
+for details. The number of servers and clients is configurable, as is
+the configuration file for each client and server.
+
+## Provisioning e2e Framework Nomad Cluster
+
+You can use the Terraform output from the previous step to generate a
+provisioning configuration file for the e2e framework.
+
+```sh
+# from the ./e2e/terraform directory
+terraform output provisioning | jq . > ../provisioning.json
+```
+
+By default the `provisioning.json` will include a `nomad_sha` field
+for each node. You can edit this file to change the version of Nomad
+you want to deploy. Because each node has its own value, you can
+create cluster of mixed versions. The provisioning framework accepts
+any of the following options:
+
+- `nomad_sha`: This is a Nomad binary identified by its full commit SHA that's
+  stored in a shared s3 bucket that Nomad team developers can access. That
+  commit SHA can be from any branch that's pushed to remote.  (Ex.
+  `"nomad_sha": "0b6b475e7da77fed25727ea9f01f155a58481b6c"`)
+- `nomad_local_binary`: This is a path to a Nomad binary on your own host.
+  (Ex. `"nomad_local_binary": "/home/me/nomad"`)
+- `nomad_version`: This is a version number of Nomad that's been released to
+  HashiCorp. (Ex. `"nomad_version": "0.10.2"`)
+
+You can pass the following flags to `go test` to override the values
+in `provisioning.json` for all nodes:
+
+- `-nomad.local_file=string`: provision this specific local binary of Nomad
+- `-nomad.sha=string`: provision this specific sha from S3
+- `-nomad.version=string`: provision this version from
+  [releases.hashicorp.com](https://releases.hashicorp.com/nomad)
+
+Deploy Nomad to the cluster:
+
+```sh
+# from the ./e2e/terraform directory, set your client environment
+$(terraform output environment)
+
+cd ..
+go test -v . -provision.terraform ./provisioning.json -skipTests
 ```
-$ NOMAD_ADDR=<> NOMAD_E2E=1 go test -v
+
+## Running
+
+After completing the provisioning step above, you can set the client
+environment for `NOMAD_ADDR` and run the tests as shown below:
+
+```sh
+# from the ./e2e/terraform directory, set your client environment
+# if you haven't already
+$(terraform output environment)
+
+cd ..
+go test -v .
 ```
+
+If you want to run a specific test, you'll need to regex-escape some of the
+test's name so that the test runner doesn't skip over framework struct method
+names in the full name of the tests:
+
+```sh
+ go test -v . -run 'TestE2E/Consul/\*consul\.ScriptChecksE2ETest/TestGroup'
+ ```

e2e/terraform/README.md

@@ -1,39 +1,84 @@
-Terraform provisioner for end to end tests
-==========================================
+# Terraform infrastructure
 
-This folder contains terraform resources for provisioning a nomad cluster on AWS for end to end tests.
-It uses a Nomad binary identified by its commit SHA that's stored in a shared s3 bucket that Nomad team
-developers can access. The commit SHA can be from any branch that's pushed to remote.
+This folder contains terraform resources for provisioning EC2 instances on AWS
+to use as the target of end-to-end tests.
 
-Use [envchain](https://github.com/sorah/envchain) to store your AWS credentials.
+Terraform provisions the AWS infrastructure only, whereas the Nomad
+cluster is deployed to that infrastructure by the e2e
+framework. Terraform's output will include a `provisioning` stanza
+that can be written to a JSON file used by the e2e framework's
+provisioning step.
 
+You can use Terraform to output the provisioning parameter JSON file the e2e
+framework uses by setting the `nomad_sha` variable.
 
+## Setup
+
+You'll need Terraform 0.12+, as well as AWS credentials (`AWS_ACCESS_KEY_ID`
+and `AWS_SECRET_ACCESS_KEY`) to create the Nomad cluster. Use
+[envchain](https://github.com/sorah/envchain) to store your AWS credentials.
+
+Optionally, edit the `terraform.tfvars` file to change the number of
+Linux clients, Windows clients, or the Nomad build. You'll usually
+want to have the `nomad_sha` variable set here (or via the
+`TF_VAR_nomad_sha` env var) so that `terraform output provisioning` is
+populated with the build you want.
+
+```hcl
+region               = "us-east-1"
+instance_type        = "t2.medium"
+server_count         = "3"
+client_count         = "4"
+windows_client_count = "1"
+
+# alternatively, set this via env var: TF_VAR_nomad_sha
+# nomad_sha            = ""
 ```
-$ cd e2e/terraform/
-$ TF_VAR_nomad_sha=<nomad_sha> envchain nomadaws terraform apply
+
+Run Terraform apply to deploy the infrastructure:
+
+```sh
+cd e2e/terraform/
+TF_VAR_nomad_sha=<nomad_sha> envchain nomadaws terraform apply
 ```
 
-After this step, you should have a nomad client address to point the end to end tests in the `e2e` folder to.
+## Outputs
+
+After deploying the infrastructure, you can get connection information
+about the cluster:
+
+- `$(terraform output environment)` will set your current shell's
+  `NOMAD_ADDR` and `CONSUL_HTTP_ADDR` to point to one of the cluster's
+  server nodes, and set the `NOMAD_E2E` variable.
+- `terraform output servers` will output the list of server node IPs.
+- `terraform output linux_clients` will output the list of Linux
+  client node IPs.
+- `terraform output windows_clients` will output the list of Windows
+  client node IPs.
+- `terraform output provisioning | jq .` will output the JSON used by
+  the e2e framework for provisioning.
 
 ## SSH
 
-Terraform will output node IPs that may be accessed via ssh:
+You can use Terraform outputs above to access nodes via ssh:
 
-```
+```sh
 ssh -i keys/nomad-e2e-*.pem ubuntu@${EC2_IP_ADDR}
 ```
 
-The Windows client runs OpenSSH for conveniences, but has a different user and will drop you into a Powershell shell instead of bash:
+The Windows client runs OpenSSH for convenience, but has a different
+user and will drop you into a Powershell shell instead of bash:
 
-```
+```sh
 ssh -i keys/nomad-e2e-*.pem Administrator@${EC2_IP_ADDR}
 ```
 
-
 ## Teardown
-The terraform state file stores all the info, so the nomad_sha doesn't need to be valid during teardown.
 
-```
-$ cd e2e/terraform/
-$ envchain nomadaws TF_VAR_nomad_sha=yyyzzz terraform destroy
+The terraform state file stores all the info, so the `nomad_sha`
+doesn't need to be valid during teardown.
+
+```sh
+cd e2e/terraform/
+envchain nomadaws TF_VAR_nomad_sha=yyyzzz terraform destroy
 ```

e2e/terraform/terraform.tfvars

@@ -3,3 +3,6 @@ instance_type        = "t2.medium"
 server_count         = "3"
 client_count         = "4"
 windows_client_count = "1"
+
+# alternatively, set this via env var: TF_VAR_nomad_sha
+# nomad_sha            = ""