Update main project readme (microsoft#79)

* # This is a combination of 99 commits.
# This is the 1st commit message:

Adding docker test base image

# This is the commit message microsoft#2:

Adding test files

# This is the commit message microsoft#3:

testing

# This is the commit message microsoft#4:

Added folder

# This is the commit message microsoft#5:

Added folder

# This is the commit message microsoft#6:

Azure Devops CI Build Pipeline (microsoft#67)

* Adding azure devops build pipeline

* Adding README

* Adding azure build badge

# This is the commit message microsoft#7:

saving latest tests

# This is the commit message microsoft#8:

Adding docker test base image (microsoft#68)


# This is the commit message microsoft#9:

EOD checkin

# This is the commit message microsoft#10:

posting latest

# This is the commit message microsoft#11:

eod commits

# This is the commit message microsoft#12:

eod commit

# This is the commit message microsoft#13:

refactored test directory structure

# This is the commit message microsoft#14:

Refactored build

# This is the commit message microsoft#15:

Adding hello world azure simple app

# This is the commit message microsoft#16:

Adding hello world azure simple app

# This is the commit message microsoft#17:

refactoring test harness

# This is the commit message microsoft#18:

refactoring test harness

# This is the commit message microsoft#19:

checking latest

# This is the commit message microsoft#20:

committing latest

# This is the commit message microsoft#21:

checking in latest changes

# This is the commit message microsoft#22:

Checking in latest progress

# This is the commit message microsoft#23:

applying final touches before PR submission

# This is the commit message microsoft#24:

updating azure keyvault secret names

# This is the commit message microsoft#25:

Checking in latest progress

# This is the commit message microsoft#26:

pushing latest pipeline definition

# This is the commit message microsoft#27:

updating build pipeline variable format

# This is the commit message microsoft#28:

updating build pipeline variable format

# This is the commit message microsoft#29:

cleaning up build variables

# This is the commit message microsoft#30:

adding minor changes to azure build pipeline

# This is the commit message microsoft#31:

updating azure keyvault secret names

# This is the commit message microsoft#32:

updating azure keyvault secret names

# This is the commit message microsoft#33:

fixing docker login azdo step

# This is the commit message microsoft#34:

fixing docker login azdo step

# This is the commit message microsoft#35:

fixing docker login azdo step

# This is the commit message microsoft#36:

fixing docker login azdo step

# This is the commit message microsoft#37:

updating build pipeline variables

# This is the commit message microsoft#38:

updating build pipeline variables

# This is the commit message microsoft#39:

updating build pipeline variables

# This is the commit message microsoft#40:

updating build pipeline variables

# This is the commit message microsoft#41:

updating build pipeline variables

# This is the commit message microsoft#42:

updating build pipeline variables

# This is the commit message microsoft#43:

applying fix to git diff script

# This is the commit message microsoft#44:

applying fix to git diff script

# This is the commit message microsoft#45:

applying fix to git diff script

# This is the commit message microsoft#46:

applying fix to git diff script

# This is the commit message microsoft#47:

applying fix to git diff script

# This is the commit message microsoft#48:

resolving pipeline bug

# This is the commit message microsoft#49:

fixing build error

# This is the commit message microsoft#50:

fixing build error

# This is the commit message microsoft#51:

fixing build error

# This is the commit message microsoft#52:

fixing build error

# This is the commit message microsoft#53:

fixing build error

# This is the commit message microsoft#54:

fixing build error

# This is the commit message microsoft#55:

fixing build error

# This is the commit message microsoft#56:

fixing build error

# This is the commit message microsoft#57:

fixing build error

# This is the commit message microsoft#58:

fixing build error

# This is the commit message microsoft#59:

fixing build error

# This is the commit message microsoft#60:

fixing build error

# This is the commit message microsoft#61:

fixing build error

# This is the commit message microsoft#62:

fixing build error

# This is the commit message microsoft#63:

fixing build error

# This is the commit message microsoft#64:

fixing build error

# This is the commit message microsoft#65:

fixing build error

# This is the commit message microsoft#66:

fixing build error

# This is the commit message microsoft#67:

fixing build error

# This is the commit message microsoft#68:

fixing build error

# This is the commit message microsoft#69:

fixing build error

# This is the commit message microsoft#70:

fixing build error

# This is the commit message microsoft#71:

fixing build error

# This is the commit message microsoft#72:

fixing build error

# This is the commit message microsoft#73:

trying acr fix

# This is the commit message microsoft#74:

latest changes

# This is the commit message microsoft#75:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#76:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#77:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#78:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#79:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#80:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#81:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#82:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#83:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#84:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#85:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#86:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#87:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#88:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#89:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#90:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#91:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#92:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#93:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#94:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#95:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#96:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#97:

removing docker dependency on azure devops pipeline

# This is the commit message microsoft#98:

final commit including resource name randomization

# This is the commit message microsoft#99:

Add provider to configure az infra for Terraform

* resolving merge conflicts for squash

Updated documentation based on review comments

Integrate review comments for documentation.

Add usage example for each of the providers

Adding environment template descriptions

removing go version var from azdo pipeline as it's unused

removing unused setup folder

adding more verbose logging for azure devops failure

fixing pull request branch name resolution bug

fixing pull request branch name resolution bug

addressing nick's pr comments

documenting script and docker images

cleaning up required env var check

fixing base branch bug

Added service plan terraform templates to Azure

Adding tags for resources

Integrate review comments to add random id to name

Add tags, capacity and reserved for service plan

Minor Fixes for documentation

Integrate review comments related to resource naming

Remove Optional and Required in variables.tf

Update variables.tf

Update Service Plan Tier to Isolated

squaching commits

applied base image suggestions from nick

* squashing commits and slimming down the build image

* removing go* path from azure devops pipeline as we removed go mods

* refactoring our azure devops pipeline so it's aligned with the local test runner

* fixing azure devops pipeline bug

* fixing base image build variable reference bug

* Addressing nick's PR comments

* resolving issue with failed builds after completing branches merges

* addressing a bug where test harness directory changes were not getting picked up by the harness

* removing local echo statement

* Adding directory exists check and app service container command variable

* addressing nick's PR comments

* adding git diff extension whitelist filter

* Overhauling main project readme

* Adding james to the readme

* Update README.md

* Updating readme from the test-harness directory

README.md

@@ -1,120 +1,90 @@
-[![Build Status](https://dev.azure.com/csedallascrew/project-cobalt/_apis/build/status/Microsoft.cobalt?branchName=master)](https://dev.azure.com/csedallascrew/project-cobalt/_build/latest?definitionId=2&branchName=master)
+# Cobalt
 
-# Setup
+[![Build Status](https://dev.azure.com/csedallascrew/project-cobalt/_apis/build/status/Microsoft.cobalt?branchName=master)](https://dev.azure.com/csedallascrew/project-cobalt/_build/latest?definitionId=2&branchName=master)
 
-The artifacts used to deploy this project include bash scripts and Terraform templates.  The sections below provide guidance to deploy this project into your Azure environment.
+This project is an attempt to combine and share best practices when building production ready [cloud native](https://www.cncf.io/) managed service solutions. Cobalt's infrastructure turn-key starter [templates](/infra/templates/README.md) are based on real world engagements with enterprise customers.
 
-> The setup instructions below assume the following requirements:
-> - bash v4.0 (or newer)
->   - **NOTE FOR MAC!** The default version of bash installed on Mac is older than 4.0. Be sure to update bash using brew before executing the script. Instructions to update bash can be found [here](http://macappstore.org/bash/).
-> - Terraform v0.11.13 (or newer)
+This project puts a focus on infrastructure scalability, security, automated testing and deployment repeatability and most importantly, developer experience. Cobalt's intended audience is for developers. Feedback and suggestions are encouraged through issue requests. We welcome contributions across any one of the major cloud providers.
 
+Cobalt is a joint collaboration with project [Bedrock](https://github.com/Microsoft/bedrock). One of the project goals is predictlably create, change and improve infrastructure.
 
-## Setup the Azure Container Registry and Service Principals
+This project offers a set of continuous integration pipelines responsible for testing and deploying templated environments to cloud provider(s).
 
-1. Open a bash command prompt.
-2. Navigate to the `./setup` folder.
-3. Authenticate to Azure.
-    ``` bash
-    az login
-    ```
-4. Run `acr-sp-init.sh`.  For example, the command below will provision an Azure Container Registry (ACR) in East US and configure the two service principals in Azure Active Directory; one with _AcrPush_ permission and another with _AcrPull_ permission scoped to the ACR.  The script parameter values are used to construct the name of the resource group, ACR, and service principals.
+## How Cobalt differs to Bedrock
 
-    ``` bash
-    $ ./acr-sp-init.sh -a Cblt -l eastus -s CoreProd
-    ```
+Cobalt hosts reusable Terraform modules to scaffold managed container services like [ACI](https://docs.microsoft.com/en-us/azure/container-instances/) and [Application Services](https://docs.microsoft.com/en-us/azure/app-service/) as a couple of examples. Bedrock targets Kubernetes-based container orchestration workloads while following a [GitOps](https://medium.com/@timfpark/highly-effective-kubernetes-deployments-with-gitops-c7a0354f1446) devops flow. Cobalt templates reference Terraform modules like virtual networks, traffic manager, etc.
 
-    > Note: The script configures service principals in Azure AD and therefore requires elevated privileges.  As such, it is recommended that an interactive user with permissions to configure Azure AD run the script.
-
-### Automated Test
+## About the Repository
 
-The automated test for this setup step is in `./tests/acr-sp-init-test.sh`.  It can be executed at the command line as shown below, or as part of a CI pipeline.  If you don't want the test to cleanup resources, then pass the `-n` switch.
+### Infrastructure as Code
 
-``` bash
-az login
+Cobalt deployment environment templates are written in Terraform and can be found in the templates [folder](infra/templates). Each subfolder represents a unique deployment environment packaged with a set of Terraform scripts, overview and setup instructions, automated unit and integration tests.
 
-# Run test and cleanup resources created.
-./tests/acr-sp-init-test.sh
+Each template makes use of Terraform [modules](https://www.terraform.io/docs/modules/index.html) across both Bedrock and [Cobalt](infra/modules). Cobalt's module registry is categorized by cloud provider then resource type. Each modules represents an absraction for the set of related cloud infrastructure objects that the module will manage.
+``` 
+$ tree infra
+├───modules
+│   └───providers
+│       ├───azure
+│       │   ├───api-mgmt
+│       │   ├───app-gateway
+│       │   ├───provider
+│       │   ├───service-plan
+│       │   ├───tm-endpoint-ip
+│       │   ├───tm-profile
+│       │   └───vnet
+│       └───common
+└───templates
+    ├───azure-simple-hw
+    │   └───test
+    │       └───integration
+    └───backend-state-setup
 ```
 
-## Setup Shared / Core Infrastructure
+### Continuous Integration / Deployment + Testing
 
-### Requirements
+Cobalt Continuous Integration pipeline defintions are available in the `./devops/provider` folder. As of today, Cobalt provides a git devops worklfow [definition](devops/providers/azure-devops/templates/azure-pipelines.yml) for Azure DevOps. We welcome pipelines from other providers like Jenkins.
 
-- Azure Subscription User (with deployment rights)
-- [Terraform](https://www.terraform.io/downloads.html)
+#### Azure DevOps CI Flow
 
-### Resources
+![image](https://user-images.githubusercontent.com/7635865/56855601-73383480-690f-11e9-9ec9-3f35bedb39ec.png)
 
-The following respources will be deployed
-- Azure Resource Group
+This pipeline is configured to trigger new builds for each new branch commit.
 
-### Deployment
+1. Deployment credential secrets such as service principal and terraform remote state storage accounts are sourced in azure keyvault.
+2. The pipeline downloads secrets from keyvault and used to resolve terraform template variables.
+3. We rebuild the test harness image so we can copy the terraform template changes from the git branch over to the docker file system.
+4. We then run the test harness container, which performs the following stages.
+    * Run a lint check on all golang test files and terraform templates.
+    * Executes all golang unit tests.
+    * Generate and validate the terraform plan.
+    * Apply the terraform template resource updates to the development integration deployment environment.
+    * Run end-to-end integration tests.
+    * Tear down deployed resources.
+5. Update the build and PR status.
+6. Begin code review once the PR status is green.
 
-1. Authenticate using your Azure Principal or an Azure account with privileges to deploy resource groups.
+### Getting Started
 
-``` bash
-$ az login
-```
+The easiest way to try Cobalt is to start with our [azure-simple](https://github.com/Microsoft/cobalt/tree/master/infra/templates/azure-simple-hw) template.
 
-2. Execute the following commands:
+Setting up a cobalt deployment comprises of 5 general steps.
 
-``` bash
-$ cd ./shared
-$ terraform init
-$ terraform apply
-```
+1. You can follow these [instructions](devops/providers/azure-devops/README.md) to create an cloud-based CI pipeline definition.
+2. Our cloud deployment templates provide an configurable default setup intended for a t-shirt sized environment scenario. Pick the template folder most relevant to your use-case as a starting point. Each template folder is shipped with setup instructions.
+3. It's important to implement quality assurance that validates E2E functional assertions against your infrastructure resources. Each template comes pre-packaged with some basic integration and unit tests. We encourage you to define integration tests in the `test/integration` folder of your template that's specific to your use-case.
+4. Follow these [instructions](test-harness/README.md) to setup your local environment.
+5. Create a new local git branch and commit your changes. Run the test harness on your localhost via `./test-harness/local-run.sh`.
 
-### Environmental Variables 
+## Community
 
-To stop the command line from prompting questions use a .env file with the following environmental variables:
+[Please join us on Slack](https://publicslack.com/slacks/https-bedrockco-slack-com/invites/new) for discussion and/or questions.
 
-```
-export TF_VAR_app_name=cblt
-export TF_VAR_org=cse
-export TF_VAR_env=dev
-export TF_VAR_location=eastus
-```
-
-After saving the file set environment using:
-
-``` bash
-. .env
-```
+## Contributing
 
-Alternative use the variable.tf files in the directories and add the default key on the file as shown on the example below:
+We do not claim to have all the answers and would greatly appreciate your ideas and pull requests.
 
-```
-variable "location" {
-    type = "string"
-    description = "The name of the target location"
-    default = "eastus"
-}
-variable "env" {
-    type = "string"
-    description = "The short name of the target env (i.e. dev, staging, or prod)"
-    defailt = "dev"
-}
-variable "org" {
-    type = "string"
-    description = "The short name of the organization"
-    default = "cse"
-}
-variable "app_name" {
-    type = "string"
-    description = "The short name of the application"
-    default = "cblt"
-}
-```
-
-## Setup Application Infrastructure
-
-> Coming soon!
-
-
-# Contributing
-
-This project welcomes contributions and suggestions.  Most contributions require you to agree to a
+This project welcomes contributions and suggestions. Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
 the rights to use your contribution. For details, visit https://cla.microsoft.com.
 
@@ -124,4 +94,6 @@ provided by the bot. You will only need to do this once across all repos using o
 
 This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
-contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+
+For project level questions, please contact [Erik Schlegel](mailto:erisch@microsoft.com) or [James Nance](mailto:james.nance@microsoft.com).

test-harness/README.md

@@ -2,15 +2,54 @@
 
 ## Summary
 
-This section describes how to build integration and validation tests for your cobalt deployment environments using the terratest modules.
+This section describes how to build integration and validation tests for your cobalt deployment environments using docker and the terratest modules.
 
 Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. It provides a variety of helper functions and patterns for common infrastructure testing tasks.
 
 In addition, the cobalt test suite allows for better collaboration with embedding into CI/CD tools such as Travis or Azure DevOps Pipelines.
 
+This test harness runs automated tests for only the deployment templates that have changed by comparing the changes in your git log versus upstream master.
+
 ## Prerequisites
 - [Docker](https://docs.docker.com/install/) 18.09 or later
+- An Azure subscription
+- A [service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal)
+- An azure storage account for tracking terraform remote backend state. You can use our backend state setup [template]((/infra/templates/backend-state-setup/README.md)) to provision the storage resources.
+- [git](https://www.atlassian.com/git/tutorials/install-git)
 
 ## Test Setup Locally
 
-In this example we are using the [`azure-simple`](/infra/templates/azure-simple/readme.md) for a template integration test.
+1. Local Environment Setup
+    - You'll need to define a `.env` file in the root of the project. You can use our environment template file to start. `cp .env.template .env`
+    - Provide values for the environment values in `.env` which are required to authenticate Terraform to provision resources within your subscription.
+
+        ```shell
+        ARM_SUBSCRIPTION_ID="<az-service-principal-subscription-id>"
+        ARM_CLIENT_ID="<az-service-principal-client-id>"
+        ARM_CLIENT_SECRET="<az-service-principal-auth-secret>"
+        ARM_TENANT_ID="<az-service-principal-tenant>"
+        ARM_ACCESS_KEY="<remote-state-storage-account-primary-key>"
+        ```
+
+2. Base Image - Our test harness uses a base docker image to pre-package dependencies like Terraform, Go, Azure CLI, Terratest vendor packages, etc.
+
+    - **Optional Step** - Cobalt uses the public [msftcse](https://cloud.docker.com/u/msftcse/repository/docker/msftcse/terratest) base image by default. We also provide a utility script to generate a new base image.
+    - Rebuilding a new base image is as simple as running
+
+        ```script
+        ./test-harness/build-base-image.sh -g "<go_version>" -t "<terraform_version>"
+        ```
+
+        Keep in mind that the terraform version should align with the version from the provider [module](/infra/modules/providers/azure/provider/main.tf#L6)
+    - The base image will be tagged as
+
+        ```script
+        msftcse/cobalt-test-base:g${GO_VERSION}t${TERRAFORM_VERSION}
+        ```
+
+3. **Local Run** - Run the test runner by calling the below script from the project's root directory.
+    - Test Harness Script
+
+        ```script
+        ./test-harness/local-run.sh
+        ```