This document is intended to be an introduction to contributing to the terraform-provider-azuredevops
project. Links to background information about the project and general guidance on Terraform providers are included below:
If you are looking for background information on the project or related technologies (Terraform, Go and Azure DevOps), consider checking out some of these resources first:
- Introduction to Azure DevOps
- Getting started with Terraform
- Getting started with Go
- README.md for project
If you are familiar with the technologies used for this project but are looking for general guidance on Terraform provider development, consider checking out some of these resources first:
- Introduction to Provider Development
- Terraform provider discovery documentation
- Terraform Acceptance Testing
- Terraform Schema Behaviors
If you are still reading, then you are in the right place!
If, after reading through the content here, you are seeking more detailed information, you may want to checkout some of the following resources:
- Getting Started Guide written for the
terraform-provider-azurerm
provider. While it targets a different provider there are some great findings that you can read about. - Development Environment for Go Lang on Mac
This section describes how to get your developer workspace running for the first time so that you're ready to start making contributions. If you have already done this, check out Development SDLC.
These steps assume you are running with
bash
. If you are using Windows, run all commands using WSL. They are not tested using GitBash.
To develop using docker:
Note: It is highly recommended to use VSCode as described for rich intellisense, debugging, and access to standard go development tools.
- Clone the repo:
git clone https://github.com/microsoft/terraform-provider-azuredevops.git
If using VSCode's Remote Development Extension:
- Open the project in VSCode
- When prompted by VSCode, select "Reopen in Container"
- In the command palette (ctrl+shift+p on Windows/Linux, command+shift+p on Mac) type:
Go: Install/Update tools
and select it - Note: SSH credentials are automatically mapped into the container
- When committing, you will be prompted to configure git
user.name
anduser.email
Without VSCode:
- Build the development environment:
docker build -f .devcontainer/Dockerfile -t dev .
- From bash or powershell, mount the root directory into the docker container, and drop into a bash shell in the container:
docker run -it -v ${PWD}:/workspaces/terraform-provider-azuredevops dev
Continue with the guide to run the provider locally.
The recommended development environment is Linux or Mac. If you're on Windows you should install WSL so that your environment more closely mirrors a Linux environment.
You will need the following dependencies installed in order to get started:
- Terraform version 0.11.x +
- Go version 1.12.x +
- An editor of your choice. We recommend Visual Studio Code but any editor will do.
Note: This project uses Go Modules making it safe to work with it outside of your existing GOPATH. The instructions that follow assume a directory in your home directory outside of the standard
GOPATH
.
Note These steps assume you are running with bash
. If you are using Windows, run all commands using WSL. They are not tested using GitBash.
If you are using Go 1.13+ , you can get this below error during the execution of the provided compilation script
verifying golang.org/x/mod@v0.1.0: golang.org/x/mod@v0.1.0: open /mnt/d/Go/pkg/sumdb/sum.golang.org/latest: no such file or directory
As mentionned in the Changelog 1.13 https://golang.org/doc/go1.13#modules For solve it, run in your terminal the commands:
go env -w GOPROXY=direct
go env -w GOSUMDB=off
Setup your workspace
$ DEV_ROOT="$HOME/workspace"
$ mkdir -p "$DEV_ROOT"
$ cd "$DEV_ROOT"
Get the code
$ git clone https://github.com/microsoft/terraform-provider-azuredevops.git
$ cd terraform-provider-azuredevops/
To build and install the provider locally execute the make install
command which will execute the build.sh --Install --SkipTests
command for you.
$ make install
For more information about the process see below.
Build & test the azure devops provider
Running the build.sh
script will orchestrate a few things for you:
- Verify that all required Go packages are installed. This may take a few minutes the first time you run the script as the packages will be cached locally. Subsequent runs will be much faster
- Run all unit tests
- Compile the provider codebase
$ ./scripts/build.sh
...
[INFO] Executing unit tests
...
[INFO] Build finished successfully
After this script runs you should see a ./bin/
directory with the compiled terraform provider.
The build.sh
suppports the following parameters:
-s | --SkipTests
: Skip executing the unit tests
-d | --DebugBuild
: Execute go build with flags that disables optimization and adds debug symbols to the created binary
-i | --Install
: After successful build perform a local install of the provider by executing local-install.sh
$ ls -lah ./bin/
...
-rwxrwxrwx 1 ... terraform-provider-azuredevops_v0.0.1
Install the provider
Terraform provider plugins are not intended to be run directly. You can see this for yourself:
$ ./bin/terraform-provider-azuredevops_v0.0.1
This binary is a plugin. These are not meant to be executed directly.
Please execute the program that consumes these plugins, which will
load any plugins automatically
In order to use the provider locally it must be installed into a location discoverable by Terraform. The local-install.sh
script does this for you.
$ ./scripts/local-install.sh
[INFO] Installing provider to /home/$USER/.terraform.d/plugins/
To learn more about the plugin discovery process, refer to the official Terraform provider discovery documentation.
Note: These steps assume you have built the provider locally using the previous steps. The samples in the
./examples/
folder use syntax specific to Terraform 12 + and are not compatible with older versions of Terraform.
Configure the provider
You can now use the provider just like you normally would. Try it out by using the project examples:
$ cd examples/github-based-cicd-simple/
# AZDO_ORG_SERVICE_URL will be the URL of the AzDO org that you want to provison
# resources inside of.
# ex: https://dev.azure.com/<your org name>
$ export AZDO_ORG_SERVICE_URL="..."
# AZDO_PERSONAL_ACCESS_TOKEN will be the personal access token that grants access
# to provision and manage resources in Azure DevOps.
# You'll need to configure permissions for AzDO and Github PATS detailed in ./website/docs
# documentation: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops
$ export AZDO_PERSONAL_ACCESS_TOKEN="..."
# Note: AZDO_GITHUB_SERVICE_CONNECTION_PAT is not specifically required
# by the provider, but it is required by the example in this folder.
# documentation: https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line
$ export AZDO_GITHUB_SERVICE_CONNECTION_PAT="..."
$ terraform init
...
Terraform has been successfully initialized!
...
The provider has now been initialized and can be used like any other provider in Terraform. You can try for yourself by running any of the terraform plan|apply|destroy|...
commands.
This section outlines typical development scenarios for the repository, relevant areas of the codebase and provides guidance for writing tests and documentation for your changes.
Please find an open issue in the backlog that has no asignee and request it to be assigned to you by a core contributor if you don't have the permissions to do it yourself. This will help us track which work items are being worked on.
The repository generally follows the patterns suggested by HashiCorp. This pattern is implemented by all major terraform providers.
Here is a simplified version of the repository:
$ tree -L 2
.
├── azdosdkmocks --------> Generated mocks for AzDO Go SDK
├── azuredevops ---------> Provider implementation
│ ├── config.go -------> AzDO SDK initialization lives here
│ ├── provider.go -----> Exports the AzDO terraform provider
│ ├── data_*.go -------> data_*.go files contain terraform data sources implementations
│ ├── resource_*.go ---> resource_*.go files contain terraform resource implementations
│ └── utils -----------> Utilities used across the codebase
├── docs ----------------> Developer documentation
├── go.mod --------------> Describes project dependencies
├── scripts -------------> All scripts live here
└── website -------------> Client facing documentation
After you picked an issue and figured out where you will implement it, you will quickly realize that HashiCorp has taken a very opinionated approach to building Terraform providers. The following section will outline a few common scenarios for terraform plugin development, and point you towards different pieces of the code that you may need to work with.
If you have not already gone through the guide published by HashiCorp, now is a good time to do so.
Scenario 1: Change resource or data source schema
If you need to add, remove or modify the schema of a data source or resource, you will need to first identify the relevant file. The naming scheme used is as follows:
data_foo.go
- an implementation for the Terraform data source for the foo Azure DevOps resourceresource_foo.go
- an implementation for the Terraform resource for the foo Azure DevOps resource
Open the file and look for the schema. Here is a simple schema found in data_group.go
. This is fairly simple and only defines three attributes. More complicated ones can be found in the build definition code. The official documentation for the schema can be found here.
Scenario 2: Modify an existing resource or data source
If you need to modify the business logic in an existing resource or data source, you will find the relevant code in one of the Create
, Read
, Update
or Delete
functions.
The prototype of these functions are all quite similar. Here is an example of a create function. Keep note of the following details:
d *schema.ResourceData
is passed to the provider by Terraform. It contains the resource configuration specified by the client using the provider, along with any data pulled from the Terraform state.m interface{}
is, in the case of this provider, a structure containing all of the (intialized) clients needed to make API calls to Azure DevOps.- Flatten/Expand is a common "idiom" used across terraform providers. It is a standard approach to marshaling and unmarshaling API data structures into the internal terraform state.
Scenario 3: Implement a new resource or data source
If you need to implement a new resource or data source, you should first review a few implementations in order to understand the patterns in the codebase, which generally match the way that other Terraform providers are built.
This scenario is a mix of Scenario 1 and Scenario 2. However, after implementing the schema and CRUD operations, you will need to register your newly created resource or data source. The code for doing this is located in provider.go
.
In order to accelerate development and ensure a common structure of all components (resource, data source), the project offers several Visual Studio Code snippets that should preferably be used to create new code.
General information about how to work with snippets inside Visual Studio Code are available in the official documentation.
If you are going to create a new data source care for the wording of your Go function and the Terraform data source name. If your data source is able to return more than one entity, be sure to use the plural form of the object you're returning with the data source.
Example: returning multiple groups from Azure DevOps
Because the data source returns more than one group object from Azure DevOps
- Data source name
azuredevops_groups
- Go function name
func dataGroups() *schema.Resource
Reference: Terraform Data Source Names
If the new data source is able to return more than one object, be sure you are using a schema.TypeSet
to return the elements.
Example: returning multiple projects from Azure DevOps as set with azuredevops_projects
"projects": {
Type: schema.TypeSet,
Computed: true,
Set: getProjectHash,
Elem: &schema.Resource{
Schema: map[string]*schema.Schema{
"name": {
Type: schema.TypeString,
Computed: true,
},
"project_id": {
Type: schema.TypeString,
Computed: true,
},
"project_url": {
Type: schema.TypeString,
Computed: true,
},
"state": {
Type: schema.TypeString,
Computed: true,
},
},
},
},
Managing resources in the cloud often means using APIs that return a state even though the affected resource has not yet reached the expected state. One might be tempted to use the Sleep
function to wait for the resource to be in the correct state for further actions. But that is considered bad practice, because the time how long a cloud resource needs to reach it's final (expected) state isn't really predictable since the deployment, removal or change of a cloud resource could depend on many factors, like replication, database locks, runtime etc. that, as said, the time how long the client must wait for a resource reaching a state isn't foreseeable.
Thus, as a provider developer, you should take delays in resource APIs into account in the CRUD functions of the resource. Terraform supports configurable timeouts to assist in these situations.
Terraform provides the following mechanisms to wait for a state change of a resource:
-
Using the
retry
mechanism of theschema.Resource
(Retry) -
Using the
WaitForState
function ofresource.StateChangeConf
(StateChangeConf)
Both means, if implemented correctly, guarantee that subsequent actions within the provider implementation can be confident that the affected resource has reached an expected state.
Shortcut: tf-azdo-rs
Purpose: Create new resource
How to use:
- Create a new
.go
file, name it according to the naming scheme and save it. - Type
tf-azdo-rs
+[Ctrl+Space]
. The snippet will prompt you for the new name of the resource. - Press
[ESC]
- Save the file again
Shortcut: tf-azdo-ds
Purpose: Create new data source
How to use:
- Create a new
.go
file, name it according to the naming scheme and save it. - Type
tf-azdo-ds
+[Ctrl+Space]
. The snippet will prompt you for the new name of the resource. - Press
[ESC]
- Save the file again
Shortcut: tf-azdo-test
Purpose: Create new test file
How to use:
-
Create a new
_test.go
file, name it according to the naming scheme and save it. -
Type
tf-azdo-test
+[Ctrl+Space]
. The snippet will prompt you for the following parameters:- The name of the resource or data source which is under test
- The function to test
- Test test you want to perform
-
Press
[ESC]
-
Save the file.
Shortcut: tf-azdo-test-func
Purpose: Create new test function
How to use:
-
Open an existing
_test.go
file, navigate to the end of the file -
Type
tf-azdo-test-func
+[Ctrl+Space]
. The snippet will prompt you for the following parameters:- The name of the resource or data source which is under test
- The function to test
- Test test you want to perform
-
Press
[ESC]
-
Save the file.
Running Unit Tests
The unit tests are executed whenever ./scripts/build.sh
is run. This can be run locally, but will also be run on every automated build and will be a gate for any PR against this repository. The tests can also be run in isolation by running the following:
$ ./scripts/unittest.sh
Running Acceptance Tests (Integration Tests)
Note: Running acceptance tests provisions and deletes actual resources in AzDO. This can cost money and can be dangerous if you are not running them in isolation!
The acceptance tests for terraform providers are typically implemented as Acceptance Tests. Acceptance tests can be invoked by running the following:
When generating a new PAT, the acceptance tests require the provisioning of certain resources this token will need to provide permission for creating. For simplicity you can authorize the scope of access associated with this token to
Full Access
with a short expiration date. For custom defined scope, refer to the Azure DevOps Provider: Authenticating using the Personal Access Token document that provides more information on how to configure this correctly.
# AZDO_ORG_SERVICE_URL will be the URL of the AzDO org that you want to provison
# resources inside of.
# ex: https://dev.azure.com/<your org name>
$ export AZDO_ORG_SERVICE_URL="..."
# AZDO_PERSONAL_ACCESS_TOKEN will be the personal access token that grants access
# to provision and manage resources in Azure DevOps.
# documentation: https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops
$ export AZDO_PERSONAL_ACCESS_TOKEN="..."
# Note: AZDO_GITHUB_SERVICE_CONNECTION_PAT is not specifically required
# by the provider, but it is required by the acceptance tests in order to test
# the authentication with GitHub for build definitions hosted in GitHub.
# documentation: https://help.github.com/en/articles/creating-a-personal-access-token-for-the-command-line
$ export AZDO_GITHUB_SERVICE_CONNECTION_PAT="..."
# Note: AZDO_TEST_AAD_USER_EMAIL will be the e-mail address that is not included in the current organization
# ,but needs to be an account in the Azure Active Directory. The e-mail is used for acceptance testing of the User Entitlement resource.
$ export AZDO_TEST_AAD_USER_EMAIL="..."
$ ./scripts/acctest.sh
Writing your own tests
There is a lot of context to cover here, so check out our dedicated testing document for more information on writing unit and acceptance tests.
There is a lot of context to cover here, so check out our dedicated debugging document for more information on debugging the provider.
Most changes should involve updates to the client-facing reference documentation. Please update the existing documentation to reflect any changes you have made to the codebase.
Name | Description | Link |
---|---|---|
Index | Table of contents | index.md |
Resources | Resources reference | resources |
Data Sources | Data Sources reference | data sources |
Guides | Guide and tutorial docs | guides |
Azure DevOps has a rich set of REST API's available for integrating with Terraform. A convenient way to explore the APIs is through postman. Once you have postman installed, you can clone/fork this repo and follow the instructions in the README to setup postman to be able to explore a large set of the rest API's. A second resource that is very helpful in understanding how to call/use the apis is the Azure DevOps Cli extension found here.
After the feature or bug is completed, you'll want to post a PR to get it reviewed by somebody in the core development team. Here are some useful tips to keep in mind:
-
The steps run by the CI are located here. You should manually execute these commands to make sure that they pass before you post your PR.
-
The below command runs everything, but the acceptance tests:
$ make ci
-
-
The PR template for the repository is located here. Please be ready to fill this template out when posting the PR.
-
If your PR is a draft, please mark it as a draft and put
WIP
(stands for "Work In Progress") in the title. This helps reviewers prioritize which reviews to look at. -
If your PR contains changes to the
go.mod
orgo.sum
files, or thevendor/
directory, please elaborate on why the change is being made.
- If a PR contains changes to the
go.mod
orgo.sum
files, or thevendor/
directory, carefully understand them and make sure that these changes are (1) minimal and (2) necessary. This helps the stability of the product over time. Dependency upgrades to get off of old, deprecated or unsupported libraries should be done in isolated PRs, if possible.
If you find an api is missing from the postman collection, please submit a PR. We intend make the collection as complete as possible over time.
You can create a github release of this project or a fork of this project using goreleaser. The gorelease will create binaries for Mac, Linux and Windows based of the fork you're making the release from. To be able to make the release you need to have the required permissions to create a release for this project or a fork.
To make the release for your fork change the
owner
in.goreleaser.yml
to be the name of the account holding the fork.
Making the release:
- Open this project in the dev container or install gorelaser
- Create a Github PAT token with all of the
Repo
permissions - Set
GITHUB_TOKEN
enviorment variable to the above PAT tokenexport GITHUB_TOKEN=<Your PAT token>
- Set the release version you would like to create
RELEASE_VERSION="v0.1.7"
- Create
git tag $RELEASE_VERSION && goreleaser release --rm-dist
More information about using goreleaser here: https://goreleaser.com/
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.