At its core, terraform-validator is a thin layer on top of config-validator, a shared library that takes in a policy library and a set of CAI assets and reports back any violations of the specified policies.
terraform-validator consumes a Terraform plan and uses it to build CAI Assets, which then get run through config-validator. These built Assets only exist locally, in memory.
If an existing bundle (for example, CIS v1.1) doesn't support a check you need, please consider contributing a new constraint template to the policy-library repository.
The first step in determining if a GCP resource is supported is to figure out the name of the corresponding Terraform resource. You can often do this by searching for the GCP resource name in the Terraform google provider documentation.
You can run gcloud asset list to list existing assets and their details. For each asset, the output shows its ancestors, asset type, name, and the last updated time. For example,
$ gcloud asset list --project='my-project'
---
ancestors:
- projects/999999
- folders/888888
- organizations/777777
assetType: compute.googleapis.com/Project
name: //compute.googleapis.com/projects/my-project
updateTime: '2022-02-22T22:00:20.265968Z'
...
The output can help validate Name, Type, Ancestry attribute of a converted CAI asset.
Note that you may need to create an actual resource in GCP if the asset you would like to list does not exist. Moreover, you will need to have relevant permission to proceed, especially to listing assets in folders and organizations.
A resource is "supported" by terraform-validator if it has an entry in converters/google/resources/resource_converters.go. For example, you could search resource_converters.go for google_compute_disk to see if that resource is supported.
Adding support for a resource has four steps:
- Make changes to Magic Modules to add or modify resource conversion code. Run Magic Modules generation to update your local terraform validator.
- Add tests for the new resource into Magic Modules.
- Make PRs for Magic Modules with your changes.
Each of these is discussed in more detail below.
Note: terraform-validator can only support resources that are supported by the GA terraform provider, not beta resources.
Magic Modules uses a shared code base to generate terraform-validator and the google and google-beta Terraform providers.
Most Terraform resources are represented as yaml files which are grouped by product.
Each product has an api.yaml file (which defines the basic API schema) and a terraform.yaml file (which defines any terraform-specific overrides.)
A terraform.yaml file can specify exclude_validator: true on a resource to skip terraform-validator autogeneration, or exclude_resource: true to skip autogeneration for both terraform-validator and the providers.
Auto-generating terraform-validator code based on yaml files is strongly preferred. If this does not automatically add an entry to resource_converters.go, you can manually add an entry to resource_converters.go.erb using the autogenerated resource converter function.
If an autogenerated converter is not possible, you can instead place a handwritten file in the magic-modules/mmv1/third_party/validator folder.
Most resources will only need a resource converter with a conversion func. For the Resource resource within Product, this might look like:
// The type comes from https://cloud.google.com/asset-inventory/docs/supported-asset-types
const ProductResourceAssetType string = "compute.googleapis.com/Route"
func resourceConverterProductResource() ResourceConverter {
return ResourceConverter{
AssetType: ProductResourceAssetType,
Convert: GetProductResourceCaiObject,
}
}
func GetProductResourceCaiObject(d TerraformResourceData, config *Config) ([]Asset, error) {
// This function does basic conversion of a Terraform resource to a CAI Asset.
// The asset path (name) will substitute in variables from the Terraform resource.
// The format should match what is specified at https://cloud.google.com/asset-inventory/docs/supported-asset-types
name, err := assetName(d, config, "//whatever.googleapis.com/projects/{{project}}/whatevers/{{name}}")
if err != nil {
return []Asset{}, err
}
if obj, err := GetProductResourceApiObject(d, config); err == nil {
return []Asset{{
Name: name,
Type: ProductResourceAssetType,
Resource: &AssetResource{
Version: "v1", // or whatever the correct version is
DiscoveryDocumentURI: "https://www.googleapis.com/path/to/rest/api/docs",
DiscoveryName: "Whatever", // The term used to refer to this resource by the official documentation
Data: obj,
},
}}, nil
} else {
return []Asset{}, err
}
}
func GetProductResourceApiObject(d TerraformResourceData, config *Config) (map[string]interface{}, error) {
obj := make(map[string]interface{})
// copy values from the terraform resource to obj
// return any errors encountered
// ...
return obj, nil
}For IAM resources, the resource converter should include IAM policy, IAM member and IAM binding. The convert functions are similar to the above example. In addition, the converter needs to add the merge functions and fetch full resource functions. This might look like
func resourceConverterProductResourceIamPolicy() ResourceConverter {
return ResourceConverter{
// The type comes from https://cloud.google.com/asset-inventory/docs/supported-asset-types
AssetType: "whatever.googleapis.com/asset-type",
Convert: GetProductResourceIamPolicyCaiObject,
MergeCreateUpdate: MergeProductResourceIamPolicy,
}
}
func resourceConverterProductResourceIamMember() ResourceConverter {
return ResourceConverter{
// The type comes from https://cloud.google.com/asset-inventory/docs/supported-asset-types
AssetType: "whatever.googleapis.com/asset-type",
Convert: GetProductResourceIamMemberCaiObject,
FetchFullResource: FetchProductResourceIamPolicy,
MergeCreateUpdate: MergeProductResourceIamMember,
MergeDelete: MergeProductResourceIamMemberDelete,
}
}
func resourceConverterProductResourceIamBinding() ResourceConverter {
return ResourceConverter{
// The type comes from https://cloud.google.com/asset-inventory/docs/supported-asset-types
AssetType: "whatever.googleapis.com/asset-type",
Convert: GetProductResourceIamBindingCaiObject,
FetchFullResource: FetchProductResourceIamPolicy,
MergeCreateUpdate: MergeProductResourceIamBinding,
MergeDelete: MergeProductResourceIamBindingDelete,
}
}
func MergeProductResourceIamPolicy(existing, incoming Asset) Asset {
existing.IAMPolicy = incoming.IAMPolicy
return existing
}
func MergeProductResourceIamBinding(existing, incoming Asset) Asset {
return mergeIamAssets(existing, incoming, mergeAuthoritativeBindings)
}
func MergeProductResourceIamBindingDelete(existing, incoming Asset) Asset {
return mergeDeleteIamAssets(existing, incoming, mergeDeleteAuthoritativeBindings)
}
func MergeProductResourceIamMember(existing, incoming Asset) Asset {
return mergeIamAssets(existing, incoming, mergeAdditiveBindings)
}
func MergeProductResourceIamMemberDelete(existing, incoming Asset) Asset {
return mergeDeleteIamAssets(existing, incoming, mergeDeleteAdditiveBindings)
}
func FetchProductResourceIamPolicy(d TerraformResourceData, config *Config) (Asset, error) {
// Check one or more identity fields are available
if _, ok := d.GetOk("some-identity"); !ok {
return Asset{}, ErrEmptyIdentityField
}
return fetchIamPolicy(
NewProductResourceApiObjectUpdater,
d,
config,
// Asset name template, the format should match what is specified at https://cloud.google.com/asset-inventory/docs/supported-asset-types
"//whatever.googleapis.com/projects/{{project}}/whatevers/{{name}}",
// The type comes from https://cloud.google.com/asset-inventory/docs/supported-asset-types
"whatever.googleapis.com/asset-type",
)
}After creating the handwritten file, you will need to add a copy entry in magic-modules/mmv1/provider/terraform_validator.rb specifying the handwritten file path and the desired destination in terraform-validator repository.
If your handwritten file is a .erb file, add an entry into the compile_file_list parameter inside function def compile_common_files.
compile_file_list(output_folder, [
...
['converters/google/resources/[YOUR-FILE].go', # destination in terraform-validator
'third_party/[PATH]/[YOUR-FILE].go.erb'], # your handwritten file location
...If your handwritten file is a .go file, add an entry into the copy_file_list list inside function def copy_common_files.
copy_file_list(output_folder, [
...
['converters/google/resources/[YOUR-FILE].go', # destination in terraform-validator
'third_party/validator/[YOUR-FILE].go'], # your handwritten file location
...You will also need to add an entry to resource_converters.go.erb, which is used to generate converters/google/resources/resource_converters.go. For IAM resource, you will need to add entries each for IAM policy, IAM binding, and IAM member. Each entry in resource_converters.go.erb maps a terraform resource name to a function that returns a ResourceConverter - in this case:
// ...
"google_product_resource": resourceConverterProductResource(),
// ...To generate terraform-validator code locally, run the following from the root of the magic-modules repository:
make validator OUTPUT_PATH="/path/to/your/terraform-validator"
You can then run make test inside your terraform-validator repository to make sure that your code compiles and passes basic unit tests.
Terraform Validator tests require setting up a few files in testdata/templates. Using the previous example, these files would be:
- example_product_resource.tf
- A basic terraform file that creates the minimum resources necessary for the google_product_resource resource.
- example_product_resource.tfplan.json
- A plan json file generated from example_product_resource.tf.
- example_product_resource.json
- The results of running
terraform-validator convert example_product_resource.tfplan.json
- The results of running
It's easiest to set up a test project to create the initial versions of these files. The idea is to use the terraform-validator binary to invoke a convert operation, and replace the strings specific to your test project in the generated files. The followings are typical steps that you can take:
- Run
make buildto compile the terraform-validator binary. - Create
example_product_resource.tf, where the content is the resource that you would like to test, and place it in a new folder. - Within the new folder, run these commands to get resource files in json format.
terraform init
terraform plan -out=example_product_resource.tfplan
terraform show -json example_product_resource.tfplan > example_product_resource.tfplan.json- Using the newly compiled binary, run
terraform-validator convert example_product_resource.tfplan.json --project=[YOUR-PROJECT] > example_product_resource.jsonOnce you have initial versions completed, you need to make the following replacements in the .tfplan.json and .json files:
- Test project ancestry =>
{{.Ancestry}}/project/{{.Provider.project}} - Test project id (without ancestry) =>
{{.Provider.project}} - Test project number =>
{{.Project.Number}} - Test organization id =>
{{.OrgID}} - Test folder id =>
{{.FolderID}} - Test billing account =>
{{.Project.BillingAccountName}}
Add your template test files, eg. example_product_resource.tf, example_product_resource.tfplan.tfplan.json, example_product_resource.json to folder magic-modules/mmv1/third_party/validator/tests/data.
By default, the Terraform Validator tests will compare the expected example_product_resource.json result with the actual result from the test environment, and fail if they do not match. If you need more control over how matching is done (eg. check that an array contains a value), you can use a custom compare function, and configure the test in cli_test.go.erb and read_test.go.erb. This is currently being done for *_iam_member and *_iam_binding resources in order to ignore preexisting premissions in the test environment.
Run the following from the root of the magic-modules repository again, it should auto-generate an entry with the name of your files (i.e. example_product_resource) to the lists of test cases in test/cli_test.go and test/read_test.go, along with your test files copied to terraform-validator repository.
make validator OUTPUT_PATH="/path/to/your/terraform-validator"
Now run your tests and make sure they pass locally before proceeding. (But you can also go ahead and open PRs if you're running into issues you can't figure out how to resolve.)
Now that you have your code working locally, open a PR for Magic Modules.
For the Magic Modules PR, the most important check is terraform-validator-test - the other checks only matter if you're also making changes to the terraform provider.
If the terraform-validator-test is failing, make sure you can run the unit and integration tests successfully locally.