Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor backend and generic resources to support wider range of resources #299

Merged
merged 6 commits into from
Sep 18, 2020
Merged

Refactor backend and generic resources to support wider range of resources #299

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5984056
Add resource_uri parameter to support wider ranges of resources
Sep 17, 2020
3ef950f
Remove redundant #collect_resources method
Sep 17, 2020
8455b80
Fix a typo
Sep 18, 2020
43d797b
Update warning message
Sep 18, 2020
0c3fda0
Add resource_groups and policy_definitions
Sep 18, 2020
4520418
Revert "Add resource_groups and policy_definitions"
Sep 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 30 additions & 11 deletions docs/resources/azure_generic_resource.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,16 +31,18 @@ where

The following parameters can be passed for targeting a specific Azure resource.

| Name | Description |
|-------------------|----------------------------------------------------------------------------------------------------------|
| resource_group | Azure resource group that the targeted resource has been created in. `MyResourceGroup` |
| name | Name of the Azure resource to test. `MyVM` |
| resource_provider | Azure resource provider of the resource to be tested. `Microsoft.Compute/virtualMachines` |
| resource_path | Relative path to the resource if it is defined on another resource. Resource path of a subnet in a virtual network would be: `{virtualNetworkName}/subnets`. |
| resource_id | Unique id of Azure resource to be tested. `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachines/{vmName}` |
| Name | Description |
|--------------------------------------|----------------------------------------------------------------------------------------------------------|
| resource_group | Azure resource group that the targeted resource has been created in. `MyResourceGroup` |
| name | Name of the Azure resource to test. `MyResourceName` |
| resource_provider | Azure resource provider of the resource to be tested. `Microsoft.Compute/virtualMachines` |
| resource_path | Relative path to the resource if it is defined on another resource. Resource path of a subnet in a virtual network would be: `{virtualNetworkName}/subnets`. |
| resource_id | Unique id of Azure resource to be tested. `/subscriptions/{subscriptionId}/resourceGroups/{resourceGroup}/providers/Microsoft.Compute/virtualMachines/{vmName}` |
| resource_uri | Azure REST API URI of the resource to be tested. This parameter should be used when a resource does not reside in a resource group. It requires `add_subscription_id` and `name` parameters to be provided together. `/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/policyDefinitions/` |
| add_subscription_id | Indicates whether the `resource_uri` contains the subscription id. `true` or `false` |
| tag_name<superscript>*</superscript> | Tag name defined on the Azure resource. `name` |
| tag_value | Tag value of the tag defined with the `tag_name`. `external_linux` |
| api_version | API version to use when interrogating the resource. If not set or the provided api version is not supported by the resource provider then the latest version for the resource provider will be used. `2017-10-9`, `latest`, `default` |
| tag_value | Tag value of the tag defined with the `tag_name`. `external_linux` |
| api_version | API version to use when interrogating the resource. If not set or the provided api version is not supported by the resource provider then the latest version for the resource provider will be used. `2017-10-9`, `latest`, `default` |

<superscript>*</superscript> When resources are filtered by a tag name and value, the tags for each resource are not returned in the results.

Expand All @@ -50,9 +52,10 @@ Either one of the parameter sets can be provided for a valid query:
- `name`
- `resource_group`, `resource_provider` and `name`
- `resource_group`, `resource_provider`, `resource_path` and `name`
- `add_subscription_id`, `resource_uri` and `name`
- `tag_name` and `tag_value`

Different parameter combinations can be tried. If it is not supported either the InSpec resource or the Azure Rest API will raise an error.
Different parameter combinations can be tried. If it is not supported, either the InSpec resource or the Azure Rest API will raise an error.

If the Azure Resource Manager endpoint returns multiple resources for a given query, this singular generic resource will fail. In that case, the [plural generic resource](azure_generic_resources.md) should be used.

Expand All @@ -72,7 +75,7 @@ The following properties are applicable to almost all resources.
| tags | The tag `key:value pairs` if defined on the resource. |
| properties | The resource properties. |

For more properties, refer to [Azure documents](https://docs.microsoft.com/en-us/rest/api/resources/resources/list#genericresourceexpanded).
For more properties, refer to specific Azure documents for the resource being tested.

## Examples

Expand Down Expand Up @@ -110,6 +113,22 @@ describe azure_generic_resource(resource_provider: 'Microsoft.DevTestLab/labs',
its('properties.allowClaim') { should cmp false }
end
```
### Test a Resource Group
```ruby
describe azure_generic_resource(add_subscription_id: true, resource_uri: '/resourcegroups/', name: 'my_group') do
it { should exist }
its('tags') { should include(:owner) }
its('tags') { should include(owner: 'John Doe') }
end
```
### Test a Policy Definition
```ruby
describe azure_generic_resource(add_subscription_id: true, resource_uri: 'providers/Microsoft.Authorization/policyDefinitions', name: 'my_policy') do
it { should exist }
its('properties.policyRule.then.effect') { should cmp 'deny' }
its('properties.policyType') { should cmp 'Custom' }
end
```
For more examples, please see the [integration tests](/test/integration/verify/controls/azure_generic_resource.rb).

## Matchers
Expand Down
17 changes: 14 additions & 3 deletions docs/resources/azure_generic_resources.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ All of them are optional.
| resource_provider | Azure resource provider of the resources to be tested. | `Microsoft.Compute/virtualMachines` |
| tag_name<superscript>*</superscript> | Tag name defined on the Azure resources. | `name` |
| tag_value | Tag value of the tag defined with the `tag_name`. | `external_linux` |
| resource_uri | Azure REST API URI of the resources to be tested. This parameter should be used when resources do not reside in resource groups. It requires `add_subscription_id` parameter to be provided together. `/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/policyDefinitions/` |
| add_subscription_id | Indicates whether the `resource_uri` contains the subscription id. `true` or `false` |

<superscript>*</superscript> When resources are filtered by a tag name and value, the tags for each resource are not returned in the results.

Expand All @@ -52,6 +54,7 @@ Either one of the parameter sets can be provided for a valid query:
- `substring_of_resource_group` and `resource_provider`
- `tag_name`
- `tag_name` and `tag_value`
- `add_subscription_id` and `resource_uri`

Different parameter combinations can be tried. If it is not supported either the InSpec resource or the Azure Rest API will raise an error.

Expand All @@ -66,12 +69,14 @@ It is advised to use these parameter sets to narrow down the targeted resources
| tags | A list of `tag:value` pairs defined on the resources. | `tags`|
| types | A list of resource types. | `type`|
| locations | A list of locations where resources are created in. | `location`|
| created_times | A list of created times of the resources. | `created_time`|
| changed_times | A list of changed times of the resources. | `changed_time`|
| provisioning_states | A list of provisioning states of the resources. | `provisioning_state`|
| created_times<superscript>**</superscript> | A list of created times of the resources. | `created_time`|
| changed_times<superscript>**</superscript> | A list of changed times of the resources. | `changed_time`|
| provisioning_states<superscript>**</superscript> | A list of provisioning states of the resources. | `provisioning_state`|

<superscript>*</superscript> For information on how to use filter criteria on plural resources refer to [FilterTable usage](https://github.com/inspec/inspec/blob/master/docs/dev/filtertable-usage.md#a-where-method-you-can-call-with-hash-params-with-loose-matching).

<superscript>**</superscript> These properties are not available when `resource_uri` is used.

## Examples

### Test All Virtual Machines in Your Subscription
Expand Down Expand Up @@ -111,6 +116,12 @@ describe azure_generic_resources.where{ created_time > Time.now - 86400 } do
it { should exist }
end
```
### Test Policy Definitions
```ruby
describe azure_generic_resources(add_subscription_id: true, resource_uri: 'providers/Microsoft.Authorization/policyDefinitions') do
it { should exist }
end
```
Please see [here](https://github.com/inspec/inspec/blob/master/docs/dev/filtertable-usage.md) for more information on how to leverage FilterTable capabilities on plural resources.

For more examples, please see the [integration tests](/test/integration/verify/controls/azure_generic_resources.rb).
Expand Down
85 changes: 50 additions & 35 deletions libraries/azure_backend.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,8 +151,10 @@ def rescue_wrong_api_call(url, params = {})
response = @azure.rest_get_call(url, params)
rescue UnsuccessfulAPIQuery::UnexpectedHTTPResponse::InvalidApiVersionParameter => e
api_version_suggested = e.suggested_api_version(params['api-version'])
Inspec::Log.warn "Incompatible api version: #{params['api-version']}\n"\
unless params['api-version'] == 'failed_attempt'
Inspec::Log.warn "Incompatible api version: #{params['api-version']}\n"\
"Trying with the latest api version suggested by the Azure Rest API: #{api_version_suggested}."
end
if api_version_suggested.nil?
Inspec::Log.warn 'Failed to acquire suggested api version from the Azure Rest API.'
else
Expand Down Expand Up @@ -234,33 +236,22 @@ def construct_resource_id
#
def get_resource(opts = {})
Helpers.validate_parameters(resource_name: @__resource_name__,
require_any_of: %i(resource_uri resource_provider resource_path resource_group),
required: %i(resource_uri),
allow: %i(api_version),
opts: opts)
api_version = opts[:api_version] || 'latest'
argument_error_message = 'Parameters error. For singular resources `resource_uri`; '\
'for plural resources `resource_provider` and/or `resource_path` should be provided.'
if opts.key?(:resource_uri) && opts.keys.any? { |k| %i(resource_provider resource_path).include?(k) }
raise ArgumentError, argument_error_message
end
if opts.key?(:resource_uri)
uri_subdomain = opts[:resource_uri]
else
uri_subdomain = ["/subscriptions/#{@azure.credentials[:subscription_id]}/providers",
opts[:resource_provider],
opts[:resource_path]].compact.join('/').gsub('//', '/')
end
_resource_group, provider, r_type = Helpers.res_group_provider_type_from_uri(uri_subdomain)
# Add resource_group if provided.
unless opts[:resource_group].nil?
uri_subdomain = uri_subdomain.sub('/providers/', "/resourceGroups/#{opts[:resource_group]}/providers/")
if opts[:resource_uri].include?('providers')
# If the resource provider is unknown then this method can't find the api_version.
# The latest api_version will de acquired from the error message via #rescue_wrong_api_call method.
_resource_group, provider, r_type = Helpers.res_group_provider_type_from_uri(opts[:resource_uri])
end
# Some resource names can contain spaces. Decode them before parsing with URI.
url = URI.join(@azure.resource_manager_endpoint_url, uri_subdomain.gsub(' ', '%20'))
url = URI.join(@azure.resource_manager_endpoint_url, opts[:resource_uri].gsub(' ', '%20'))
api_version = api_version.downcase
if %w{latest default}.include?(api_version)
api_version_info = {}
# api_version is not a specific version yet: latest or default.
api_version_info = get_api_version(provider, r_type, api_version)
api_version_info = get_api_version(provider, r_type, api_version) if provider
# Something was wrong at get_api_version, and we will try to get a valid api_version via rescue_wrong_api_call
# by providing an invalid api_version intentionally.
api_version_info[:api_version] = 'failed_attempt' if api_version_info[:api_version].nil?
Expand Down Expand Up @@ -305,20 +296,23 @@ def get_api_version(provider, resource_type, api_version_status = 'latest')
end
return response unless response[:api_version].nil?

# If the resource manager api version is updated earlier, use that.
api_version_mgm = @resource_manager_endpoint_api || @azure.resource_manager_endpoint_api_version
url = Helpers.construct_url([
@azure.resource_manager_endpoint_url,
'subscriptions',
@azure.credentials[:subscription_id], 'providers',
provider
])
provider_details, suggested_api_version = rescue_wrong_api_call(url, { 'api-version' => api_version_mgm })
# If suggested_api_version is not nil, then the resource manager api version should be updated.
unless suggested_api_version.nil?
@resource_manager_endpoint_api = suggested_api_version
Inspec::Log.warn "Resource manager endpoint api version should be updated with #{suggested_api_version} in `libraries/backend/helpers.rb`"
# Use the cached provider details if exist.
if @azure.provider_details[provider.to_sym].nil?
# If the resource manager api version is updated earlier, use that.
api_version_mgm = @resource_manager_endpoint_api || @azure.resource_manager_endpoint_api_version
url = Helpers.construct_url([@azure.resource_manager_endpoint_url, 'subscriptions',
@azure.credentials[:subscription_id], 'providers',
provider])
provider_details, suggested_api_version = rescue_wrong_api_call(url, { 'api-version' => api_version_mgm })
# If suggested_api_version is not nil, then the resource manager api version should be updated.
unless suggested_api_version.nil?
@resource_manager_endpoint_api = suggested_api_version
Inspec::Log.warn "Resource manager endpoint api version should be updated to #{suggested_api_version} in `libraries/backend/helpers.rb`"
end
else
provider_details = @azure.provider_details[provider.to_sym]
end

resource_type_details = provider_details[:resourceTypes].select { |rt| rt[:resourceType] == resource_type }&.first
# For some resource types the api version might be available with their parent resource.
if resource_type_details.nil? && resource_type.include?('/')
Expand All @@ -329,6 +323,8 @@ def get_api_version(provider, resource_type, api_version_status = 'latest')
Inspec::Log.warn "Couldn't get the #{api_version_status} API version for `#{provider}/#{resource_type}`. " \
'Please make sure that the provider/resourceType are in the correct format, e.g. `Microsoft.Compute/virtualMachines`.'
else
# Caching provider details.
@azure.provider_details[provider.to_sym] = provider_details if @azure.provider_details[provider.to_sym].nil?
api_versions = resource_type_details[:apiVersions]
api_versions_stable = api_versions.reject { |a| a.include?('preview') }
api_versions_preview = api_versions.select { |a| a.include?('preview') }
Expand Down Expand Up @@ -377,6 +373,25 @@ def validate_short_desc(resource_list, filter, singular = true)
end
end

def validate_resource_uri
Helpers.validate_params_required(%i(add_subscription_id), @opts)
if @opts[:add_subscription_id] == true
@opts[:resource_uri] = "/subscriptions/#{@azure.credentials[:subscription_id]}/#{@opts[:resource_uri]}"
.gsub('//', '/')
end
end

def validate_resource_provider
# Ensure that the provided resource id is for the correct resource provider.
if @opts.key?(:resource_id) && !@opts[:resource_id].downcase.include?(@opts[:resource_provider].downcase)
raise ArgumentError, "Resource provider must be #{@opts[:resource_provider]}."
end
if @opts.key?(:resource_uri) && !@opts[:resource_uri].downcase.include?(@opts[:resource_provider].downcase)
raise ArgumentError, "Resource provider must be #{@opts[:resource_provider]}."
end
true
end

# Get the paginated result.
# The next_link url won't be validated since it is provided by the Azure Rest API.
# @see https://docs.microsoft.com/en-us/rest/api/azure/#async-operations-throttling-and-paging
Expand Down Expand Up @@ -416,7 +431,7 @@ def specific_resource_constraint(resource_provider, opts)
# This should be used to ensure to fail the resources properly if they can not be created.
def catch_failed_resource_queries
yield
# Inform user if it is an API incompatibility issue and recommend how to solve it.
# Inform user if it is an API incompatibility issue and recommend how to solve it.
rescue UnsuccessfulAPIQuery::UnexpectedHTTPResponse::InvalidApiVersionParameter => e
api_version_suggested_list = e.suggested_api_version
message = "Incompatible api version is provided.\n"\
Expand Down Expand Up @@ -652,7 +667,7 @@ def include?(opt)
end
if opt.is_a?(Hash)
raise ArgumentError, 'Only one item can be provided' if opt.keys.size > 1
return @item[opt.keys.first] == opt.values.first
return @item[opt.keys.first&.to_sym] == opt.values.first
end
@item.key?(opt.to_sym)
end
Expand Down
Loading