Breaking API change for azurerm_application_insights_web_test - now returns a 201

[heoelri]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.1.9

AzureRM Provider Version

3.5.0

Affected Resource(s)/Data Source(s)

azurerm_monitor_scheduled_query_rules_alert

Terraform Configuration Files

resource "azurerm_monitor_scheduled_query_rules_alert" "game_service_exception_responses" {
  name                = "${local.prefix}-${local.location_short}-AppInsights-CatalogServiceExceptionReponses"
  location            = var.location
  resource_group_name = var.resource_group_name

  data_source_id = azurerm_application_insights.stamp.id
  description    = "Alert will be triggered when CatalogService exception response threshold reached. Every 5xx response to a request counts as an exception"

  enabled = var.alerts_enabled

  # Count all requests with server error result code grouped into 5-minute bins
  query       = <<-QUERY
  requests
  | where cloud_RoleName startswith "CatalogService" and name !contains "Health" and resultCode startswith "5"
  QUERY
  severity    = 1
  frequency   = 5
  time_window = 5

  trigger {
    operator  = "GreaterThan"
    threshold = 10
  }

  action {
    action_group = [
      var.azure_monitor_action_group_resource_id
    ]
  }
}

Debug Output/Panic Output

Error: creating/updating Application Insights Web Test: (Name "ace2e****-appinsights-webtest-api-global" / Resource Group "ace2e****-monitoring-rg"): insights.WebTestsClient#CreateOrUpdate: Failure responding to request: StatusCode=201 -- Original Error: autorest/azure: Service returned an error. Status=201 Code="Unknown" Message="Unknown service error" Details=[{"etag":"\"3d00c71d-0000-4700-0000-628224db0000\"","id":"/subscriptions/afffa704-****-****-****-b473f632ecb5/resourceGroups/ace2e****-monitoring-rg/providers/microsoft.insights/webtests/ace2e****-appinsights-webtest-api-global","kind":"ping","location":"swedencentral","name":"ace2e****-appinsights-webtest-api-global","properties":{"Configuration":{"WebTest":"\u003cWebTest Name=\"ace2e****-appinsights-webtest-api-global\" Id=\"00000000-0000-0000-0000-000000000000\" Enabled=\"True\" CssProjectStructure=\"\" CssIteration=\"\" Timeout=\"30\" WorkItemIds=\"\" xmlns=\"http://microsoft.com/schemas/VisualStudio/TeamTest/2010\" Description=\"\" CredentialUserName=\"\" CredentialPassword=\"\" PreAuthenticate=\"True\" Proxy=\"default\" StopOnError=\"False\" RecordedResultFile=\"\" ResultsLocale=\"\"\u003e\n  \u003cItems\u003e\n    \u003cRequest Method=\"GET\" Guid=\"00000000-0000-0000-0000-000000000000\" Version=\"1.1\" Url=\"https://ace2e****-global-fd.azurefd.net/catalogservice/api/1.0/catalogitem/?limit=1\" ThinkTime=\"0\" Timeout=\"30\" ParseDependentRequests=\"False\" FollowRedirects=\"False\" RecordResult=\"True\" Cache=\"False\" ResponseTimeGoal=\"0\" Encoding=\"utf-8\" ExpectedHttpStatusCode=\"200\" ExpectedResponseUrl=\"\" ReportingName=\"\" IgnoreHttpStatusCode=\"False\" /\u003e\n  \u003c/Items\u003e\n\u003c/WebTest\u003e"},"Description":"","Enabled":true,"Frequency":300,"Kind":"ping","Locations":[{"Id":"us-tx-sn1-azr"},{"Id":"us-il-ch1-azr"},{"Id":"emea-fr-pra-edge"},{"Id":"emea-se-sto-edge"},{"Id":"apac-hk-hkn-azr"},{"Id":"apac-jp-kaw-edge"},{"Id":"latam-br-gru-edge"},{"Id":"emea-au-syd-edge"}],"Name":"ace2e****-appinsights-webtest-api-global","Request":null,"RetryEnabled":false,"SyntheticMonitorId":"ace2e****-appinsights-webtest-api-global","Timeout":30,"ValidationRules":null,"provisioningState":"Succeeded"},"tags":{"hidden-link:/subscriptions/afffa704-****-****-****-b473f632ecb5/resourceGroups/ace2e****-monitoring-rg/providers/Microsoft.Insights/components/ace2e****-global-appi":"Resource"},"type":"microsoft.insights/webtests"}]
│ 
│   with azurerm_application_insights_web_test.api_ping,
│   on monitoring_webtests.tf line 7, in resource "azurerm_application_insights_web_test" "api_ping":
│    7: resource "azurerm_application_insights_web_test" "api_ping" {
│

Expected Behaviour

This should successfully create an AppInsights Availability test. This is the case - but the creation causes an error and fails the terraform apply run.

Actual Behaviour

The execution of terraform apply fails creating this resource.

Steps to Reproduce

No response

Important Factoids

No response

References

• Application Insights API now returns an undocumented 201 Azure/azure-sdk-for-go#2465

[heoelri]

Related to Azure/azure-sdk-for-go#2465

@jhendrixMSFT FYI

[ALTrijssenaar]

Same here! Started to fail on the 201 since last saturday. After that upgraded to TerraForm 1.1.9 and Azure RM Provider 3.6, but is still complaining:

insights.WebTestsClient#CreateOrUpdate: Failure responding to request: StatusCode=201 -- Original Error: autorest/azure: Service returned an error. Status=201 Code="Unknown" Message="Unknown service error"

[Sravs461]

Started to fail on the 201 since last Thursday(May 12th 2022). we are using TerraForm 0.14.11 and Azure RM Provider 3.6, but complaining with :

insights.WebTestsClient#CreateOrUpdate: Failure responding to request: StatusCode=201 -- Original Error: autorest/azure: Service returned an error. Status=201 Code="Unknown" Message="Unknown service error"

[danielrothmann]

We're seeing the same behaviour for the azurerm_application_insights_web_test resource since today. We're using Terraform 0.14.9 and AzureRM provider 3.6.0. Seems like an unexpected change to the contract of the Application Insights API was deployed recently.

[tareks]

Note that this appears to be intermittent. Sometimes despite the 201, the resource is created and the provider will happily succeed. In other instances, it does not and terminates with a failure.

In case of a failure, a retry/re-apply will result in another error because the resource will have been created by then but terraform does not know about it (not in state).

[vap78]

We are experiencing the same issue with azurerm_application_insights_web_test resource. The web test is created but the run fails because the API call returns 201 and not 200.

The proposed workaround - re-running the script - does not work unless I manually update the state. Terraform attempts to update the web test again.

Here is the exact call (anonymized) that fails with Terraform traces set to debug:

PUT /subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxx/resourceGroups/ resource-group/providers/Microsoft.Insights/webtests/sample-test?api-version=2015-05-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.18.1 (amd64-windows) go-autorest/v14.2.1 Azure-SDK-For-Go/v64.0.0 insights/2020-02-02 HashiCorp Terraform/1.0.0 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/dev pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 2230
Content-Type: application/json; charset=utf-8
X-Ms-Correlation-Request-Id: 3c24e9ce-86ec-c953-bc0a-5be1dba367e5
Accept-Encoding: gzip

{"kind":"ping","location":"westeurope","properties":{"Configuration":{"WebTest":"...","Description":"","Enabled":true,"Frequency":300,"Kind":"ping","Locations":[{"Id":"emea-nl-ams-azr"}],"Name":"sample-test","RetryEnabled":true,"SyntheticMonitorId":"sample-test","Timeout":20},"tags":{"hidden-link:/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxx/resourceGroups/resource-group/providers/Microsoft.Insights/components/test-insights-component":"Resource"}}: timestamp=2022-05-16T11:55:14.488+0300
2022-05-16T11:55:16.305+0300 [DEBUG] provider.terraform-provider-azurerm_v3.6.0_x5.exe: AzureRM Response for https://management.azure.com/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxx/resourceGroups/resource-group/providers/Microsoft.Insights/webtests/sample-test?api-version=2015-05-01:
HTTP/2.0 201 Created
Content-Length: 2609
Access-Control-Expose-Headers: Request-Context
Cache-Control: no-cache
Content-Type: application/json; charset=utf-8
Date: Mon, 16 May 2022 08:55:15 GMT
Expires: -1
Pragma: no-cache
Request-Context: appId=cid-v1:7f83c1fe-8c94-4d55-9337-4ddc696f61ed
Server: Microsoft-IIS/10.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 3c24e9ce-86ec-c953-bc0a-5be1dba367e5
X-Ms-Ratelimit-Remaining-Subscription-Writes: 1199
X-Ms-Request-Id: 2f8980e2-aa53-4d08-ab80-dd1eabca3727
X-Ms-Routing-Request-Id: WESTEUROPE:20220516T085515Z:2f8980e2-aa53-4d08-ab80-dd1eabca3727
X-Powered-By: ASP.NET

{
  "id": "/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxx/resourceGroups/resource-group/providers/microsoft.insights/webtests/sample-test",
  "name": "sample-test",
  "type": "microsoft.insights/webtests",
  "location": "westeurope",
  "tags": {
    "hidden-link:/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxx/resourceGroups/resource-group/providers/Microsoft.Insights/components/test-insights-component": "Resource"
  },
  "kind": "ping",
  "etag": "\"bd009b45-0000-0200-0000-628211730000\"",
  "properties": {
    "SyntheticMonitorId": "sample-test",
    "Name": "sample-test",
    "Description": "",
    "Enabled": true,
    "Frequency": 300,
    "Timeout": 20,
    "Kind": "ping",
    "RetryEnabled": true,
    "Locations": [
      {
        "Id": "emea-nl-ams-azr"
      }
    ],
    "Configuration": {
      "WebTest": "<WebTest xmlns=\"http://microsoft.com/schemas/VisualStudio/TeamTest/2010\" Name=\"healtcheck\" Id=\"34718aff-75ca-4569-8a4b-bd1b3832823a\" Enabled=\"True\" CssProjectStructure=\"\" CssIteration=\"\" Timeout=\"20\" WorkItemIds=\"\" Description=\"\" Proxy=\"default\" StopOnError=\"False\" RecordedResultFile=\"\" ResultsLocale=\"\">\n\t<Items>\n\t  <Request Method=\"GET\" Guid=\"a38f9def-f162-141f-a84a-b6e4c74da461\" Version=\"1.1\" Url=\"https://host/health/\" ThinkTime=\"0\" Timeout=\"20\" ParseDependentRequests=\"False\" FollowRedirects=\"True\" RecordResult=\"True\" Cache=\"False\" ResponseTimeGoal=\"0\" Encoding=\"utf-8\" ExpectedHttpStatusCode=\"200\" ExpectedResponseUrl=\"\" ReportingName=\"\" IgnoreHttpStatusCode=\"False\"/>\n\t</Items>\n    \t<ValidationRules>\n\t  <ValidationRule Classname=\"Microsoft.VisualStudio.TestTools.WebTesting.Rules.ValidationRuleFindText, Microsoft.VisualStudio.QualityTools.WebTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\" DisplayName=\"Find Text\" Description=\"Verifies the existence of the specified text in the response.\" Level=\"High\" ExectuionOrder=\"BeforeDependents\">\n\t\t<RuleParameters>\n\t\t  <RuleParameter Name=\"FindText\" Value=\"&quot;status&quot;:&quot;UP&quot;\" />\n\t\t  <RuleParameter Name=\"IgnoreCase\" Value=\"True\" />\n\t\t  <RuleParameter Name=\"UseRegularExpression\" Value=\"False\" />\n\t\t  <RuleParameter Name=\"PassIfTextFound\" Value=\"True\" />\n\t\t</RuleParameters>\n\t  </ValidationRule>\n\t</ValidationRules>\n\n\t</WebTest>"
    },
    "Request": null,
    "ValidationRules": null,
    "provisioningState": "Succeeded"
  }
}: timestamp=2022-05-16T11:55:16.305+0300
╷
│ Error: creating/updating Application Insights Web Test: (Name "sample-test" / Resource Group "resource-group"): insights.WebTestsClient#CreateOrUpdate: Failure responding to request: StatusCode=201 -- Original Error: autorest/azure: Service returned an error. Status=201 Code="Unknown" Message="Unknown service error" Details=[{"etag":"\"bd009b45-0000-0200-0000-628211730000\"","id":"/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxx/resourceGroups/resource-group/providers/microsoft.insights/webtests/sample-test","kind":"ping","location":"westeurope","name":"sample-test","properties":{"Configuration":{"WebTest":"\u003cWebTest xmlns=\"http://microsoft.com/schemas/VisualStudio/TeamTest/2010\" Name=\"healtcheck\" Id=\"34718aff-75ca-4569-8a4b-bd1b3832823a\" Enabled=\"True\" CssProjectStructure=\"\" CssIteration=\"\" Timeout=\"20\" WorkItemIds=\"\" Description=\"\" Proxy=\"default\" StopOnError=\"False\" RecordedResultFile=\"\" ResultsLocale=\"\"\u003e\n\t\u003cItems\u003e\n\t  \u003cRequest Method=\"GET\" Guid=\"a38f9def-f162-141f-a84a-b6e4c74da461\" Version=\"1.1\" Url=\"https://host/health/\" ThinkTime=\"0\" Timeout=\"20\" ParseDependentRequests=\"False\" FollowRedirects=\"True\" RecordResult=\"True\" Cache=\"False\" ResponseTimeGoal=\"0\" Encoding=\"utf-8\" ExpectedHttpStatusCode=\"200\" ExpectedResponseUrl=\"\" ReportingName=\"\" IgnoreHttpStatusCode=\"False\"/\u003e\n\t\u003c/Items\u003e\n    \t\u003cValidationRules\u003e\n\t  \u003cValidationRule Classname=\"Microsoft.VisualStudio.TestTools.WebTesting.Rules.ValidationRuleFindText, Microsoft.VisualStudio.QualityTools.WebTestFramework, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a\" DisplayName=\"Find Text\" Description=\"Verifies the existence of the specified text in the response.\" Level=\"High\" ExectuionOrder=\"BeforeDependents\"\u003e\n\t\t\u003cRuleParameters\u003e\n\t\t  \u003cRuleParameter Name=\"FindText\" Value=\"\u0026quot;status\u0026quot;:\u0026quot;UP\u0026quot;\" /\u003e\n\t\t  \u003cRuleParameter Name=\"IgnoreCase\" Value=\"True\" /\u003e\n\t\t  \u003cRuleParameter Name=\"UseRegularExpression\" Value=\"False\" /\u003e\n\t\t  \u003cRuleParameter Name=\"PassIfTextFound\" Value=\"True\" /\u003e\n\t\t\u003c/RuleParameters\u003e\n\t  \u003c/ValidationRule\u003e\n\t\u003c/ValidationRules\u003e\n\n\t\u003c/WebTest\u003e"},"Description":"","Enabled":true,"Frequency":300,"Kind":"ping","Locations":[{"Id":"emea-nl-ams-azr"}],"Name":"sample-test","Request":null,"RetryEnabled":true,"SyntheticMonitorId":"sample-test","Timeout":20,"ValidationRules":null,"provisioningState":"Succeeded"},"tags":{"hidden-link:/subscriptions/xxxxx-xxxx-xxxx-xxxx-xxxxxx/resourceGroups/resource-group/providers/Microsoft.Insights/components/test-insights-component":"Resource"},"type":"microsoft.insights/webtests"}]
│
│   with module.healthprobe.azurerm_application_insights_web_test.web-test,
│   on .terraform\modules\healthprobe\modules\healthprobe\main.tf line 47, in resource "azurerm_application_insights_web_test" "web-test":
│   47: resource "azurerm_application_insights_web_test" "web-test" {

According to the Microsoft documentation the used API should return 200:
https://docs.microsoft.com/en-us/rest/api/application-insights/web-tests/create-or-update

[Euan-McVie]

Worth noting that this appears to be MS updating their API but not the SDK issue.
https://github.com/Azure/azure-sdk-for-go/blob/44df3c4a9c9d2643026e5aaaa2740421578d2b2a/services/appinsights/mgmt/2020-02-02/insights/webtests.go#L122
Still expects a 200. So not directly terraform issue though depending on how MS resolve it it might require sdk update in terraform.

[tombuildsstuff]

@vap78 thanks for the debug logs, I've opened this upstream PR which'll fix this in the SDK.

[tombuildsstuff]

Based on our Acceptance Tests it appears the upstream API bug has been reverted - can anyone confirm if this is still an issue for them (and if so, in which region)? Thanks!

[heoelri]

Based on our Acceptance Tests it appears the upstream API bug has been reverted - can anyone confirm if this is still an issue for them (and if so, in which region)? Thanks!

It is currently working for me, given that it worked already sporadically before not sure how reliable that is. I reached out the the responsible parties internally and shared the link to this issue - waiting for a response.

[tombuildsstuff]

Workaround on our side: #16845

Whilst the API does appear to have been reverted, since the upstream Swagger PR hasn't been merged yet, we're using a fork of the Azure SDK client there rather than the official SDK, but we can switch back if/when that's merged.

[ALTrijssenaar]

Based on our Acceptance Tests it appears the upstream API bug has been reverted - can anyone confirm if this is still an issue for them (and if so, in which region)? Thanks!

It is also working on our side! Tnx!!!

[vap78]

Based on our Acceptance Tests it appears the upstream API bug has been reverted - can anyone confirm if this is still an issue for them (and if so, in which region)? Thanks!

Can confirm that it works for region West Europe. Thanks!

[github-actions]

This functionality has been released in v3.7.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.