[IOPLT-641] Add opex alarms for trial-system #1143
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jira Pull Request LinkThis Pull Request refers to the following Jira issue IOPLT-641 |
|
Terraform Plan 📖Terraform Plan�[0m�[1mInitializing the backend...�[0m
�[0m�[32m
Successfully configured the backend "azurerm"! Terraform will automatically
use this backend unless the backend configuration changes.�[0m
�[0m�[1mInitializing provider plugins...�[0m
- Finding hashicorp/azurerm versions matching ">= 3.30.0, <= 3.40.0"...
- Installing hashicorp/azurerm v3.40.0...
- Installed hashicorp/azurerm v3.40.0 (signed by HashiCorp)
Terraform has created a lock file �[1m.terraform.lock.hcl�[0m to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.�[0m
�[0m�[1m�[32mTerraform has been successfully initialized!�[0m�[32m�[0m
�[0m�[32m
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.�[0m
Acquiring state lock. This may take a few moments...
data.azurerm_resource_group.this: Reading...
data.azurerm_resource_group.this: Read complete after 0s [id=/subscriptions/ec285037-c673-4f58-b594-d7c480da4e8b/resourceGroups/dashboards]
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# azurerm_monitor_scheduled_query_rules_alert.alarm_availability_0 will be created
+ resource "azurerm_monitor_scheduled_query_rules_alert" "alarm_availability_0" {
+ auto_mitigation_enabled = false
+ data_source_id = "/subscriptions/ec285037-c673-4f58-b594-d7c480da4e8b/resourceGroups/io-p-rg-external/providers/Microsoft.Network/applicationGateways/io-p-appgateway"
+ description = (known after apply)
+ enabled = true
+ frequency = 10
+ id = (known after apply)
+ location = "westeurope"
+ name = "ts-p-opex_iobackend_api_trial_system-availability @ _api_v1_trials_trialId_subscriptions"
+ query = <<-EOT
let api_hosts = datatable (name: string) ["app-backend.io.italia.it", "api-app.io.pagopa.it"];
let threshold = 0.99;
AzureDiagnostics
| where originalHost_s in (api_hosts)
| where requestUri_s matches regex "/api/v1/trials/[^/]+/subscriptions"
| summarize
Total=count(),
Success=count(httpStatus_d < 500) by bin(TimeGenerated, 5m)
| extend availability=toreal(Success) / Total
| where availability < threshold
EOT
+ query_type = "ResultCount"
+ resource_group_name = "dashboards"
+ severity = 1
+ tags = {
+ "CostCenter" = "TS310 - PAGAMENTI & SERVIZI"
+ "CreatedBy" = "Terraform"
+ "Environment" = "Prod"
+ "Owner" = "trial-system"
+ "Source" = "https://github.com/pagopa/io-backend"
}
+ time_window = 30
+ action {
+ action_group = [
+ "/subscriptions/ec285037-c673-4f58-b594-d7c480da4e8b/resourceGroups/ts-p-itn-monitor-rg-01/providers/Microsoft.Insights/actiongroups/ioptrialsystemerror",
]
}
+ trigger {
+ operator = "GreaterThanOrEqual"
+ threshold = 4
}
}
# azurerm_monitor_scheduled_query_rules_alert.alarm_time_0 will be created
+ resource "azurerm_monitor_scheduled_query_rules_alert" "alarm_time_0" {
+ auto_mitigation_enabled = false
+ data_source_id = "/subscriptions/ec285037-c673-4f58-b594-d7c480da4e8b/resourceGroups/io-p-rg-external/providers/Microsoft.Network/applicationGateways/io-p-appgateway"
+ description = (known after apply)
+ enabled = true
+ frequency = 10
+ id = (known after apply)
+ location = "westeurope"
+ name = "ts-p-opex_iobackend_api_trial_system-responsetime @ _api_v1_trials_trialId_subscriptions"
+ query = <<-EOT
let api_hosts = datatable (name: string) ["app-backend.io.italia.it", "api-app.io.pagopa.it"];
let threshold = 1;
AzureDiagnostics
| where originalHost_s in (api_hosts)
| where requestUri_s matches regex "/api/v1/trials/[^/]+/subscriptions"
| summarize
watermark=threshold,
duration_percentile_95=percentiles(timeTaken_d, 95) by bin(TimeGenerated, 5m)
| where duration_percentile_95 > threshold
EOT
+ query_type = "ResultCount"
+ resource_group_name = "dashboards"
+ severity = 1
+ tags = {
+ "CostCenter" = "TS310 - PAGAMENTI & SERVIZI"
+ "CreatedBy" = "Terraform"
+ "Environment" = "Prod"
+ "Owner" = "trial-system"
+ "Source" = "https://github.com/pagopa/io-backend"
}
+ time_window = 30
+ action {
+ action_group = [
+ "/subscriptions/ec285037-c673-4f58-b594-d7c480da4e8b/resourceGroups/ts-p-itn-monitor-rg-01/providers/Microsoft.Insights/actiongroups/ioptrialsystemerror",
]
}
+ trigger {
+ operator = "GreaterThanOrEqual"
+ threshold = 4
}
}
# azurerm_portal_dashboard.this will be created
+ resource "azurerm_portal_dashboard" "this" {
+ dashboard_properties = jsonencode(
{
+ lenses = {
+ "0" = {
+ order = 0
+ parts = {
+ "0" = {
+ metadata = {
+ inputs = [
+ {
+ isOptional = true
+ name = "resourceTypeMode"
},
+ {
+ isOptional = true
+ name = "ComponentId"
},
+ {
+ isOptional = true
+ name = "Scope"
+ value = {
+ resourceIds = [
+ "/subscriptions/ec285037-c673-4f58-b594-d7c480da4e8b/resourceGroups/io-p-rg-external/providers/Microsoft.Network/applicationGateways/io-p-appgateway",
]
}
},
+ {
+ isOptional = true
+ name = "PartId"
},
+ {
+ isOptional = true
+ name = "Version"
+ value = "2.0"
},
+ {
+ isOptional = true
+ name = "TimeRange"
+ value = "PT4H"
},
+ {
+ isOptional = true
+ name = "DashboardId"
},
+ {
+ isOptional = true
+ name = "DraftRequestParameters"
+ value = {
+ scope = "hierarchy"
}
},
+ {
+ isOptional = true
+ name = "Query"
+ value = <<-EOT
let api_hosts = datatable (name: string) ["app-backend.io.italia.it", "api-app.io.pagopa.it"];
let threshold = 0.99;
AzureDiagnostics
| where originalHost_s in (api_hosts)
| where requestUri_s matches regex "/api/v1/trials/[^/]+/subscriptions"
| summarize
Total=count(),
Success=count(httpStatus_d < 500) by bin(TimeGenerated, 5m)
| extend availability=toreal(Success) / Total
| project TimeGenerated, availability, watermark=threshold
| render timechart with (xtitle = "time", ytitle= "availability(%)")
EOT
},
+ {
+ isOptional = true
+ name = "ControlType"
+ value = "FrameControlChart"
},
+ {
+ isOptional = true
+ name = "SpecificChart"
+ value = "Line"
},
+ {
+ isOptional = true
+ name = "PartTitle"
+ value = "Availability (5m)"
},
+ {
+ isOptional = true
+ name = "PartSubTitle"
+ value = "/api/v1/trials/{trialId}/subscriptions"
},
+ {
+ isOptional = true
+ name = "Dimensions"
+ value = {
+ aggregation = "Sum"
+ splitBy = []
+ xAxis = {
+ name = "TimeGenerated"
+ type = "datetime"
}
+ yAxis = [
+ {
+ name = "availability"
+ type = "real"
},
+ {
+ name = "watermark"
+ type = "real"
},
]
}
},
+ {
+ isOptional = true
+ name = "LegendOptions"
+ value = {
+ isEnabled = true
+ position = "Bottom"
}
},
+ {
+ isOptional = true
+ name = "IsQueryContainTimeRange"
+ value = false
},
]
+ settings = {
+ content = {
+ PartTitle = "Availability (5m)"
+ Query = <<-EOT
let api_hosts = datatable (name: string) ["app-backend.io.italia.it", "api-app.io.pagopa.it"];
let threshold = 0.99;
AzureDiagnostics
| where originalHost_s in (api_hosts)
| where requestUri_s matches regex "/api/v1/trials/[^/]+/subscriptions"
| summarize
Total=count(),
Success=count(httpStatus_d < 500) by bin(TimeGenerated, 5m)
| extend availability=toreal(Success) / Total
| project TimeGenerated, availability, watermark=threshold
| render timechart with (xtitle = "time", ytitle= "availability(%)")
EOT
}
}
+ type = "Extension/Microsoft_OperationsManagementSuite_Workspace/PartType/LogsDashboardPart"
}
+ position = {
+ colSpan = 6
+ rowSpan = 4
+ x = 0
+ y = 0
}
}
+ "1" = {
+ metadata = {
+ inputs = [
+ {
+ isOptional = true
+ name = "resourceTypeMode"
},
+ {
+ isOptional = true
+ name = "ComponentId"
},
+ {
+ isOptional = true
+ name = "Scope"
+ value = {
+ resourceIds = [
+ "/subscriptions/ec285037-c673-4f58-b594-d7c480da4e8b/resourceGroups/io-p-rg-external/providers/Microsoft.Network/applicationGateways/io-p-appgateway",
]
}
},
+ {
+ isOptional = true
+ name = "PartId"
},
+ {
+ isOptional = true
+ name = "Version"
+ value = "2.0"
},
+ {
+ isOptional = true
+ name = "TimeRange"
+ value = "PT4H"
},
+ {
+ isOptional = true
+ name = "DashboardId"
},
+ {
+ isOptional = true
+ name = "DraftRequestParameters"
+ value = {
+ scope = "hierarchy"
}
},
+ {
+ isOptional = true
+ name = "Query"
+ value = <<-EOT
let api_url = "/api/v1/trials/[^/]+/subscriptions";
let api_hosts = datatable (name: string) ["app-backend.io.italia.it", "api-app.io.pagopa.it"];
AzureDiagnostics
| where originalHost_s in (api_hosts)
| where requestUri_s matches regex api_url
| extend HTTPStatus = case(
httpStatus_d between (100 .. 199), "1XX",
httpStatus_d between (200 .. 299), "2XX",
httpStatus_d between (300 .. 399), "3XX",
httpStatus_d between (400 .. 499), "4XX",
"5XX")
| summarize count() by HTTPStatus, bin(TimeGenerated, 5m)
| render areachart with (xtitle = "time", ytitle= "count")
EOT
},
+ {
+ isOptional = true
+ name = "ControlType"
+ value = "FrameControlChart"
},
+ {
+ isOptional = true
+ name = "SpecificChart"
+ value = "Pie"
},
+ {
+ isOptional = true
+ name = "PartTitle"
+ value = "Response Codes (5m)"
},
+ {
+ isOptional = true
+ name = "PartSubTitle"
+ value = "/api/v1/trials/{trialId}/subscriptions"
},
+ {
+ isOptional = true
+ name = "Dimensions"
+ value = {
+ aggregation = "Sum"
+ splitBy = []
+ xAxis = {
+ name = "httpStatus_d"
+ type = "string"
}
+ yAxis = [
+ {
+ name = "count_"
+ type = "long"
},
]
}
},
+ {
+ isOptional = true
+ name = "LegendOptions"
+ value = {
+ isEnabled = true
+ position = "Bottom"
}
},
+ {
+ isOptional = true
+ name = "IsQueryContainTimeRange"
+ value = false
},
]
+ settings = {
+ content = {
+ Dimensions = {
+ aggregation = "Sum"
+ splitBy = [
+ {
+ name = "HTTPStatus"
+ type = "string"
},
]
+ xAxis = {
+ name = "TimeGenerated"
+ type = "datetime"
}
+ yAxis = [
+ {
+ name = "count_"
+ type = "long"
},
]
}
+ PartTitle = "Response Codes (5m)"
+ Query = <<-EOT
let api_url = "/api/v1/trials/[^/]+/subscriptions";
let api_hosts = datatable (name: string) ["app-backend.io.italia.it", "api-app.io.pagopa.it"];
AzureDiagnostics
| where originalHost_s in (api_hosts)
| where requestUri_s matches regex api_url
| extend HTTPStatus = case(
httpStatus_d between (100 .. 199), "1XX",
httpStatus_d between (200 .. 299), "2XX",
httpStatus_d between (300 .. 399), "3XX",
httpStatus_d between (400 .. 499), "4XX",
"5XX")
| summarize count() by HTTPStatus, bin(TimeGenerated, 5m)
| render areachart with (xtitle = "time", ytitle= "count")
EOT
+ SpecificChart = "StackedArea"
}
}
+ type = "Extension/Microsoft_OperationsManagementSuite_Workspace/PartType/LogsDashboardPart"
}
+ position = {
+ colSpan = 6
+ rowSpan = 4
+ x = 6
+ y = 0
}
}
+ "2" = {
+ metadata = {
+ inputs = [
+ {
+ isOptional = true
+ name = "resourceTypeMode"
},
+ {
+ isOptional = true
+ name = "ComponentId"
},
+ {
+ isOptional = true
+ name = "Scope"
+ value = {
+ resourceIds = [
+ "/subscriptions/ec285037-c673-4f58-b594-d7c480da4e8b/resourceGroups/io-p-rg-external/providers/Microsoft.Network/applicationGateways/io-p-appgateway",
]
}
},
+ {
+ isOptional = true
+ name = "PartId"
},
+ {
+ isOptional = true
+ name = "Version"
+ value = "2.0"
},
+ {
+ isOptional = true
+ name = "TimeRange"
+ value = "PT4H"
},
+ {
+ isOptional = true
+ name = "DashboardId"
},
+ {
+ isOptional = true
+ name = "DraftRequestParameters"
+ value = {
+ scope = "hierarchy"
}
},
+ {
+ isOptional = true
+ name = "Query"
+ value = <<-EOT
let api_hosts = datatable (name: string) ["app-backend.io.italia.it", "api-app.io.pagopa.it"];
let threshold = 1;
AzureDiagnostics
| where originalHost_s in (api_hosts)
| where requestUri_s matches regex "/api/v1/trials/[^/]+/subscriptions"
| summarize
watermark=threshold,
duration_percentile_95=percentiles(timeTaken_d, 95) by bin(TimeGenerated, 5m)
| render timechart with (xtitle = "time", ytitle= "response time(s)")
EOT
},
+ {
+ isOptional = true
+ name = "ControlType"
+ value = "FrameControlChart"
},
+ {
+ isOptional = true
+ name = "SpecificChart"
+ value = "StackedColumn"
},
+ {
+ isOptional = true
+ name = "PartTitle"
+ value = "Percentile Response Time (5m)"
},
+ {
+ isOptional = true
+ name = "PartSubTitle"
+ value = "/api/v1/trials/{trialId}/subscriptions"
},
+ {
+ isOptional = true
+ name = "Dimensions"
+ value = {
+ aggregation = "Sum"
+ splitBy = []
+ xAxis = {
+ name = "TimeGenerated"
+ type = "datetime"
}
+ yAxis = [
+ {
+ name = "duration_percentile_95"
+ type = "real"
},
]
}
},
+ {
+ isOptional = true
+ name = "LegendOptions"
+ value = {
+ isEnabled = true
+ position = "Bottom"
}
},
+ {
+ isOptional = true
+ name = "IsQueryContainTimeRange"
+ value = false
},
]
+ settings = {
+ content = {
+ Dimensions = {
+ aggregation = "Sum"
+ splitBy = []
+ xAxis = {
+ name = "TimeGenerated"
+ type = "datetime"
}
+ yAxis = [
+ {
+ name = "watermark"
+ type = "long"
},
+ {
+ name = "duration_percentile_95"
+ type = "real"
},
]
}
+ PartTitle = "Percentile Response Time (5m)"
+ Query = <<-EOT
let api_hosts = datatable (name: string) ["app-backend.io.italia.it", "api-app.io.pagopa.it"];
let threshold = 1;
AzureDiagnostics
| where originalHost_s in (api_hosts)
| where requestUri_s matches regex "/api/v1/trials/[^/]+/subscriptions"
| summarize
watermark=threshold,
duration_percentile_95=percentiles(timeTaken_d, 95) by bin(TimeGenerated, 5m)
| render timechart with (xtitle = "time", ytitle= "response time(s)")
EOT
+ SpecificChart = "Line"
}
}
+ type = "Extension/Microsoft_OperationsManagementSuite_Workspace/PartType/LogsDashboardPart"
}
+ position = {
+ colSpan = 6
+ rowSpan = 4
+ x = 12
+ y = 0
}
}
}
}
}
+ metadata = {
+ model = {
+ filterLocale = {
+ value = "en-us"
}
+ filters = {
+ value = {
+ MsPortalFx_TimeRange = {
+ displayCache = {
+ name = "Local Time"
+ value = "Past 48 hours"
}
+ filteredPartIds = [
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432ed",
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432ef",
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432f1",
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432f3",
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432f5",
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432f7",
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432f9",
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432fb",
+ "StartboardPart-LogsDashboardPart-9badbd78-7607-4131-8fa1-8b85191432fd",
]
+ model = {
+ format = "local"
+ granularity = "auto"
+ relative = "48h"
}
}
}
}
+ timeRange = {
+ type = "MsPortalFx.Composition.Configuration.ValueTypes.TimeRange"
+ value = {
+ relative = {
+ duration = 24
+ timeUnit = 1
}
}
}
}
}
}
)
+ id = (known after apply)
+ location = "westeurope"
+ name = "ts-p-opex_iobackend_api_trial_system"
+ resource_group_name = "dashboards"
+ tags = {
+ "CostCenter" = "TS310 - PAGAMENTI & SERVIZI"
+ "CreatedBy" = "Terraform"
+ "Environment" = "Prod"
+ "Owner" = "trial-system"
+ "Source" = "https://github.com/pagopa/io-backend"
}
}
Plan: 3 to add, 0 to change, 0 to destroy.
─────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
Releasing state lock. This may take a few moments...
|
kin0992
approved these changes
Jul 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
List of Changes
Motivation and Context
Set up alarms for the endpoints of the trial system provided by io-backend.
How Has This Been Tested?
N/A
Types of changes
Checklist: