AKS error when trying to upgrade cluster #3239

dillon-courts · 2019-04-12T16:20:54Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Provider version 1.24.

Affected Resource(s)

azurerm_kubernetes_cluster

Description

Trying to upgrade aks cluster from 1.11.5 -> 1.11.9. Receiving the following error:

containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="LinkedInvalidPropertyId" Message="Property id ''at path 'properties.addonProfiles.omsagent.config.logAnalyticsWorkspaceResourceID' is invalid. Expect fully qualified resource Id that start with '/subscriptions/{subscriptionId}' or '/providers/{resourceProviderNamespace}/'."

Unsure if this happens on new cluster creation, only tried the upgrade.

The text was updated successfully, but these errors were encountered:

dillon-courts · 2019-04-12T16:34:06Z

Creating a new cluster still works

tombuildsstuff · 2019-04-14T16:02:24Z

hi @dillon-courts

Thanks for opening this issue :)

Would it be possible for you to provide the Terraform Configuration that you're using so that we can take a look?

Thanks!

dillon-courts · 2019-04-15T13:42:32Z

@tombuildsstuff - Sure, please see the following.

  backend "azurerm" {
    storage_account_name = "crdsterraformstate"
    container_name       = "tfstate"
    key                  = "terraform.tfstate"
    resource_group_name  = "terraform"
  }
}

locals {
    resource_group_name = "${var.resource_group_name_prefix}_${var.env}"
    aks_name = "aks-crds-${var.env}"
    nsg_name = "crds_${var.env}-nsg"

    frontdoor_ip_name = "crossroads-${var.env}"
    api_ip_name = "api-${var.env}"

    virtual_network_name = "crds_${var.env}-vnet"
}

# Configure the Azure Provider
provider "azurerm" {
    subscription_id = "${var.arm_subscription_id}"
    client_id       = "${var.arm_client_id}"
    client_secret   = "${var.arm_client_secret}"
    tenant_id       = "${var.arm_tenant_id}"
    version         = "~> 1.24"
}

resource "azurerm_resource_group" "crds" {
  name     = "${local.resource_group_name}"
  location = "${var.resource_group_location}"
}

resource "azurerm_network_security_group" "crds" {
  name                = "${local.nsg_name}"
  location            = "${var.resource_group_location}"
  resource_group_name = "${azurerm_resource_group.crds.name}"
}

# NSG Rules
resource "azurerm_network_security_rule" "allow-http" {
  name                        = "Allow-HTTP"
  priority                    = 1000
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "80"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = "${azurerm_resource_group.crds.name}"
  network_security_group_name = "${azurerm_network_security_group.crds.name}"
}

resource "azurerm_network_security_rule" "allow-https" {
  name                        = "Allow-HTTPS"
  priority                    = 1010
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "443"
  source_address_prefix       = "*"
  destination_address_prefix  = "*"
  resource_group_name         = "${azurerm_resource_group.crds.name}"
  network_security_group_name = "${azurerm_network_security_group.crds.name}"
}

# Static IPS

# Front Door
resource "azurerm_public_ip" "frontdoor" {
  name                         = "${local.frontdoor_ip_name}"
  location                     = "${var.resource_group_location}"
  resource_group_name          = "${azurerm_kubernetes_cluster.aks.node_resource_group}"
  allocation_method = "Static"
}
# API
resource "azurerm_public_ip" "api" {
  name                         = "${local.api_ip_name}"
  location                     = "${var.resource_group_location}"
  resource_group_name          = "${azurerm_kubernetes_cluster.aks.node_resource_group}"
  allocation_method = "Static"
}

resource "azurerm_virtual_network" "crds" {
  name                = "${local.virtual_network_name}"
  location            = "${var.resource_group_location}"
  resource_group_name = "${azurerm_resource_group.crds.name}"
  address_space       = ["10.0.0.0/18"]
}

resource "azurerm_subnet" "aks_subnet" {
  name                      = "aks-subnet"
  resource_group_name       = "${azurerm_resource_group.crds.name}"
  network_security_group_id = "${azurerm_network_security_group.crds.id}"
  address_prefix            = "10.0.8.0/21" # 10.0.8.0-10.0.15.255
  virtual_network_name      = "${azurerm_virtual_network.crds.name}"
}

resource "azurerm_subnet" "db_subnet" {
  name                      = "db-subnet"
  resource_group_name       = "${azurerm_resource_group.crds.name}"
  network_security_group_id = "${azurerm_network_security_group.crds.id}"
  address_prefix            = "10.0.16.0/24"
  virtual_network_name      = "${azurerm_virtual_network.crds.name}"
}

resource "azurerm_subnet" "vm_subnet" {
  name                      = "vm-subnet"
  resource_group_name       = "${azurerm_resource_group.crds.name}"
  network_security_group_id = "${azurerm_network_security_group.crds.id}"
  address_prefix            = "10.0.17.0/24"
  virtual_network_name      = "${azurerm_virtual_network.crds.name}"
}

resource "azurerm_kubernetes_cluster" "aks" {
  name       = "${local.aks_name}"
  location   = "${var.resource_group_location}"
  dns_prefix = "${local.aks_name}"

  resource_group_name = "${azurerm_resource_group.crds.name}"

  linux_profile {
    admin_username = "${var.linux_admin_username}"

    ssh_key {
      key_data = "${var.linux_admin_ssh_publickey}"
    }
  }

  kubernetes_version = "1.11.9"

  agent_pool_profile {
    name    = "agentpool"
    count   = "3"
    vm_size = "Standard_DS3_v2"
    os_type = "Linux"

    # Required for advanced networking
    vnet_subnet_id = "${azurerm_subnet.aks_subnet.id}"
  }

  service_principal {
    client_id       = "${var.arm_client_id}"
    client_secret   = "${var.arm_client_secret}"
  }

  network_profile {
    network_plugin     = "azure"
    dns_service_ip     = "10.0.0.10"
    docker_bridge_cidr = "172.17.0.1/16"
    service_cidr       = "10.0.0.0/21" # 10.0.0.0-10.0.7.255
  }
}

JimPaine · 2019-04-15T16:08:12Z

Hi @dillon-courts

The error seems to refer to the oms_agent within the addon_profile block, but I can't see it defined? Also which region are you running? When I look at the supported versions and their upgrade paths for West Europe 1.11.5 isn't even there, I'd be interested to see which upgrade paths it shows in the portal.

As you can see below the lowest 1.11.x version is 1.11.8

Get versions by running (change region as required)

az aks get-versions --location westeurope

Output will be like:

{
  "id": "/subscriptions/.../providers/Microsoft.ContainerService/locations/westeurope/orchestrators",
  "name": "default",
  "orchestrators": [
    {
      "default": null,
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.9.10",
      "upgrades": [
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.9.11"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.10.12"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.10.13"
        }
      ]
    },
    {
      "default": null,
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.9.11",
      "upgrades": [
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.10.12"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.10.13"
        }
      ]
    },
    {
      "default": null,
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.12",
      "upgrades": [
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.10.13"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.11.8"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.11.9"
        }
      ]
    },
    {
      "default": null,
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.10.13",
      "upgrades": [
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.11.8"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.11.9"
        }
      ]
    },
    {
      "default": null,
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.11.8",
      "upgrades": [
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.11.9"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.12.6"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.12.7"
        }
      ]
    },
    {
      "default": true,
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.11.9",
      "upgrades": [
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.12.6"
        },
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.12.7"
        }
      ]
    },
    {
      "default": null,
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.12.6",
      "upgrades": [
        {
          "orchestratorType": null,
          "orchestratorVersion": "1.12.7"
        }
      ]
    },
    {
      "default": null,
      "orchestratorType": "Kubernetes",
      "orchestratorVersion": "1.12.7",
      "upgrades": null
    }
  ],
  "type": "Microsoft.ContainerService/locations/orchestrators"
}

dillon-courts · 2019-04-15T16:29:52Z

This is eastus region. You are right that an upgrade path for 1.11.5 -> 1.11.9 is not defined. Azure seems to be very aggressive about deprecating k8s versions as this cluster is only a couple months old.

I know this has worked in the past even though the specific upgrade path was not defined. Also, the portal is offering me an upgrade. I could try it in the portal and see what happens.

VolodymyrLavrenchuk · 2019-04-18T11:32:09Z

Hello.
We have the same problem: can not update label for existing Kubernetes Service when Log Analytics was enabled and then disabled. In such scenario there is an empty omsagent section. This section does not exist in Kubernetes Service which was never linked to Log Analytics.
az aks show --resource-group <rg> --name <cluster>
{ 'addonProfiles': { 'omsagent': { 'config': {}, 'enabled': false } }
...
Here is an error:

azurerm_kubernetes_cluster.aks: Error creating/updating Managed Kubernetes Cluster 'rmaz2us3' (Resource Group 'rg-dmp-k8s-rmaz2us3-dev-westus2'): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code='LinkedInvalidPropertyId' Message='Property id '' at path 'properties.addonProfiles.omsagent.config.logAnalyticsWorkspaceResourceID' is invalid. Expect fully qualified resource Id that start with '/subscriptions/{subscriptionId}' or '/providers/{resourceProviderNamespace}/'.'

TF template:

variable "subscription-id" {}
variable "client-id" {}
variable "client-secret" {}
variable "tenant-id" {}
variable "name" {}
variable "location" {}
variable "resource-group" {}
variable "dns-name-prefix" {}
variable "kubernetes-version" {}
variable "node-count" {}
variable "node-osdisk-size" {}
variable "node-vm-size" {}
variable "service-principal" {}
variable "service-principal-secret" {}
variable "tags" {
type = "map",
default = {
PII="No",
CustomerInfo="No",
CustomerData="No",
ModuleConfig="Yes"
}
}

provider "azurerm" {
subscription_id = "${var.subscription-id}"
client_id = "${var.client-id}"
client_secret = "${var.client-secret}"
tenant_id = "${var.tenant-id}"
}

resource "azurerm_kubernetes_cluster" "aks" {
name = "${var.name}"
location = "${var.location}"
resource_group_name = "${var.resource-group}"
dns_prefix = "${var.dns-name-prefix}"
kubernetes_version = "${var.kubernetes-version}"

agent_pool_profile {
name = "default"
count = "${var.node-count}"
vm_size = "${var.node-vm-size}"
os_disk_size_gb = "${var.node-osdisk-size}"
}

service_principal {
client_id = "${var.service-principal}"
client_secret = "${var.service-principal-secret}"
}

lifecycle {
ignore_changes = [
"kubernetes_version",
"agent_pool_profile.0.count",
"agent_pool_profile.0.vm_size",
"agent_pool_profile.0.name",
"linux_profile",
"service_principal"
]
}

tags = "${var.tags}"
}

dillon-courts · 2019-04-18T11:45:01Z

I can confirm that I was able to upgrade 1.11.5 -> 1.11.9 using Azure Portal. Seems as though I should have been able to do the same using terraform.

guitmz · 2019-04-24T08:56:00Z

I am having a somewhat similar issue (#2993) and happens after the upgrade via portal.

guitmz · 2019-05-03T14:33:37Z

Also, terraform the same error when trying to update k8s service principal secret for example

dberuben · 2019-08-23T13:59:07Z

I'm facing exactly the same issue, i can't upgrade my cluster with Terraform since i updated via the interface.

dberuben · 2019-08-28T18:42:58Z

I don't have the probleme anymore ! look like it was something temporarily

djackyn · 2019-09-25T14:52:13Z

Hi,
We have the same issue if oms_agent was disabled, we cannot do anything with the AKS cluster.

maxlegault · 2019-10-01T00:45:46Z

I am also experiencing the same issue since I disable the oms agent addon profile on one cluster. Upgrades are still available through the portal interface, but no upgrade will work through terraform, resulting in the error message from the initially described issue.

tombuildsstuff · 2019-10-01T11:05:38Z

Closed the wrong issue - reopening

This fixes hashicorp#3239 where a kubernetes cluster that had first the OMS agent profile addon enabled and then disabled not usable anymore by the terraform provider. All subsequent update requests would return a bad request response complaining that the `logAnalayticsWorkspaceId` needs to be a fully qualified resource id.

This fixes #3239

ghost · 2019-10-29T10:36:52Z

This has been released in version 1.36.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.36.0"
}
# ... other configuration ...

ghost · 2019-11-04T14:50:23Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

tombuildsstuff added question service/kubernetes-cluster waiting-response labels Apr 14, 2019

ghost removed the waiting-response label Apr 15, 2019

guitmz mentioned this issue May 3, 2019

Can't update service principal client secret #3375

Closed

tombuildsstuff mentioned this issue Oct 1, 2019

r/kubernetes_cluster: allowing the Service Principal to be updated #4469

Merged

tombuildsstuff closed this as completed in #4469 Oct 1, 2019

tombuildsstuff reopened this Oct 1, 2019

maxlegault mentioned this issue Oct 4, 2019

azurerm_kubernetes_cluster - the log_analytics_workspace_id property is now optional #4513

Merged

katbyte added bug and removed question labels Oct 5, 2019

katbyte closed this as completed in #4513 Oct 5, 2019

katbyte pushed a commit that referenced this issue Oct 5, 2019

Make kubernetes log analytics workspace optional (#4513)

9abf684

This fixes #3239

katbyte added this to the v1.36.0 milestone Oct 5, 2019

ghost locked and limited conversation to collaborators Nov 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AKS error when trying to upgrade cluster #3239

AKS error when trying to upgrade cluster #3239

dillon-courts commented Apr 12, 2019 •

edited by tombuildsstuff

Loading

dillon-courts commented Apr 12, 2019

tombuildsstuff commented Apr 14, 2019

dillon-courts commented Apr 15, 2019 •

edited

Loading

JimPaine commented Apr 15, 2019

dillon-courts commented Apr 15, 2019

VolodymyrLavrenchuk commented Apr 18, 2019 •

edited

Loading

dillon-courts commented Apr 18, 2019 •

edited

Loading

guitmz commented Apr 24, 2019 •

edited

Loading

guitmz commented May 3, 2019

dberuben commented Aug 23, 2019

dberuben commented Aug 28, 2019

djackyn commented Sep 25, 2019

maxlegault commented Oct 1, 2019 •

edited

Loading

tombuildsstuff commented Oct 1, 2019

ghost commented Oct 29, 2019

ghost commented Nov 4, 2019

AKS error when trying to upgrade cluster #3239

AKS error when trying to upgrade cluster #3239

Comments

dillon-courts commented Apr 12, 2019 • edited by tombuildsstuff Loading

Community Note

Terraform (and AzureRM Provider) Version

Affected Resource(s)

Description

dillon-courts commented Apr 12, 2019

tombuildsstuff commented Apr 14, 2019

dillon-courts commented Apr 15, 2019 • edited Loading

JimPaine commented Apr 15, 2019

dillon-courts commented Apr 15, 2019

VolodymyrLavrenchuk commented Apr 18, 2019 • edited Loading

dillon-courts commented Apr 18, 2019 • edited Loading

guitmz commented Apr 24, 2019 • edited Loading

guitmz commented May 3, 2019

dberuben commented Aug 23, 2019

dberuben commented Aug 28, 2019

djackyn commented Sep 25, 2019

maxlegault commented Oct 1, 2019 • edited Loading

tombuildsstuff commented Oct 1, 2019

ghost commented Oct 29, 2019

ghost commented Nov 4, 2019

dillon-courts commented Apr 12, 2019 •

edited by tombuildsstuff

Loading

dillon-courts commented Apr 15, 2019 •

edited

Loading

VolodymyrLavrenchuk commented Apr 18, 2019 •

edited

Loading

dillon-courts commented Apr 18, 2019 •

edited

Loading

guitmz commented Apr 24, 2019 •

edited

Loading

maxlegault commented Oct 1, 2019 •

edited

Loading