Azure Virtual Machine Scale Set is not returning the publicIPPrefixID after a SKU update

[ArcturusZhang]

To reproduce:

1. Create a VMSS with a publicIPPrefixID assigned by using the following tf config

resource "azurerm_linux_virtual_machine_scale_set" "linux_nvavss" {
  name                            = "test"
  location                        = azurerm_resource_group.main.location
  resource_group_name             = azurerm_resource_group.main.name
  computer_name_prefix            = "dapzhang"
  admin_username                  = "dapzhang"
  admin_password                  = "Passw0rd123!"
  disable_password_authentication = false
  instances                       = 2

  # automatic rolling upgrade
  upgrade_mode = "Manual"
  sku          = "Standard_F2"
 
  zones        = []
  
  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }
 
  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }
 
  network_interface {
    name                          = "eth0"
    primary                       = true
    enable_accelerated_networking = true
    enable_ip_forwarding          = false
    ip_configuration {
      name                                   = "ipconfig1"
      primary                                = true
      subnet_id                              = azurerm_subnet.front.id
      load_balancer_backend_address_pool_ids = [azurerm_lb_backend_address_pool.test.id]
      public_ip_address {
        name                    = "instancePublicIP"
        idle_timeout_in_minutes = 30
        domain_name_label       = "testdomain"
        public_ip_prefix_id     = azurerm_public_ip_prefix.test.id
      }
    }
  }
}

1. Update the VMSS sku with the following request body:

   "properties": {
        "upgradePolicy": {
            "mode": "Manual"
        },
        "virtualMachineProfile": {
            "networkProfile": {
                "networkInterfaceConfigurations": [
                    {
                        "name": "eth0",
                        "properties": {
                            "primary": true,
                            "enableAcceleratedNetworking": true,
                            "dnsSettings": {
                                "dnsServers": []
                            },
                            "ipConfigurations": [
                                {
                                    "name": "ipconfig1",
                                    "properties": {
                                        "subnet": {
                                            "id": "/subscriptions/85b3dbca-0000-0000-0000-67a141095a76/resourceGroups/test/providers/Microsoft.Network/virtualNetworks/network/subnets/front"
                                        },
                                        "primary": true,
                                        "publicIPAddressConfiguration": {
                                            "name": "instancePublicIP",
                                            "properties": {
                                                "idleTimeoutInMinutes": 30,
                                                "dnsSettings": {
                                                    "domainNameLabel": "testdomain"
                                                }
                                            }
                                        },
                                        "privateIPAddressVersion": "IPv4",
                                        "applicationGatewayBackendAddressPools": [],
                                        "applicationSecurityGroups": [],
                                        "loadBalancerBackendAddressPools": [
                                            {
                                                "id": "/subscriptions/85b3dbca-0000-0000-0000-67a141095a76/resourceGroups/test/providers/Microsoft.Network/loadBalancers/acctestlb-001/backendAddressPools/test"
                                            }
                                        ],
                                        "loadBalancerInboundNatPools": []
                                    }
                                }
                            ],
                            "enableIPForwarding": false
                        }
                    }
                ]
            }
        }
    },
    "sku": {
        "name": "Standard_D3_v2",
        "tier": "Standard",
        "capacity": 2
    }
}

1. Get the updated VMSS and found publicIPPrefixID is empty.

Here are two assumptions:

1. The PATCH request erased the public ip prefix
2. The PATCH request did not change the public ip prefix, but it just did not returned after the PATCH is done

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Drewm3, @avirishuv.

[Drewm3]

@avirishuv, could you please take a look at this.

[avirishuv]

@ArcturusZhang In the second step where you are updating the VMSS, the new "publicIPAddressConfiguration" does not provide the "PublicIpPrefix" as a property. So this is getting updated with an empty value and the original value is getting removed. This seems like expected behavior - please help clarify why you think this is an issue.

[ArcturusZhang]

Hi @avirishuv thank you for your insights! I think I found the root cause of this.
Here the update version of the PublicIPAddressConfigurationProperties, it only has two properties (without publicIPPrefix). But the create version of the PublicIPAddressConfigurationProperties has more properties (with publicIPPrefix).
This is the reason in the update body, we do not have a publicIPPrefix assigned, and therefore the public ip prefix is erased.

So I assume maybe this is a swagger issue? Is the publicIPPrefix is updatable?

[avirishuv]

Thanks for the inputs, let me check the property publicIPPrefix and whether it should be added to the update version also.

[brianlehr]

Reviewing this internally with NRP team. May be bug on NRP side preventing this from being updated.
cc @NitinGoelMsft

[avirishuv]

@brianlehr any update on this?

[brianlehr]

This should be in NRP queue as a bugfix. Apologies as I'm not sure how to sort it there.

[amjads1]

Assigning back to Avi to look into the issue and route to appropriate queue.

[avirishuv]

@brianlehr @NitinGoelMsft any updates on this ?

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @aznetsuppgithub.

Issue Details

---

To reproduce:

1. Create a VMSS with a publicIPPrefixID assigned by using the following tf config

resource "azurerm_linux_virtual_machine_scale_set" "linux_nvavss" {
  name                            = "test"
  location                        = azurerm_resource_group.main.location
  resource_group_name             = azurerm_resource_group.main.name
  computer_name_prefix            = "dapzhang"
  admin_username                  = "dapzhang"
  admin_password                  = "Passw0rd123!"
  disable_password_authentication = false
  instances                       = 2

  # automatic rolling upgrade
  upgrade_mode = "Manual"
  sku          = "Standard_F2"
 
  zones        = []
  
  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }
 
  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Standard_LRS"
  }
 
  network_interface {
    name                          = "eth0"
    primary                       = true
    enable_accelerated_networking = true
    enable_ip_forwarding          = false
    ip_configuration {
      name                                   = "ipconfig1"
      primary                                = true
      subnet_id                              = azurerm_subnet.front.id
      load_balancer_backend_address_pool_ids = [azurerm_lb_backend_address_pool.test.id]
      public_ip_address {
        name                    = "instancePublicIP"
        idle_timeout_in_minutes = 30
        domain_name_label       = "testdomain"
        public_ip_prefix_id     = azurerm_public_ip_prefix.test.id
      }
    }
  }
}

1. Update the VMSS sku with the following request body:

   "properties": {
        "upgradePolicy": {
            "mode": "Manual"
        },
        "virtualMachineProfile": {
            "networkProfile": {
                "networkInterfaceConfigurations": [
                    {
                        "name": "eth0",
                        "properties": {
                            "primary": true,
                            "enableAcceleratedNetworking": true,
                            "dnsSettings": {
                                "dnsServers": []
                            },
                            "ipConfigurations": [
                                {
                                    "name": "ipconfig1",
                                    "properties": {
                                        "subnet": {
                                            "id": "/subscriptions/85b3dbca-0000-0000-0000-67a141095a76/resourceGroups/test/providers/Microsoft.Network/virtualNetworks/network/subnets/front"
                                        },
                                        "primary": true,
                                        "publicIPAddressConfiguration": {
                                            "name": "instancePublicIP",
                                            "properties": {
                                                "idleTimeoutInMinutes": 30,
                                                "dnsSettings": {
                                                    "domainNameLabel": "testdomain"
                                                }
                                            }
                                        },
                                        "privateIPAddressVersion": "IPv4",
                                        "applicationGatewayBackendAddressPools": [],
                                        "applicationSecurityGroups": [],
                                        "loadBalancerBackendAddressPools": [
                                            {
                                                "id": "/subscriptions/85b3dbca-0000-0000-0000-67a141095a76/resourceGroups/test/providers/Microsoft.Network/loadBalancers/acctestlb-001/backendAddressPools/test"
                                            }
                                        ],
                                        "loadBalancerInboundNatPools": []
                                    }
                                }
                            ],
                            "enableIPForwarding": false
                        }
                    }
                ]
            }
        }
    },
    "sku": {
        "name": "Standard_D3_v2",
        "tier": "Standard",
        "capacity": 2
    }
}

1. Get the updated VMSS and found publicIPPrefixID is empty.

Here are two assumptions:

1. The PATCH request erased the public ip prefix
2. The PATCH request did not change the public ip prefix, but it just did not returned after the PATCH is done

Author:	ArcturusZhang
Assignees:	brianlehr
Labels:	Compute - VMSS, Network, Service Attention, bug
Milestone:	-

[kkakarla4820]

@brianlehr, We were able to reproduce the issue. The exact behavior is explained below.

1. Suppose customer creates a VMSS with public ip prefix using a terraform template, all the VMs are assigned an IP address according to the public Ip prefix
2. Now when you scale the VMSS (say change the instance count from 3 to 4) and redeploy the template, this will trigger an update on VMSS. Terraform does not send “publicIp prefix” in the update request to the CRP, because of which the publicIp prefix is lost and the new VM will be assigned a random Ip address.
3. Now remember that there are 2 update groups in VMSS (one update group which has public Ip prefix in the network profile with which the old instances were created and the other update group without the public Ip prefix with which the new instance was created)
4. Whenever Terraform encounters more than one update group during scaling of VMSS, it just initiates a DELETE on the existing VMSS and recreates a new one. So during the next scaling operation (say you change the instance count from 4 to 5), the existing VMSS will be deleted and a new VMSS will be created. This behavior is again specific to Terraform. However when the request is sent to create a VMSS, Terraform does send publicIp prefix in the request and in this case, there is no issue and all the VMs will be assigned an Ip address according to the public Ip prefix (also, in this case there is only one update group in VMSS since all the VMs are created using one configuration)
5. The above cycle continues. So every alternate scale up operation will have the issue where the new VMs are assigned a random Ip address. Every other alternate scale up operation will delete the existing VMSS and recreate it with the new instance count, in which case there is no issue with the Ip address assigned to the VMs.
6. Because of this weird behavior which is specific to Terraform, a temporary solution to the issue (when the new VMs are assigned a random Ip address) is to just redeploy the template (works without making any changes too), which will destroy the existing VMSS and recreate a new one without the above mentioned issue.

Coming to the long term fix, there are 2 issues here -

1. Terraform does not send the publicIp prefix in the request to Compute when it wants to update the VMSS. At this point, we are not sure how it generates this request. Should this issue be assigned to Terraform to fix their request?
2. It is weird that it tries to delete the VMSS during every alternate scale up operation (when it encounters 2 update groups). We are not sure if this is intended behavior because when it tries to delete and recreate the VMSS, all the VMs will be assigned a new IP address which might be different from the old one (even if the new IP address is within the range of public Ip prefix). Since this behavior is specific to Terraform, should this issue be assigned to Terraform as well?

Note that Terraform sends the request to Compute and then Compute forwards the request to NRP. NRP does not own any logic to delete and recreate the VMSS if there are 2 update groups. NRP just performs the necessary operation according to the request.

[brianlehr]

@kkakarla4820 Please note we see similar behavior for non-Terraform usecases also, e.g. when we want to modify a Public IP Prefix that is associated to a VMSS Uniform instance.

[omgjlk]

Any updates here? This is still breaking Terraform uses of VMSS with public IP prefix.

[brianlehr]

@NitinGoelMsft FYI

[pranabpm]

Any recent update on this issue? my Cx is facing a blocker due to VMSS recreate every time the run terraforms without any changes to the code.

[karthikka4820]

I checked in this fix long back - karthikka4820@d3e9a50

But I do not see this change anymore in the repo. I am not sure what happened there. Will follow-up with swagger team

[pranabpm]

@karthikka4820 could you please help understand how to test this. Also does this issue also cause VMSS being recreated every time the Terraform is run even without any changes to the code?

[pranabpm]

It looks like the API bug is still open and owned by the Terraform AzureRM changes associated with that issue. There is a workaround for it in the provider that is expected to be merged in the latest release. Can someone please share the workaround with a sample code example please.

Thanks in advance!

[pranabpm]

Not sure if the solution is still implemented but if anyone looking for a workaround the network configuration do include below in Terraform to get ahead of the VMSS recreation. Hope this helps broader community.

lifecycle {
ignore_changes = [
instances,
network_interface,
custom_data
]
}