Creating VM with mana data disks very slow using azurerm_virtual_machine_data_disk_attachment

[NillsF]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.24
+ provider.azurerm v2.3.0

Affected Resource(s)

• azurerm_virtual_machine_data_disk_attachment
• azurerm_linux_virtual_machine
• azurerm_virtual_machine

Terraform Configuration Files

Using the new version with azurerm_virtual_machine_data_disk_attachment takes about 35 to 40 minutes.

provider "azurerm" {
  features {}
    version = "~>2.0"
}

resource "azurerm_resource_group" "main" {
  name     = "${var.prefix}-resources"
  location = var.location
}

resource "azurerm_virtual_network" "main" {
  name                = "${var.prefix}-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

resource "azurerm_subnet" "internal" {
  name                 = "internal"
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefix       = "10.0.2.0/24"
}

resource "azurerm_network_interface" "main" {
  name                = "${var.prefix}-nic"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location

  ip_configuration {
    name                          = "internal"
    subnet_id                     = azurerm_subnet.internal.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_linux_virtual_machine" "main" {
  name                            = "${var.prefix}-vm"
  resource_group_name             = azurerm_resource_group.main.name
  location                        = azurerm_resource_group.main.location
  size                            = "Standard_F16s_v2"
  admin_username                  = "adminuser"
  admin_password                  = "P@ssw0rd1234!"
  disable_password_authentication = false
  network_interface_ids = [
    azurerm_network_interface.main.id,
  ]

  source_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }

  os_disk {
    storage_account_type = "Standard_LRS"
    caching              = "ReadWrite"
  }
}

resource "azurerm_managed_disk" "data" {
    count = 26
  name                 = format("redo01-%d", count.index)
  location             = azurerm_resource_group.main.location
  create_option        = "Empty"
  disk_size_gb         = 10
  resource_group_name  = azurerm_resource_group.main.name
  storage_account_type = "Standard_LRS"
}

resource "azurerm_virtual_machine_data_disk_attachment" "data" {
  count = 26
  virtual_machine_id = azurerm_linux_virtual_machine.main.id
  managed_disk_id    = azurerm_managed_disk.data[count.index].id
  lun                = format("%d", count.index)
  caching            = "ReadWrite"
}

Using the old azurerm_virtual_machine object, finishing in 2m37sec.

provider "azurerm" {
  features {}
    version = "~>2.0"
}
variable "prefix" {
  default = "tfvmex"
}

resource "azurerm_resource_group" "main" {
  name     = "${var.prefix}-resources"
  location = "West US 2"
}

resource "azurerm_virtual_network" "main" {
  name                = "${var.prefix}-network"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name
}

resource "azurerm_subnet" "internal" {
  name                 = "internal"
  resource_group_name  = azurerm_resource_group.main.name
  virtual_network_name = azurerm_virtual_network.main.name
  address_prefix       = "10.0.2.0/24"
}

resource "azurerm_network_interface" "main" {
  name                = "${var.prefix}-nic"
  location            = azurerm_resource_group.main.location
  resource_group_name = azurerm_resource_group.main.name

  ip_configuration {
    name                          = "testconfiguration1"
    subnet_id                     = azurerm_subnet.internal.id
    private_ip_address_allocation = "Dynamic"
  }
}

resource "azurerm_virtual_machine" "main" {
  name                  = "${var.prefix}-vm"
  location              = azurerm_resource_group.main.location
  resource_group_name   = azurerm_resource_group.main.name
  network_interface_ids = [azurerm_network_interface.main.id]
  vm_size               = "Standard_F16s_v2"

  # Uncomment this line to delete the OS disk automatically when deleting the VM
  # delete_os_disk_on_termination = true

  # Uncomment this line to delete the data disks automatically when deleting the VM
  # delete_data_disks_on_termination = true

  storage_image_reference {
    publisher = "Canonical"
    offer     = "UbuntuServer"
    sku       = "16.04-LTS"
    version   = "latest"
  }
  storage_os_disk {
    name              = "myosdisk1"
    caching           = "ReadWrite"
    create_option     = "FromImage"
    managed_disk_type = "Standard_LRS"
  }
  storage_data_disk {
          name = "datadisk0"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 0
  }

  storage_data_disk {
          name = "datadisk1"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 1
  }

  storage_data_disk {
          name = "datadisk2"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 2
  }

  storage_data_disk {
          name = "datadisk3"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 3
  }

  storage_data_disk {
          name = "datadisk4"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 4
  }

  storage_data_disk {
          name = "datadisk5"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 5
  }

  storage_data_disk {
          name = "datadisk6"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 6
  }

  storage_data_disk {
          name = "datadisk7"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 7
  }

  storage_data_disk {
          name = "datadisk8"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 8
  }

  storage_data_disk {
          name = "datadisk9"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 9
  }

  storage_data_disk {
          name = "datadisk10"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 10
  }

  storage_data_disk {
          name = "datadisk11"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 11
  }

  storage_data_disk {
          name = "datadisk12"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 12
  }

  storage_data_disk {
          name = "datadisk13"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 13
  }

  storage_data_disk {
          name = "datadisk14"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 14
  }

  storage_data_disk {
          name = "datadisk15"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 15
  }

  storage_data_disk {
          name = "datadisk16"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 16
  }

  storage_data_disk {
          name = "datadisk17"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 17
  }

  storage_data_disk {
          name = "datadisk18"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 18
  }

  storage_data_disk {
          name = "datadisk19"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 19
  }

  storage_data_disk {
          name = "datadisk20"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 20
  }

  storage_data_disk {
          name = "datadisk21"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 21
  }

  storage_data_disk {
          name = "datadisk22"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 22

  }

  storage_data_disk {
          name = "datadisk23"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 23

  }
    storage_data_disk {
          name = "datadisk25"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 25

  }
    storage_data_disk {
          name = "datadisk24"
          create_option = "Empty"
          disk_size_gb = 10
          lun = 24

  }
  os_profile {
    computer_name  = "hostname"
    admin_username = "testadmin"
    admin_password = "Password1234!"
  }
  os_profile_linux_config {
    disable_password_authentication = false
  }
  tags = {
    environment = "staging"
  }
}

Debug Output

Panic Output

Expected Behavior

I would expect the a VM with large amount of data disks to finish creating in the same time across old version of azure_virtual_machine as compared to new azurerm_linux_virtual_machine.

Actual Behavior

It takes 35 to 40 minutes using azurerm_linux_virtual_machine and 2min37 for azure_virtual_machine.

Steps to Reproduce

1. Use files provided earlier in this issue
2. terraform plan
3. terraform apply

Important Factoids

References

• azurerm_linux_virtual_machine does not support data disks which is blocking for cloud-init configuration #6074
• Support for disk attachment to VMs at creation time #6117

[tombuildsstuff]

hi @NillsF

Thanks for opening this issue.

Unlike the older azurerm_virtual_machine resource - to be able to correctly represent the behaviour of the Azure API in Terraform for the new Virtual Machine resources we've had to split these out into a separate attachment resource which starts/stops the VM as required by the API. Unfortunately at this time the Azure API doesn't provide the granular levels of API's we need to be able to make these changes in parallel - as such we're forced to use a lock within the provider to ensure these happen sequentially.

We recently opened an issue on the Azure API Specs repository about this which got closed in favour of this issue (which we've got a back-channel going about too) - unfortunately at this time we're waiting on Microsoft to expose the functionality needed to be able to make these changes in parallel.

Since this behaviour is unfortunately intentional due to the lack of support in the Azure API - I'm going to close this issue for the moment - however once the upstream API supports attaching data disks in parallel we can remove these locks and this issue should disappear.

Thanks!

[NillsF]

Thx @tombuildsstuff ,

Thx for the detailed reply. I understand an update to the Azure API would allow for further parallelism which is now not possible.

Would you mind if I ask you two follow-up questions related to this (as potential workarounds):

1. Using the legacy VM object, we're able to create VM with large data disks very quickly. Would you recommend for now to leverage that legacy object to speed up those deployments? Are you planning to add a native datadisks object to the new VM objects as referred in azurerm_linux_virtual_machine does not support data disks which is blocking for cloud-init configuration #6074 and Support for disk attachment to VMs at creation time #6117?
2. Would there be an option in the provider to interpret the 'count' on the azurerm_virtual_machine_data_disk_attachment in the provier itself and submit a single API call with the 26 disks in a single call?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!