Unable to use SSD as vSAN capacity disk

[adarobin]

Terraform Version

0.13.1

vSphere Provider Version

1.23.0

Affected Resource(s)

• vsphere_compute_cluster

Terraform Configuration Files

resource "vsphere_compute_cluster" "cluster" {
    provider = vsphere.nested
    name = var.nested_cluster_name
    datacenter_id = vsphere_datacenter.datacenter.moid

    drs_enabled = true
    drs_automation_level = "fullyAutomated"
    ha_enabled = true
    ha_admission_control_policy = "disabled"
    vsan_enabled = true

    # I commented this out to confirm the dynamic block wasn't the source of the issue.
    # dynamic "vsan_disk_group" {
    #     for_each = var.esxi_hostname_ip_map
    #     iterator = host
    #     content {
    #         cache = compact([ for disk in vsphere_virtual_machine.nested_esxi[host.key].disk: disk.device_address == "scsi:0:1" ? "naa.${replace(lower(disk.uuid),"-","")}" : "" ])[0]
    #         storage = compact([ for disk in vsphere_virtual_machine.nested_esxi[host.key].disk: disk.device_address == "scsi:0:2" ? "naa.${replace(lower(disk.uuid),"-","")}" : "" ])
    #     }
    # }

    vsan_disk_group {
        cache = "naa.6000c29c4f89ebfb2cecf5e510451d2a"
        storage = [ "naa.6000c2978c0f17d04fa18da9f996e7ae" ]
    }

    host_system_ids = [ for k,v in var.esxi_hostname_ip_map : vsphere_host.nested_esxi[k].id ]
    depends_on = [
        vsphere_virtual_machine.vcenter,
    ]
}

Debug Output

Panic Output

Expected Behavior

terraform apply finished without error and the vSAN disk group was created.

Actual Behavior

module.nested-esxi-lab.vsphere_compute_cluster.cluster: Modifying... [id=domain-c9]

Error: A general system error occurred: Disk: naa.6000c2978c0f17d04fa18da9f996e7ae is an SSD and can not be used as a vSAN HDD

  on ../terraform-vsphere-nested-vsphere-lab/main.tf line 197, in resource "vsphere_compute_cluster" "cluster":
 197: resource "vsphere_compute_cluster" "cluster" {

Steps to Reproduce

If I manually set the capacityFlash flag on the SSD, then terraform apply is successful.

The vCenter GUI has a drop down selection between HDD and SSD for the capacity tier and then filters out drives that don't match. SSD disks seem to get the capacityFlash tag added to them automatically in this process.

References

• https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vsan-planning.doc/GUID-42E65085-1DA5-4C0B-A397-3497CBBC600E.html

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[adarobin]

Since this is nested ESXi deployment I am messing around with, I was able to add a provisioner block to my ESXi virtual machines as a workaround:

  provisioner "remote-exec" {
    connection {
      type     = "ssh"
      user     = "root"
      password = random_password.esxi_root_password.result
      host     = "${each.key}.${var.domain}"
    }

    inline = [
      "esxcli vsan storage tag add -d ${compact([ for disk in self.disk: disk.device_address == "scsi:0:2" ? "naa.${replace(lower(disk.uuid),"-","")}" : "" ])[0]} -t capacityFlash",
    ]
  }

[bill-rich]

Hi @adarobin! I have been having trouble reproducing the issue. Can you tell me what vSphere version you're using?

[adarobin]

@bill-rich This was with vSphere 7.0

[bill-rich]

Thanks! That may be what I was missing. I'll get a 7.0 environment to test with and see about getting this fixed.

[tenthirtyam]

Hi @adarobin - are you still seeing this issue in latest provider and TF?

Ryan

[tenthirtyam]

Marking this issue as closed as there is a workable solution with the provisioner.

From experience in both nested and physical all-flash vSAN deployment, this is a common requirement to tag SSDs as capacity flash. In fact, it's a requirement in Cloud Foundation before running Cloud Builder if the cache and capacity are the same size.

Ryan

[adarobin]

@tenthirtyam I haven’t been able to test this as my home lab is a bit of a mess right now. The workaround doesn’t work with anything newer than 7.0 Update 1 as newer releases have SHA-1 disabled for SSH by default.

[tenthirtyam]

I'll reopen and take a look as some options against your module. We have some methods we (VMware) use to automate this that might be an option with a provisioner, but I'd be reticent to add the capacitFlash tagging to the scope of the provider.

Ryan

[adarobin]

@tenthirtyam I’m ok with this being closed, I just wanted to call out that the workaround doesn’t work for 7.0 U2+ if other folks run into it. It should work once hashicorp/terraform#30134 is fixed.

[tenthirtyam]

Thanks for the update, Adam.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.