Add reboot_vapp_on_removal flag to vcd_vapp_network and vcd_vapp_org_network

[Didainius]

Closes #989

This PR adds new reboot_vapp_on_removal flag (false by default) to vcd_vapp_network and vcd_vapp_org_network resources.

Problem

It was never possible to delete a vApp network from a running vApp in UI, however, VCD API did not enforce this rule up to VCD version 10.4.1 and Terraform users were actually able to remove vApp networks from powered on vApps. In VCD 10.4.1, API started validating this error and Terraform is no longer capable of removing it.

Using Terraform became inconvenient and a very common case is hitting an error during terraform destroy operation (sample HCL snippet at the bottom). The error itself looks the following:
Error: error removing vApp network: detaching vApp network MY-NETWORK-NAME (id '9bdad09b-0d6f-4fd6-94cc-3149c6cb7996'): API Error: 409: [ b260b5d9-7716-413d-8b7f-55a2d4356d6c ] The requested operation could not be executed on vApp "7a37828a-cfa5-487a-bcaf-6e5e9d616b03". Stop the vApp and try again.

Solution

A new flag reboot_vapp_on_removal in vcd_vapp_network and vcd_vapp_org_network resources is introduced to signal that a parent vApp can be powered off (if not already powered off) during vApp network removal. After vApp network removal it will power on the vApp again (unless it was already powered off before the operation). This functionality should help not only in destroy scenarios but also in some other cases where vApp power off is not critical.

Note. The flag is effective only in destroy (delete) operations. Enabling the flag would cause diff to be reported, but the update operation will do nothing if only the value of reboot_vapp_on_removal has changed.

After this PR, an attempt to remove vApp network on VCD 10.4.1 without reboot_vapp_on_removal=true would result in such a message to a user:

...
vcd_vapp_network.test: Destroying... [id=urn:vcloud:network:9f916ab6-fda8-41d0-aba8-110bf278bdb4]
╷
│ Error: error removing vApp network: detaching vApp network TestAccVcdNsxtVappNetworkRemoval (id '9f916ab6-fda8-41d0-aba8-110bf278bdb4'): API Error: 409: [ 801aa769-4f3d-413a-9867-8b12bcd1f5b6 ] The requested operation could not be executed on vApp "0b263e7e-a46b-4c6c-a723-af4034389d38". Stop the vApp and try again. 
│ 
│ Parent vApp must be powered off in VCD 10.4.1+ to remove a vApp network. 
│ You can use 'reboot_vapp_on_removal=true' flag to power off vApp before removing network
│ 
...

Additional changes

• vcd_vapp_network and vcd_vapp_org_network had identical delete functions. This PR adds additional complexity to delete functions and there is a risk of having them out of sync with future PRs. Both resources now use the same function and we can split them in future if it is needed.

Snippet to replicate error

resource "vcd_vapp" "test" {
  org      = "cloud"
  vdc      = "nsxt-vdc-cloud"
  name     = "TestAccVcdNsxtVappNetworkRemoval"
  power_on = true
}

resource "vcd_vapp_network" "test" {
  org = "cloud"
  vdc = "nsxt-vdc-cloud"

  name      = "TestAccVcdNsxtVappNetworkRemoval"
  vapp_name = vcd_vapp.test.name
  gateway   = "192.168.2.1"
  netmask   = "255.255.255.0"

  static_ip_pool {
    start_address = "192.168.2.51"
    end_address   = "192.168.2.100"
  }

  # error would occur without this flag on VCD 10.4.1
  # reboot_vapp_on_removal = true
}

resource "vcd_vapp_org_network" "test" {
  org = "cloud"
  vdc = "nsxt-vdc-cloud"

  vapp_name        = vcd_vapp.test.name
  org_network_name = "nsxt-net-cloud-r"

  # error would occur without this flag on VCD 10.4.1
  # reboot_vapp_on_removal = true
}

resource "vcd_vapp_vm" "test" {
  vapp_name     = vcd_vapp.test.name
  name          = "TestAccVcdNsxtVappNetworkRemoval"
  computer_name = "emptyVM"
  memory        = 1048
  cpus          = 2
  cpu_cores     = 1

  os_type          = "sles10_64Guest"
  hardware_version = "vmx-14"

  power_on = true

  depends_on = [vcd_vapp_network.test, vcd_vapp_org_network.test]
}

[lvirbalas]

Thank you for amendments!

[adambarreiro]

Great improvement, thanks!

[adezxc]

Thanks for the improvement!