'Error retrieving network myOrgVdcNetwork from network list' errors during plan or apply

[RussPitcher]

Overview

We are running into an issue where terraform fails to check state successfully when running an apply or plan. There are a large number of OrgVDC networks (vcd_network_direct) in the environment (180 currently) and when the refresh runs on a state that manages only 29 of these networks it fails to complete and reports that it cannot retrieve three of the networks from the network list.

Having done some basic internal testing it would seem that this may be related to the paging of records in some way, as the max page size for query rest calls seems to be 128. We first had this issue with a single OrgVDC network and to work around it and create more networks we removed the definition of the problem vcd_network_direct from the .tf file and the state file. Terraform plan then worked fine. Then three network configs were added and a terraform apply was run. It started fine but during the create the errors shown below started appearing. All three networks were successfully created in vCD but now there were three errors on other networks during the state refresh.

Affected Resource(s)

vcd_network_direct

Terraform config

Define and create a large number of external networks and orgVdc networks - greater than 128
We have multiple config files like this:

resource "vcd_external_network" "ExtNet-MyOrgVdcNetwork" {
	name        = "10_MyOrgVdcNetwork"
	description = "Reference for vCD external network"
  
	ip_scope {
	  gateway    = "10.10.159.225"
	  netmask    = "255.255.255.224"
	  dns1       = "8.8.8.8"
	  dns2       = "4.4.4.4"
	  dns_suffix = "private.local"
  
	  static_ip_pool {
		start_address = "10.10.159.240"
		end_address   = "10.10.159.245"
	  }
	}
  
	vsphere_network {
	  name    = "vxw-dvs-54-virtualwire-123-sid-34211-ExtNet-MyOrgVdcNetwork"
	  type    = "DV_PORTGROUP"
	  vcenter = "FakeVCS001"
	}
	lifecycle {
	  ignore_changes = [
		ip_scope
	  ]
	}
  
	retain_net_info_across_deployments = "false"
  }
  
  resource "vcd_network_direct" "MyOrgVdcNetwork" {
	org              = "MyFakeOrg"
	vdc              = "MyOrgVdc"
	name             = "MyOrgVdcNetwork"
	external_network = vcd_external_network.ExtNet-MyOrgVdcNetwork.name
  }
  

Steps to Reproduce

Run terraform plan

Expected Behavior

Terraform should refresh the state and report no changes required

Actual Behavior

State refresh fails with one or more errors as follows:

Error: error retrieving network myOrgVdcNetwork from network list

Important Factoids

The affected OrgVDC networks remain the same unless the overall number of networks in VDC changes. We can reduce the config and state to a single problematic OrgVDC network and the problem remains.

[dataclouder]

Thanks for reporting this issue. We will look into it

[RussPitcher]

I've just updated to terraform 1.3.3 and version 3.7 of the vcd plugin with the same results but a slightly more detailed error message:

│ Error: error retrieving network MyOrgVdcNetwork from network list
│
│   with vcd_network_direct.MyOrgVdcNetwork,
│   on MyNetworkConfig.tf line 32, in resource "vcd_network_direct" "MyOrgVdcNetwork":
│   32: resource "vcd_network_direct" "MyOrgVdcNetwork" {
│

[RussPitcher]

I believe I have found the issue which is similar to my initial guess. If I've understood the code correctly the function genericVcdNetworkDirectRead reads and returns the OrgVDC network details. This calls vdc.GetNetworkList() from the go-vcloud-director module.

func (vdc *Vdc) GetNetworkList() ([]*types.QueryResultOrgVdcNetworkRecordType, error) {
	// Find the list of networks with the wanted name
	result, err := vdc.client.QueryWithNotEncodedParams(nil, map[string]string{
		"type":          "orgVdcNetwork",
		"filter":        fmt.Sprintf("vdc==%s", url.QueryEscape(vdc.Vdc.ID)),
		"filterEncoded": "true",
	})
	if err != nil {
		return nil, fmt.Errorf("[findEdgeGatewayConnection] error returning the list of networks for VDC: %s", err)
	}
	return result.Results.OrgVdcNetworkRecord, nil
}

This pulls back all the orgVdcNetwork objects from the specified Vdc, but it does not implement paging. We have confirmed this with Fiddler:

This means that the default page size of 25 is used. As the VDC that we currently have problems with has 42 networks many of them are being missed and that is causing the errors. My conclusion is that the GetNetworkList function in the go-vcloud-director module needs to be updated to support paged reads.

I've added an issue in that repository:

[dataclouder]

This issue should be now solved with the inclusion of PR 518 in go-vcloud-director