RetryableError often happens when deleting a deployment

[dsboulder]

When I delete batches of VMs, I often see 1 of them fail with this error:

  Started deleting instances > web_server/63 (0077c4d7-336d-4aa3-8377-a30160d068d0)
  Started deleting instances > web_server/42 (5991b662-351b-4937-8e66-68eb53ff2a72)
  Started deleting instances > web_server/37 (051d2470-879e-4015-b03e-9847cf5cecb1)
  Started deleting instances > web_server/6 (15006c6f-a4ee-47d8-a52f-909f5cb7603e)
   Failed deleting instances > web_server/66 (f1ef615a-80f8-4cd0-a481-4f8fe20491f0): Unknown CPI error 'Bosh::AzureCloud::AzureError' with message 'http_delete - http code: 429
request id: db8192fb-b0a4-4894-9a08-2ae7ea8ebdbb
correlation id: 25f585cc-bdb8-4c2f-8dba-b56a098065a1
routing id: WESTUS:20160721T224932Z:25f585cc-bdb8-4c2f-8dba-b56a098065a1
 message: {
  "error": {
    "code": "RetryableError",
    "message": "A retryable error occured.",
    "details": [
      {
        "code": "RetryableErrorDueToAnotherOperation",
        "message": "Operation AllocateTenantNetworkResourcesOperation (f9827d3e-8995-4afd-ac35-123c1b9e78a4) is updating resource a43d33d7-17fa-441d-9b46-77f68146e4c4. The call can be retried in 14 seconds."
      }
    ]
  }
}' (00:02:59)
     Done deleting instances > web_server/6 (15006c6f-a4ee-47d8-a52f-909f5cb7603e) (00:05:10)
     Done deleting instances > web_server/39 (1d0587a3-88ff-4fa4-834a-bdb52422e3ec) (00:05:10)
     Done deleting instances > web_server/43 (919eaf78-dec0-4e7a-b903-b49c2b858d37) (00:05:10)
     Done deleting instances > web_server/42 (5991b662-351b-4937-8e66-68eb53ff2a72) (00:05:11)
     Done deleting instances > web_server/37 (051d2470-879e-4015-b03e-9847cf5cecb1) (00:05:14)
     Done deleting instances > web_server/87 (91b451e7-9920-430c-b650-dbe17e415a7b) (00:05:14)

[AbelHu]

@dsboulder Could you send a little more logs about the context of the error to chou.hu@microsoft.com? We need to know what happened before this error.

[dsboulder]

Yes, @AbelHu I just emailed you my task debug log, it has all the requests in it

[cppforlife]

Using azure cpi v13.

[AbelHu]

After talking with feature team, 429 is the standard throttling response status. Clients can retry the operation. We will fix it in CPI code.

[dsboulder]

Thank you!