Azure: reduce client polling interval from default 60s to 5s

[skriss]

Signed-off-by: Steve Kriss steve@heptio.com

The Azure go client defaults to a 60s polling interval to check for API call results. This is a significant amount of time to wait for calls that don't take very long to return. Lower this to five seconds for our snapshot/disk operations to improve performance.

[ncdc]

@paulbouwer @jimzim @noelbundick I know we aren't on v14 of the Azure SDK yet, but does this change make sense for now?

[nrb]

If I've found the right place, this is reduced to 30s by default, and the field that controls it is renamed in the newer SDK. https://godoc.org/github.com/Azure/azure-sdk-for-go/services/classic/management#pkg-constants

[noelbundick]

@ncdc makes sense to me!

I'd make sure to test this with a "real world" disk size. Doing a snapshot & restore in the same region should be quick. I'm not sure exactly how long it will take (particularly the restore disk create operation) with a disk of, let's say 1TB with lots of data & not just zeroes.

If that takes a long time, let's say an hour - a 1 second interval will eat up 3,600 requests of the allowed 15,000 read calls allowed for the entire subscription per hour. If a user had multiple restores and/or other CI/CD or everyday activities going on in a busy subscription, that might be particularly impactful, when a 5s interval might be sufficient instead.

If that scenario is measured in seconds or minutes, no problem!

Just something to think about :)

Details on ARM limits:

• https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-request-limits
• https://docs.microsoft.com/en-us/azure/azure-subscription-service-limits#subscription-limits---azure-resource-manager

[ncdc]

@skriss it might be worth creating a 1TB disk and filling it with random data. Then take a snapshot & time it. The create a disk from the snapshot & time it.

[skriss]

Good idea, thanks @noelbundick @ncdc. Will do. Even backing off to 5s will be a significant improvement, and not much worse than 1.

[nrb]

Testing this out with the 1GB volume nginx example appears to have definitely made Ark more responsive. I have not tested this with anything larger, though.

[skriss]

@ncdc @noelbundick @nrb I tested a backup & restore with a disk with ~100GB data on it (on the way to generating a full TB) and the disk snapshot and restore operations are ~instantaneous - no more than a couple seconds.

[skriss]

Did the same test with a ~700GB and it's also ~instantaneous. Based on that, I'm comfortable leaving the timeout where it is. Does anybody think it would still be a good idea to back it off to something slightly longer?

[ncdc]

You could do something like 1s for up to 10 or 15s, then up to 2s, or 5s, etc.

[skriss]

I don't think that's practically possible with the Azure SDK -- all of the polling is handled within the SDK.

[ncdc]

Ok, so we should be good to move forward with this, since the snaps are fast.

[skriss]

Yeah, I feel comfortable as-is based on observed behavior.

[ncdc]

Although honestly I think 5s would be perfectly reasonable too.

[skriss]

Let's go with 5s for now. Still a significant improvement and slightly more conservative from an API calls perspective.

[skriss]

woohoo!