importing aws route53 records is painfully slow

[russ-te2]

this dominates the runtime of our periodic imports -- any reason it takes so long? It seems a cli53 export -> transformation would be orders of magnitude faster.

[patrykorwat]

Our algorithm is as follows:

1. Fetch a list of hosted zones
2. For each hosted zone fetch a list of zone's records
3. Reimport each resource separately by Terraform AWS provider plugin

That last step might take significant amount of time and might be the cause of your problems. In the meantime, we also send requests to fetch resource schema from the plugin for each detected resource (it takes quite a lot of CPU cycles). That's I think is the only performance improvement that we can make since:

1. Terraform import functionality doesn't allow batching
2. We want to make use of the plugin to format Terraform resources correctly

[patrykorwat]

@sergeylanzman I think general improvements to terraformer is everything that can be done, without changing the core assumptions on how we interact with Terraform plugins.
AWS Route 53 has limitation on the number of how many calls can be made to the service and it is 5 per second. After that, it'll throttle requests (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSLimitations.html#limits-api-requests-route-53).

On my account, I can import ~240 AWS Route 53 resources in around 2 minutes, which makes on average 2 reqs per second. That sounds good enough for me not to initiate request throttling and I think that number might higher close to US East coast (am placed in SE Asia) due to lower latencies. The import time is same as terraform plan command.

[sergeylanzman]

Thanks! @meshuga