Ability to import in bulk

[tmccombs]

Current Terraform Version

$ terraform -v   
Terraform v0.12.6-dev

Use-cases

While starting to manage infrastructure with terraform, I have hundreds of resources that I need to import. I want to be able to import all resources from a file that I have generated (in my case using the AWS API).

Attempted Solutions

I've gotten something that works by generating a script that has hundreds of calls to the terraform import command. However, it is very slow to run, because every single call has to acquire the lock, fetch the state (from S3), perform the import, store the state, and release the lock. It would be much more efficient if acquiring the lock, fetching the state, storing the state, and releasing the lock was done just once for all resources.

Proposal

Add on option to the terraform import command which accepts a file. The file should have a resource address and resource id seperated by a space on each line, and it should then acquire the lock and fetch the state, then import all resources in that file, then store the state and release the lock.

[rupa]

Want to chime in that we (at NS1) are quite interested in the outcome of this issue/PR. We have some bulk import use cases. It can take quite some time to get 50k+ resources imported and seems like this could help a lot!

[ashishbista]

I'd like to import all resources for certain providers. For example, I'm using mysql provider to create users and grants. I am creating hundreds of mysql resources. Most of those resources are already created using some other process. I'd be nice to have an ability to import all existing mysql users and grants to a terraform state file. Without this, when I apply terraform, it complains about the existing resources. And, there is no way to import all existing mysql resources in one shot.

[yehudacohen]

Also have a use case that this would help with tremendously. When trying to help customers manage sets of existing infrastructure using terraform, the ability to only import one resource at a time feels crippling.

[yosefvb]

Is there anything I can do to help merge? This solution would be very helpful! @pkolyvas @teamterraform

[cmaster11]

This would indeed be a very useful feature :)

For now, e.g. for Route53 entries, https://mrkaran.dev/posts/terraform-route53-import/

[varmax2511]

Importing multiple resources is a highly sought out feature. The implemenation should support parallelism to improve efficiency of import with multiple resources

[zeako]

for anyone still interested, Google recently open sourced this https://github.com/GoogleCloudPlatform/terraformer

[tmccombs]

Another possible solution for this could be to allow defining imports in the terraform code itself, similar to the new moved block. This could either be a new import block, or a meta attribute (such as lifecycle.import) where you can specify the import id of a resource. And of course if the resource already exists, then the import directive is ignored.

[paulca99]

I've read a few of these bulk-import tickets now, I'm in a bit of a pickle defining our DR procedures. We've over 100 RDS tables , defined in terraform , full of data , backed up in AWS Backup Vault. If some maniac decides to delete all these tables we'd initiate the restore from AWS Backup, which would create 100 new tables, but then would we have to import each back into the tf state one by one ? The state is quite large and single import takes over a minute, So that'd be at least a couple of hours before we could actually terraform apply to create the rest of the missing infrastructure. I suppose we could manually hack the state file to switch these tables state to point to the new ones, but that smells bad to me.

[kmoe]

Bulk import is possible in Terraform v1.5+ with the use of import blocks: https://developer.hashicorp.com/terraform/language/import.

[acdha]

@kmoe I gave that a try last week and it worked really well. Thanks for all of the work getting that over the line.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.