[WIP] Import resources into Terraform during refresh #3345
Conversation
|
@ross is this building correctly on your end? |
Missed adding a change on terraform.Context. Should build now. Thanks for letting me know. |
|
Added unit tests for the |
|
This functionality will determine whether my team adopts Terraform. Would love to see this make it into a release soon! ⚡ |
Are you able to try it out? I can see import being an adoption blocker for almost anything with existing infrastructure. |
|
@ross sure. Do I need to build off your branch? And will it work for OpenStack? |
Yeah, you should be able to follow the developing terraform instructions in the main README. When it says to clone this repository you'd instead clone mine and check out this branch. If you haven't built terraform before the vagrant route may be the easiest.
It should, though I've only tested it with AWS so YMMV. There are some ~bugs in the providers where details of the metadata are't pulled down fully during refresh. Those will show up as places where plan is showing changes no matter how much you specify in the .tf files. Let me know if you run in to any problems or have questions. I'd really like to see this move forward and the best way to do that is to get some people kicking the tires. |
|
👍 This or the associated PR would be awesome for our team moving to Terraform |
Adds a -import option to terraform refresh that allows specifying a mapping between resources names and ids so that existing infrastructure can be imported and managed by terraform.
- reworks import file format, `module resource id` - cleans up importResources a bit and avoids the helper method - walks modules pulling out their resources so that we only import things that have been configured Better, but still a bit more cleaning up and polishing to go.
@devth Just pushed a new test branch, https://github.com/ross/terraform/tree/refresh-import-v0.6.3 |
|
Just ran in to some cases with the Heroku provider where this implementation isn't sufficent. The same would be the case with the other branch. Basically its provider requires data beyond the ID to successfully make Read calls. Heroku's url structure has nested nouns: This will either require extending the import file format to allow specifying the other data or perhaps some sort of copying of the configured (tf) data over in to the initial setup. The former would definitely work, but the latter (if possible) would be nice. |
|
Does this work for named resources like security groups and the google resources, where you specify a name that has to be unique across your project / account? |
I've used it with security groups and that works fine if Terraform can manage them this can import them. I haven't tried Google resources, but they should work. Only known problem at this point is Heroku addons where more than the ID is required to refresh the resources. I looked in to clean solutions to that yesterday that so far haven't been fruitful. There's something I know will work, but I'm trying to find a cleaner way. |
|
@ross not sure why but I'm still hitting the same issue I did on So it looks like the right version but maybe something strange is going on. |
Some providers need more than the ID to look up their resources. Heroku addons being a good example, they need their app.
|
OK. Pushed a small commit that provides a way to get the "extra" attributes to the state file during import so that refresh would work. That gets heroku addons working. I feel like there should be a better place to be doing this where that data could be fetched from the config, but didn't find it and I'm not familiar enough with the codebase to tackle the refactoring that would probably be required to get import down in to the lower levels cleanly. |
No clue what's up there. I don't have an openstack setup to run/test against. |
|
Is there a way to import several similar AWS resources (i.e. subnets or instances) to a single object in Terraform with count attribute? |
Haven't specifically tried that, but if I had to guess it'd work if you provide |
|
Looks like this hasn't been touched in awhile. @ross Are you still interested in doing this work, or do you want to hand it off? I'd be willing to pick up where you left off if you don't have time. This is a key feature we'd need to adopt Terraform. |
I was originally hoping to get some feedback/direction before working on it more, but either way I won't likely have the time to do so in the near-term so feel free to take it and run. |
|
For those watching this thread (particularly @ross + @blakesmith), not sure if you're aware but it appears that @mitchellh is working on a feature branch for importing pre-existing resources: f-import |
Thanks for the ping @PaulCapestany, that's a massive set of changes. Will keep an 👀 out. |
|
Ah sorry @ross I didn't see this PR! I didn't intend to circumvent your existing work. I am indeed working on such things in my branch. The core is actually very complete. However, let me swing back and review this and see if there is anything I should bring in to my branch. We'll be releasing more details on import shortly... |
No worries. It's unlikely there's anything of use here. I was mostly hacking it in without touching the details/internals which was looking to be needed (and I didn't know enough to really dig into.) |
|
Hi @ross After Mitchell's comment (#3345 (comment)) - I am going to close out this PR. Thanks so much for the work that was done on this :) I am looking forward to seeing these features make Terraform in the future Paul |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Adds an optional parameter to
terraform refresh,-import=paththat enables existing resources to be pulled under terraform management. It's a different take on #2022 that I believe allows a bit more functionality/flexibility.It ignores resource mappings when the resource is already under terraform management. It also allows ignoring mappings for things that aren't yet configured. Those two things allow you to build a large list of mappings (mechanism not included) and then iterate piece by piece on config for them until you're happy with plan's changes for them, ideally nothing. You'll only see the plan for things that are configured so if you start working on a single resource that'll be the only thing included. As soon as you add others to the .tf's they'll start showing up. As #2022 mentions there are gaps in the schema of some resources that cause small plan changes and in some situations even suggest/require resource replacement. Doesn't seem like those should be tackled as part of this PR, but it would be nice to knock them out.
The mapping format is pretty simplistic: module, resource, and id.
New to the innards of terraform so there's likely things that could be done more cleanly were I more deeply familiar with it. The only thing I'm not particularly happy with was
Context.UpdateState. Since Context.state isn't accessible it was either that, making it public, or completely reloading the context prior to refreshing. None of the options seemed great, but this route avoids double verifying and re-reading inputs.Thoughts & suggestions welcome.
fixes #581