[WIP] Add cli command to export provider/provisioner/resource schemas

[VladRassokhin]

This feature allows to run command like terraform schemas aws and get json with information about required/optional properties of providers/provisioners/resources and their types. (and much more info)

Originally it was required to add 'properties completion' and 'required properties inspection' features in IntelliJ HCL/Terraform plugin and I think it could be shared with community, so vim/emacs/etc users could use it.

Example of generated output for provisioner local-exec (terraform schemas -indent -json local-exec) could be found here. There are other files for other providers/provisioners in the same directory, look around.

Still some things should be done:

• Documentation
• Tests
• Output formats
  • JSON
  • XML
  • plain text (could be useful for bash completion or something like that)
• Check backward compatibility of third party plugins (new command added for provisioners and providers, at least recompilation required)
  • Source compatibility
  • Binary compatibility

[apparentlymart]

This is a really interesting idea!

Are there some standard formats this could support that would then allow its output to be used with existing tools to generate documentation? e.g. would it make sense for the JSON output to be JSON-schema compatible?

I think there are a couple challenges here:

• This changes the interface for plugins to include the new Export function (which maybe could be named ExportSchema instead?). This means that it'll fail when run against third-party plugins that were compiled against older versions of Terraform.
• In Terraform's current architecture, the "schema" helper is considered an implementation detail of a provider as far as core Terraform is concerned. In order to make this work while preserving that architecture, you've invented another set of structures for schema within Terraform core that are at a similar -- but not quite the same -- level of abstraction. I expect we'll want to think hard about whether this is the right separation of concerns, and whether the core model of schema is at the right level of abstraction when compared to the schema helper's idea of schema.

With all of that said, I really like the idea of Terraform's configuration model being more self-describing. In future it might make it easier to keep docs in sync with implementation, and in the further future it may allow more sophisticated tools for working with Terraform configurations, allowing automation or interactive configuration. I'm going to add the "thinking" tag to this, which is a marker that this change requires some philosophy as well as code. 😀

[VladRassokhin]

@apparentlymart

Are there some standard formats this could support that would then allow its output to be used with existing tools to generate documentation? e.g. would it make sense for the JSON output to be JSON-schema compatible?

Don't know about any of such formats. :( Swagger maybe?

This changes the interface for plugins to include the new Export function (which maybe could be named ExportSchema instead?). This means that it'll fail when run against third-party plugins that were compiled against older versions of Terraform.

If plugin is old and terraform core is new, error message would be shown instead of json (afaik).
New method added to ResourceProvider, but since it already implemented in rpc it would be in any plugin after recompilation (no changes to source code).
Situation with Provisioners is quite different. Plugin source code should be a bit modified (see how builtin provisioners modified)

In Terraform's current architecture, the "schema" helper is considered an implementation detail of a provider as far as core Terraform is concerned. In order to make this work while preserving that architecture, you've invented another set of structures for schema within Terraform core that are at a similar -- but not quite the same -- level of abstraction. I expect we'll want to think hard about whether this is the right separation of concerns, and whether the core model of schema is at the right level of abstraction when compared to the schema helper's idea of schema.

Well, since new model uses just maps, arrays and strings any change into new schema class would be still binary backward compatible. Situation is different with Schema struct which already present in Terraform code (and used only in plugins). But there's a lot things to think and discuss.

In future it might make it easier to keep docs in sync with implementation

Yep, it's second usage of this feature that come into my mind.

[josephholsten]

@VladRassokhin how's this coming? looks like it needs to be updated to work with master

[VladRassokhin]

@josephholsten I'm rebasing it from time to time to update schemas in my IntelliJ plugin, but have no spare time to finish this feature at the moment.
So, will rebase it again on weekend and maybe implement some of the items from list above.

[stack72]

Hi @VladRassokhin

What is the status of this PR? Is this something that can be closed? There hasn't been activity in a while

Thanks

Paul

[stack72]

I am going to close this PR out - thanks for the work here. I haven't received any response about the status of the PR since June. If you still feel this is work you'd like to get merged in, please do open another PR

Thanks again

Paul

[stack72]

I am going to close this PR out - thanks for the work here. I haven't received any response about the status of the PR since June. If you still feel this is work you'd like to get merged in, please do open another PR

Thanks again

Paul

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.