Descriptions for nested schemas are missing (inconsistent behaviour)

[Integralist]

There are two issues:

1. A Description field set on the parent Schema isn't displayed any where.
2. In some cases a nested Schema type(s) will not have its Description value pulled into the documentation.

Here's an example schema that shows a Description field has been set on the top-level Schema:

s.Schema[h.GetKey()] = &schema.Schema{
	Type:        schema.TypeSet,
	Required:    true,
	Description: "A set of Domain names to serve as entry points for your Service",
	Elem: &schema.Resource{
		Schema: map[string]*schema.Schema{
			"name": {
				Type:        schema.TypeString,
				Required:    true,
				Description: "The domain that this Service will respond to",
			},
			"comment": {
				Type:        schema.TypeString,
				Optional:    true,
				Description: "An optional comment about the Domain.",
			},
		},
	},
}

Yet when generating the documentation for this resource I discovered that description value is not present anywhere:

Here's an example schema definition with a nested schema under the 'rules' field:

Schema: map[string]*schema.Schema{
	"publishers": {
		Type:        schema.TypeList,
		Optional:    true,
		Description: "A list of publishers to be used as filters for the data set.",
		Elem:        &schema.Schema{Type: schema.TypeString},
	},
	"tags": {
		Type:        schema.TypeList,
		Optional:    true,
		Description: "A list of tags to be used as filters for the data set.",
		Elem:        &schema.Schema{Type: schema.TypeString},
	},
	"exclude_modsec_rule_ids": {
		Type:        schema.TypeList,
		Optional:    true,
		Description: "A list of modsecurity rules IDs to be excluded from the data set.",
		Elem:        &schema.Schema{Type: schema.TypeInt},
	},
	"rules": {
		Type:        schema.TypeList,
		Computed:    true,
		Description: "The list of rules that results from any given combination of filters.",
		Elem: &schema.Resource{
			Schema: map[string]*schema.Schema{
				"modsec_rule_id": {
					Type:        schema.TypeInt,
					Required:    true,
					Description: "The modsecurity rule ID.",
				},
				"latest_revision_number": {
					Type:        schema.TypeInt,
					Required:    true,
					Description: "The modsecurity rule's latest revision.",
				},
				"type": {
					Type:        schema.TypeString,
					Computed:    true,
					Description: "The modsecurity rule's type.",
				},
			},
		},
	},
},

In the generated schema output (see below example, from running tfplugindocs generate) we can see the Description fields from the top-level Schema types are included in the output, but the nested Schema Description fields are not included (see <MISSING DESCRIPTION>)...

<!-- schema generated by tfplugindocs -->
## Schema

### Optional

- **exclude_modsec_rule_ids** (List of Number) A list of modsecurity rules IDs to be excluded from the data set.
- **id** (String) The ID of this resource.
- **publishers** (List of String) A list of publishers to be used as filters for the data set.
- **tags** (List of String) A list of tags to be used as filters for the data set.

### Read-only

- **rules** (List of Object) The list of rules that results from any given combination of filters. (see [below for nested schema](#nestedatt--rules))

<a id="nestedatt--rules"></a>
### Nested Schema for `rules`

Read-only:

- **latest_revision_number** (Number) <MISSING DESCRIPTION>
- **modsec_rule_id** (Number) <MISSING DESCRIPTION>
- **type** (String) <MISSING DESCRIPTION>

[Integralist]

Interestingly this isn't consistent behaviour 🤔

The following nested schema data was getting the descriptions templated into the Markdown as we would have hoped...

Schema: map[string]*schema.Schema{
	"service_id": {
		Type:        schema.TypeString,
		Required:    true,
		ForceNew:    true,
		Description: "Service Id",
	},

	"acl_id": {
		Type:        schema.TypeString,
		Required:    true,
		ForceNew:    true,
		Description: "ACL Id",
	},
	"entry": {
		Type:        schema.TypeSet,
		Optional:    true,
		Description: "ACL Entries",
		MaxItems:    gofastly.MaximumACLSize,
		Elem: &schema.Resource{
			Schema: map[string]*schema.Schema{
				"id": {
					Type:        schema.TypeString,
					Description: "",
					Computed:    true,
				},
				"ip": {
					Type:        schema.TypeString,
					Description: "An IP address that is the focus for the ACL",
					Required:    true,
				},
				"subnet": {
					Type:        schema.TypeString,
					Optional:    true,
					Description: "An optional subnet mask applied to the IP address",
				},
				"negated": {
					Type:        schema.TypeBool,
					Optional:    true,
					Default:     false,
					Description: "A boolean that will negate the match if true",
				},
				"comment": {
					Type:        schema.TypeString,
					Optional:    true,
					Description: "A personal freeform descriptive note",
				},
			},
		},
	},
},

The generated Markdown shows the Description field values are being used:

<a id="nestedblock--entry"></a>
### Nested Schema for `entry`

Required:

- **ip** (String) An IP address that is the focus for the ACL

Optional:

- **comment** (String) A personal freeform descriptive note
- **negated** (Boolean) A boolean that will negate the match if true
- **subnet** (String) An optional subnet mask applied to the IP address

Read-only:

- **id** (String) The ID of this resource.

Also interesting is that it has been able to generate an adequate description for the 'read-only' ID field (which in the Schema was left with a blank string for its value) 🤔

[paultyng]

The explanation for this goes in to Terraform internals. And just an aside, to debug you may want to start with running terraform providers schema -json, this is the JSON file the tool works with so easier to debug in there.

The key there is that this is a List of Object, not a Block List. Those are two distinct different types in HCL/Terraform. A block has attributes, but an object type has fields (in syntax blocks look like thing { but an object attribute looks like thing = {).

Object fields don't have metadata currently, so they can't contain attributes like description/sensitive, etc. helper/schema was written pretty early in the evolution of Terraform and its abstraction doesn't fully match the concepts of schema going across the wire.

We do have some thoughts on how to address this for document generation, basically decoupling the the generation from the schema JSON so that it can potentially have richer information supplied, but its probably a ways down the road at this point.

[Integralist]

Thanks @paultyng for the feedback/update 👍🏻

[paultyng]

Also worth noting there is an ongoing internal project to look at normalizing these things so you don't have the trade-offs, if/when that lands we can update the SDK and generator to use it, but probably a ways out.

Unfortunately for the time being this probably means you will have a few docs you'll need to write schema for by hand (or just sed in some descriptions or something).

[elXaptation]

@paultyng Thank you for the details. Is this work/effort being tracked anywhere I can follow along?

[pontovinte]

I was about to ask the same, is there someone actively working on this?

[Macaber]

I was about to ask the same, is there someone actively working on this?