Mildwonkey/protocolv6 #27699

mildwonkey · 2021-02-05T20:58:49Z

Plugin protocol v6 take 3: Yes, this is my third pass at the same PR.

I've incorporated the previous PR's feedback into new commits in a fresh branch so that the commits are (gasp) useful to review individually again. Thanks to everyone who chimed in, this has all been very helpful.

The only thing I've removed from this PR (besides bugs!) vs the earlier one is the panic in format/diff when a NestedType is encountered. I promise I have not forgotten that this needs to be done.

@paddycarver : I did make the various message name changes we discussed in the protocol this time around, so please do let me know if you see anything still missing.

This is the first commit for plugin protocol v6. This is currently unused (dead) code; future commits will add the necessary conversion packages, extend configschema, and modify the providers.Interface. The new plugin protocol includes the following changes: - A new field has been added to Attribute: NestedType. This will be the key new feature in plugin protocol v6 - Several massages were renamed for consistency with the verb-noun pattern seen in _most_ messages. - The prepared_config has been removed from PrepareProviderConfig (renamed ValidateProviderConfig), as it has never been used. - The provisioner service has been removed entirely. This has no impact on built-in provisioners. 3rd party provisioners are not supported by the SDK and are not included in this protocol at all.

This commit adds a new field, NestedType, to the Attribute schema, and extends the current Attribute decoderSpec to account for the new type. The codepaths are mostly unused and included in a separate commit to verify that the included changes do not impact any other tests yet.

- rename ProposedNewObject to ProposedNew: Now that there is an actual configschema.Object it will be clearer if the function names match the type the act upon. - extract attribute-handling logic from assertPlanValid and extend A new function, assertPlannedAttrsValid, takes the existing functionality and extends it to validate attributes with NestedTypes. The NestedType-specific handling is in assertPlannedObjectValid, which is very similar to the block-handling logic, except that nulls are a valid plan (an attribute can be null, but not a block).

plugin6 includes a `convert` package to handle conversion between the plugin protocol and configschema, and the GRPCProviderPlugin interface implementation for protocol v6.

codecov · 2021-02-05T20:58:55Z

Codecov Report

Merging #27699 (8c2abbc) into master (f279bf5) will increase coverage by 0.09%.
The diff coverage is 66.94%.

Impacted Files	Coverage Δ
plans/objchange/all_null.go	`14.28% <0.00%> (-85.72%)`	⬇️
plans/objchange/plan_valid.go	`37.00% <34.65%> (-4.48%)`	⬇️
plugin6/convert/diagnostics.go	`50.64% <50.64%> (ø)`
plugin6/convert/schema.go	`65.69% <65.69%> (ø)`
plans/objchange/objchange.go	`80.47% <85.29%> (+3.13%)`	⬆️
configs/configschema/implied_type.go	`79.41% <92.30%> (+41.91%)`	⬆️
configs/configschema/decoder_spec.go	`81.60% <100.00%> (+3.23%)`	⬆️
configs/configschema/empty_value.go	`92.30% <100.00%> (+0.64%)`	⬆️
terraform/node_resource_abstract_instance.go	`73.14% <100.00%> (ø)`
terraform/node_resource_plan.go	`96.11% <0.00%> (-1.95%)`	⬇️
... and 7 more

jbardin · 2021-02-09T14:45:13Z

The new objchange.ProposedNew doe not appear to descend into the new NestedTyped objects to construct the proposed new objects. The code it taking the entire attribute as a single value and applying the config or prior values, which will fail for computed values. What we need here is to recursively descend similarly to proposeNewNestedBlock to determine what is optional, required and computed, to set those accordingly.

I don't recall where validation for things like computed values set in the configuration happens, but that should be done for nested types as well.

mildwonkey · 2021-02-09T14:49:02Z

🤦 Your comment triggered some unfortunately and uncomfortable memories of me seeing that and forgetting - I'll go add some tests, then fix them! Thank you ❤️

with NestedType objects. There are a handful of mostly cosmetic changes in this PR which likely make the diff awkward to read; I renamed several functions to (hopefully) clarifiy which funcs worked with Blocks vs other types. I also extracted some small code snippets into their own functions for reusability. The code that descends into attributes with NestedTypes is similar to the block-handling code, and differs in all the ways blocks and attributes differ: null is valid for attributes, unlike blocks which can only be present or empty.

plans/objchange/objchange_test.go

jbardin · 2021-02-11T21:04:00Z

plans/objchange/objchange.go

+			if config.Type().IsTupleType() {
+				newV = cty.EmptyTupleVal
+			} else {
+				newV = cty.ListValEmpty(schema.ImpliedType())


I think this is where we're going to want cty.NullVal(schema.ImpliedType()). I'm not sure offhand how to handle the tuple case, I'll have to run some tests

jbardin · 2021-02-11T21:05:41Z

plans/objchange/objchange.go

 			} else {
-				newV = cty.SetValEmpty(blockType.Block.ImpliedType())
+				newV = cty.MapValEmpty(schema.ImpliedType())


And again here, I think we want a case for a null value rather than an empty value. Just like the tuple case above, I'm not sure what the object case will be, but I'll see what the tests look like.

A handful of bugs popped up while extending the testing in plans/objchange. The main themes were failing to recurse through deeply nested NestedType attributes and improperly building up the ImpliedType. This commit fixes those issues and extends the test coverage to match.

…osedNewNestedType

mildwonkey · 2021-02-12T18:43:26Z

@jbardin addressing those last two comments led me down a delightful rabbit hole of rough edges and bugs in the supporting code in configs/configschema, so there are a handful of new changes and a bushelful of new tests, but I did try to keep the commits separate so it's not too hard to review 😁

Thank you so much for all the reviews.

paddycarver · 2021-02-17T00:17:40Z

docs/plugin-protocol/tfplugin6.0.proto

+// Schema is the configuration schema for a Resource or Provider.
+message Schema {
+    message Block {
+        int64 version = 1;


I'm still not super clear why blocks get versions separate from the schema version or when you would want to set it/what you would want to set it to.

Good question and I also do not know; would you like me to investigate removing this field entirely?

UPDATE: Each resource can have its own schema, and that's where it gets set. (ha i just refactored all this code, you'd think I would have remembered!)
Terraform uses that to determine if the resource' state needs to be updated to match the new schema:

terraform/terraform/upgrade_resource_state.go

Line 44 in d89c8f5

if src.SchemaVersion > currentVersion {

What I don't know is usage statistics.

@jbardin is disagreeing with me and says everything I thought was looking at the resource-specific Block schema version is actually looking at the overall Schema.Version, so I'm back to 🤷 , sorry

paddycarver · 2021-02-17T00:19:06Z

docs/plugin-protocol/tfplugin6.0.proto

+        DynamicValue state = 1;
+        repeated Diagnostic diagnostics = 2;
+    }
+}


I noticed we removed provisioners from the protocol--is that intentional?

paddycarver · 2021-02-17T00:20:07Z

I'm incredibly late to the party, but so glad this merged. 🎉 Only a couple comment/questions from my glance through it, one about provisioners being removed and one about "version" in blocks.

ghost · 2021-03-19T01:50:11Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

mildwonkey added 4 commits February 5, 2021 13:19

plugin6: new plugin package to support plugin protocol v6

1a8d873

plugin6 includes a `convert` package to handle conversion between the plugin protocol and configschema, and the GRPCProviderPlugin interface implementation for protocol v6.

mildwonkey requested review from a team February 5, 2021 21:00

mildwonkey added 2 commits February 10, 2021 09:58

ContainsSensitive must descend into nested objects

6aa90a5

jbardin reviewed Feb 11, 2021

View reviewed changes

mildwonkey added 2 commits February 12, 2021 13:34

return the properly-typed nulls, instead of empty containers, in prop…

8c2abbc

…osedNewNestedType

bflad mentioned this pull request Feb 15, 2021

resource/aws_iam_role: Add exclusive list support for a role's inline and managed policies hashicorp/terraform-provider-aws#5904

Merged

mildwonkey mentioned this pull request Feb 16, 2021

Mildwonkey/nested type format #27793

Merged

jbardin approved these changes Feb 16, 2021

View reviewed changes

mildwonkey merged commit 7943312 into master Feb 16, 2021

mildwonkey deleted the mildwonkey/protocolv6 branch February 16, 2021 20:29

paddycarver reviewed Feb 17, 2021

View reviewed changes

This was referenced Mar 10, 2021

Confirm/ensure support of Terraform 0.15 hashicorp/terraform-ls#425

Closed

Reflect protocol v6 changes in schema JSON output #28050

Closed

Reflect v0.15 schema changes hashicorp/terraform-json#28

Merged

This was referenced Mar 18, 2021

Initial Terraform Plugin Protocol Version 6 (tfprotov6) Implementation hashicorp/terraform-plugin-go#65

Closed

Terraform Plugin Protocol Version 6 / Providers Schema JSON Version 0.2 Support hashicorp/terraform-plugin-docs#49

Closed

ghost locked as resolved and limited conversation to collaborators Mar 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mildwonkey/protocolv6 #27699

Mildwonkey/protocolv6 #27699

mildwonkey commented Feb 5, 2021 •

edited

Loading

codecov bot commented Feb 5, 2021 •

edited

Loading

jbardin commented Feb 9, 2021

mildwonkey commented Feb 9, 2021

jbardin Feb 11, 2021

jbardin Feb 11, 2021

mildwonkey commented Feb 12, 2021

paddycarver Feb 17, 2021

mildwonkey Feb 17, 2021 •

edited

Loading

mildwonkey Feb 17, 2021

paddycarver Feb 17, 2021

paddycarver commented Feb 17, 2021

ghost commented Mar 19, 2021

Mildwonkey/protocolv6 #27699

Mildwonkey/protocolv6 #27699

Conversation

mildwonkey commented Feb 5, 2021 • edited Loading

codecov bot commented Feb 5, 2021 • edited Loading

Codecov Report

jbardin commented Feb 9, 2021

mildwonkey commented Feb 9, 2021

jbardin Feb 11, 2021

Choose a reason for hiding this comment

jbardin Feb 11, 2021

Choose a reason for hiding this comment

mildwonkey commented Feb 12, 2021

paddycarver Feb 17, 2021

Choose a reason for hiding this comment

mildwonkey Feb 17, 2021 • edited Loading

Choose a reason for hiding this comment

mildwonkey Feb 17, 2021

Choose a reason for hiding this comment

paddycarver Feb 17, 2021

Choose a reason for hiding this comment

paddycarver commented Feb 17, 2021

ghost commented Mar 19, 2021

mildwonkey commented Feb 5, 2021 •

edited

Loading

codecov bot commented Feb 5, 2021 •

edited

Loading

mildwonkey Feb 17, 2021 •

edited

Loading