Skip to content

Commit

Permalink
tfsdk: Initial ResourceWithUpgradeState implementation (#292)
Browse files Browse the repository at this point in the history
Reference: #42
Reference: #228

Support provider defined `UpgradeResourceState` RPC handling, by introducing an optional `ResourceWithUpgradeState` interface type, with an `UpgradeState` method. Each underlying state version upgrade implementation is expected to consume the prior state, perform any necessary data manipulations, then respond with the upgraded state.

This framework implementation differs from the terraform-plugin-sdk implementation:

- State upgraders are specified via a mapping, rather than a slice with underlying version field. This should prevent certain classes of coding issues.
- State upgraders must be wholly contained from the prior state version to the current schema version. The framework does not loop through each successive version because attempting to recreate the `tfprotov6.RawState` for each intermediate version request would be very problematic. For example, terraform-plugin-go does not implement functionality for marshalling a `RawState`. Provider developers can use their own coding techniques to reduce code duplications when multiple versions need the same logic.
- Specifying the full prior schema is now an optional implementation detail. Working with the lower level data types is more challenging, however this has been a repeated feature request.

There are some quirks and potential future enhancements to the framework `UpgradeResourceState` handling:

- Past and current versions Terraform CLI will call `UpgradeResourceState` even if the state version matches the current schema version. This implementation keeps the framework's prior logic to roundtrip the existing state into the upgraded state. It may be possible to stop this Terraform CLI behavior with protocol version 6, although the logic would need to remain for backwards compatibility.
- It may be possible to help provider developers simplify logic by attempting to automatically populate compatible parts of the upgraded state from the prior state. This can potentially be done at a later time.
  • Loading branch information
bflad authored Apr 21, 2022
1 parent 361a0bd commit 04be243
Show file tree
Hide file tree
Showing 9 changed files with 931 additions and 147 deletions.
3 changes: 3 additions & 0 deletions .changelog/292.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
```release-note:feature
tfsdk: Added optional `ResourceWithUpgradeState` interface, which allows for provider defined logic when the `UpgradeResourceState` RPC is called
```
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ terraform-plugin-framework is a module for building Terraform providers. It is b

terraform-plugin-framework is still in **technical preview**. We are committed to moving forward with the module, but cannot guarantee any of its interfaces will not change as long as it is in version 0. We're waiting for more feedback, usage, and maturity before we're comfortable committing to APIs with the same years-long support timelines that [terraform-plugin-sdk](https://github.com/hashicorp/terraform-plugin-sdk) brings. We do not expect practitioner experiences to break or change as a result of these changes, only the abstractions surfaced to provider developers.

terraform-plugin-framework is also not at full feature parity with terraform-plugin-sdk yet. Notably, it doesn't offer support for [upgrading resource state](https://github.com/hashicorp/terraform-plugin-framework/issues/42) or [using timeouts](https://github.com/hashicorp/terraform-plugin-framework/issues/62). We plan to add these features soon. See [Which SDK Should I Use?](https://terraform.io/docs/plugin/which-sdk.html) on terraform.io for more information.
terraform-plugin-framework is also not at full feature parity with terraform-plugin-sdk yet. Notably, it doesn't offer support for [using timeouts](https://github.com/hashicorp/terraform-plugin-framework/issues/62). We plan to add these features soon. See [Which SDK Should I Use?](https://terraform.io/docs/plugin/which-sdk.html) on terraform.io for more information.

We believe terraform-plugin-framework is still a suitable and reliable module to build Terraform providers on, and encourage community members that can afford occasional breaking changes to build with it. terraform-plugin-framework will eventually become a new major version of terraform-plugin-sdk, at which point its interfaces will be stable, but we need real-world use and feedback before we can be comfortable making those commitments. When that happens, this repository will be archived.

Expand Down
121 changes: 121 additions & 0 deletions tfsdk/resource_upgrade_state.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
package tfsdk

import (
"context"

"github.com/hashicorp/terraform-plugin-framework/diag"
"github.com/hashicorp/terraform-plugin-go/tfprotov6"
)

// Optional interface on top of Resource that enables provider control over
// the UpgradeResourceState RPC. This RPC is automatically called by Terraform
// when the current Schema type Version field is greater than the stored state.
// Terraform does not store previous Schema information, so any breaking
// changes to state data types must be handled by providers.
//
// Terraform CLI can execute the UpgradeResourceState RPC even when the prior
// state version matches the current schema version. The framework will
// automatically intercept this request and attempt to respond with the
// existing state. In this situation the framework will not execute any
// provider defined logic, so declaring it for this version is extraneous.
type ResourceWithUpgradeState interface {
// A mapping of prior state version to current schema version state upgrade
// implementations. Only the specified state upgrader for the prior state
// version is called, rather than each version in between, so it must
// encapsulate all logic to convert the prior state to the current schema
// version.
//
// Version keys begin at 0, which is the default schema version when
// undefined. The framework will return an error diagnostic should the
// requested state version not be implemented.
UpgradeState(context.Context) map[int64]ResourceStateUpgrader
}

// Implementation handler for a UpgradeResourceState operation.
//
// This is used to encapsulate all upgrade logic from a prior state to the
// current schema version when a Resource implements the
// ResourceWithUpgradeState interface.
type ResourceStateUpgrader struct {
// Schema information for the prior state version. While not required,
// setting this will populate the UpgradeResourceStateRequest type State
// field similar to other Resource data types. This allows for easier data
// handling such as calling Get() or GetAttribute().
//
// If not set, prior state data is available in the
// UpgradeResourceStateRequest type RawState field.
PriorSchema *Schema

// Provider defined logic for upgrading a resource state from the prior
// state version to the current schema version.
//
// The context.Context parameter contains framework-defined loggers and
// supports request cancellation.
//
// The UpgradeResourceStateRequest parameter contains the prior state data.
// If PriorSchema was set, the State field will be available. Otherwise,
// the RawState must be used.
//
// The UpgradeResourceStateResponse parameter should contain the upgraded
// state data and can be used to signal any logic warnings or errors.
StateUpgrader func(context.Context, UpgradeResourceStateRequest, *UpgradeResourceStateResponse)
}

// Request information for the provider logic to update a resource state
// from a prior state version to the current schema version. An instance of
// this is supplied as a parameter to the StateUpgrader function defined in a
// ResourceStateUpgrader, which ultimately comes from a Resource's
// UpgradeState method.
type UpgradeResourceStateRequest struct {
// Previous state of the resource in JSON (Terraform CLI 0.12 and later)
// or flatmap format, depending on which version of Terraform CLI last
// wrote the resource state. This data is always available, regardless
// whether the wrapping ResourceStateUpgrader type PriorSchema field was
// present.
//
// This is advanced functionality for providers wanting to skip the full
// redeclaration of older schemas and instead use lower level handlers to
// transform data. A typical implementation for working with this data will
// call the Unmarshal() method.
RawState *tfprotov6.RawState

// Previous state of the resource if the wrapping ResourceStateUpgrader
// type PriorSchema field was present. When available, this allows for
// easier data handling such as calling Get() or GetAttribute().
State *State
}

// Response information for the provider logic to update a resource state
// from a prior state version to the current schema version. An instance of
// this is supplied as a parameter to the StateUpgrader function defined in a
// ResourceStateUpgrader, which ultimately came from a Resource's
// UpgradeState method.
type UpgradeResourceStateResponse struct {
// Diagnostics report errors or warnings related to upgrading the resource
// state. An empty slice indicates a successful operation with no warnings
// or errors generated.
Diagnostics diag.Diagnostics

// Upgraded state of the resource, which should match the current schema
// version. If set, this will override State.
//
// This field is intended only for advanced provider functionality, such as
// skipping the full redeclaration of older schemas or using lower level
// handlers to transform data. Call tfprotov6.NewDynamicValue() to set this
// value.
//
// All data must be populated to prevent data loss during the upgrade
// operation. No prior state data is copied automatically.
DynamicValue *tfprotov6.DynamicValue

// Upgraded state of the resource, which should match the current schema
// version. If DynamicValue is set, it will override this value.
//
// This field allows for easier data handling such as calling Set() or
// SetAttribute(). It is generally recommended over working with the lower
// level types and functionality required for DynamicValue.
//
// All data must be populated to prevent data loss during the upgrade
// operation. No prior state data is copied automatically.
State State
}
177 changes: 151 additions & 26 deletions tfsdk/serve.go
Original file line number Diff line number Diff line change
Expand Up @@ -557,16 +557,6 @@ func (s *server) upgradeResourceState(ctx context.Context, req *tfprotov6.Upgrad
return
}

// This implementation assumes the current schema is the only valid schema
// for the given resource and will return an error if any mismatched prior
// state is given. This matches prior behavior of the framework, but is now
// more explicit in error handling, rather than just passing through any
// potentially errant prior state, which should have resulted in a similar
// error further in the resource lifecycle.
//
// TODO: Implement resource state upgrades, rather than just using the
// current resource schema.
// Reference: https://github.com/hashicorp/terraform-plugin-framework/issues/42
resourceSchema, diags := resourceType.GetSchema(ctx)

resp.Diagnostics.Append(diags...)
Expand All @@ -575,32 +565,167 @@ func (s *server) upgradeResourceState(ctx context.Context, req *tfprotov6.Upgrad
return
}

resourceSchemaType := resourceSchema.TerraformType(ctx)
// Terraform CLI can call UpgradeResourceState even if the stored state
// version matches the current schema. Presumably this is to account for
// the previous terraform-plugin-sdk implementation, which handled some
// state fixups on behalf of Terraform CLI. When this happens, we do not
// want to return errors for a missing ResourceWithUpgradeState
// implementation or an undefined version within an existing
// ResourceWithUpgradeState implementation as that would be confusing
// detail for provider developers. Instead, the framework will attempt to
// roundtrip the prior RawState to a State matching the current Schema.
//
// TODO: To prevent provider developers from accidentially implementing
// ResourceWithUpgradeState with a version matching the current schema
// version which would never get called, the framework can introduce a
// unit test helper.
// Reference: https://github.com/hashicorp/terraform-plugin-framework/issues/113
if req.Version == resourceSchema.Version {
logging.FrameworkTrace(ctx, "UpgradeResourceState request version matches current Schema version, using framework defined passthrough implementation")

rawStateValue, err := req.RawState.Unmarshal(resourceSchemaType)
resourceSchemaType := resourceSchema.TerraformType(ctx)

if err != nil {
rawStateValue, err := req.RawState.Unmarshal(resourceSchemaType)

if err != nil {
resp.Diagnostics.AddError(
"Unable to Read Previously Saved State for UpgradeResourceState",
"There was an error reading the saved resource state using the current resource schema.\n\n"+
"If this resource state was last refreshed with Terraform CLI 0.11 and earlier, it must be refreshed or applied with an older provider version first. "+
"If you manually modified the resource state, you will need to manually modify it to match the current resource schema. "+
"Otherwise, please report this to the provider developer:\n\n"+err.Error(),
)
return
}

// NewDynamicValue will ensure the Msgpack field is set for Terraform CLI
// 0.12 through 0.14 compatibility when using terraform-plugin-mux tf6to5server.
// Reference: https://github.com/hashicorp/terraform-plugin-framework/issues/262
upgradedStateValue, err := tfprotov6.NewDynamicValue(resourceSchemaType, rawStateValue)

if err != nil {
resp.Diagnostics.AddError(
"Unable to Convert Previously Saved State for UpgradeResourceState",
"There was an error converting the saved resource state using the current resource schema. "+
"This is always an issue in the Terraform Provider SDK used to implement the resource and should be reported to the provider developers.\n\n"+
"Please report this to the provider developer:\n\n"+err.Error(),
)
return
}

resp.UpgradedState = &upgradedStateValue

return
}

resource, diags := resourceType.NewResource(ctx, s.p)

resp.Diagnostics.Append(diags...)

if resp.Diagnostics.HasError() {
return
}

resourceWithUpgradeState, ok := resource.(ResourceWithUpgradeState)

if !ok {
resp.Diagnostics.AddError(
"Unable to Upgrade Resource State",
"This resource was implemented without an UpgradeState() method, "+
fmt.Sprintf("however Terraform was expecting an implementation for version %d upgrade.\n\n", req.Version)+
"This is always an issue with the Terraform Provider and should be reported to the provider developer.",
)
return
}

resourceStateUpgraders := resourceWithUpgradeState.UpgradeState(ctx)

// Panic prevention
if resourceStateUpgraders == nil {
resourceStateUpgraders = make(map[int64]ResourceStateUpgrader, 0)
}

resourceStateUpgrader, ok := resourceStateUpgraders[req.Version]

if !ok {
resp.Diagnostics.AddError(
"Unable to Upgrade Resource State",
"This resource was implemented with an UpgradeState() method, "+
fmt.Sprintf("however Terraform was expecting an implementation for version %d upgrade.\n\n", req.Version)+
"This is always an issue with the Terraform Provider and should be reported to the provider developer.",
)
return
}

upgradeResourceStateRequest := UpgradeResourceStateRequest{
RawState: req.RawState,
}

if resourceStateUpgrader.PriorSchema != nil {
logging.FrameworkTrace(ctx, "Initializing populated UpgradeResourceStateRequest state from provider defined prior schema and request RawState")

priorSchemaType := resourceStateUpgrader.PriorSchema.TerraformType(ctx)

rawStateValue, err := req.RawState.Unmarshal(priorSchemaType)

if err != nil {
resp.Diagnostics.AddError(
"Unable to Read Previously Saved State for UpgradeResourceState",
fmt.Sprintf("There was an error reading the saved resource state using the prior resource schema defined for version %d upgrade.\n\n", req.Version)+
"Please report this to the provider developer:\n\n"+err.Error(),
)
return
}

upgradeResourceStateRequest.State = &State{
Raw: rawStateValue,
Schema: *resourceStateUpgrader.PriorSchema,
}
}

upgradeResourceStateResponse := UpgradeResourceStateResponse{
State: State{
Schema: resourceSchema,
},
}

// To simplify provider logic, this could perform a best effort attempt
// to populate the response State by looping through all Attribute/Block
// by calling the equivalent of SetAttribute(GetAttribute()) and skipping
// any errors.

logging.FrameworkDebug(ctx, "Calling provider defined StateUpgrader")
resourceStateUpgrader.StateUpgrader(ctx, upgradeResourceStateRequest, &upgradeResourceStateResponse)
logging.FrameworkDebug(ctx, "Called provider defined StateUpgrader")

resp.Diagnostics.Append(upgradeResourceStateResponse.Diagnostics...)

if resp.Diagnostics.HasError() {
return
}

if upgradeResourceStateResponse.DynamicValue != nil {
logging.FrameworkTrace(ctx, "UpgradeResourceStateResponse DynamicValue set, overriding State")
resp.UpgradedState = upgradeResourceStateResponse.DynamicValue
return
}

if upgradeResourceStateResponse.State.Raw.Type() == nil || upgradeResourceStateResponse.State.Raw.IsNull() {
resp.Diagnostics.AddError(
"Unable to Read Previously Saved State for UpgradeResourceState",
"There was an error reading the saved resource state using the current resource schema. "+
"This resource was implemented in a Terraform Provider SDK that does not support upgrading resource state yet.\n\n"+
"If the resource previously implemented different resource state versions, the provider developers will need to revert back to the previous implementation. "+
"If this resource state was last refreshed with Terraform CLI 0.11 and earlier, it must be refreshed or applied with an older provider version first. "+
"If you manually modified the resource state, you will need to manually modify it to match the current resource schema. "+
"Otherwise, please report this to the provider developer:\n\n"+err.Error(),
"Missing Upgraded Resource State",
fmt.Sprintf("After attempting a resource state upgrade to version %d, the provider did not return any state data. ", req.Version)+
"Preventing the unexpected loss of resource state data. "+
"This is always an issue with the Terraform Provider and should be reported to the provider developer.",
)
return
}

// NewDynamicValue will ensure the Msgpack field is set for Terraform CLI
// 0.12 through 0.14 compatibility when using terraform-plugin-mux tf6to5server.
// Reference: https://github.com/hashicorp/terraform-plugin-framework/issues/262
upgradedStateValue, err := tfprotov6.NewDynamicValue(resourceSchemaType, rawStateValue)
upgradedStateValue, err := tfprotov6.NewDynamicValue(upgradeResourceStateResponse.State.Schema.TerraformType(ctx), upgradeResourceStateResponse.State.Raw)

if err != nil {
resp.Diagnostics.AddError(
"Unable to Convert Previously Saved State for UpgradeResourceState",
"There was an error converting the saved resource state using the current resource schema. "+
"Unable to Convert Upgraded Resource State",
fmt.Sprintf("An unexpected error was encountered when converting the state returned for version %d upgrade to a usable type. ", req.Version)+
"This is always an issue in the Terraform Provider SDK used to implement the resource and should be reported to the provider developers.\n\n"+
"Please report this to the provider developer:\n\n"+err.Error(),
)
Expand Down
22 changes: 11 additions & 11 deletions tfsdk/serve_provider_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,7 @@ type testServeProvider struct {
validateResourceConfigImpl func(context.Context, ValidateResourceConfigRequest, *ValidateResourceConfigResponse)

// upgrade resource state
// TODO: Implement with UpgradeResourceState support
// Reference: https://github.com/hashicorp/terraform-plugin-framework/issues/42
// upgradeResourceStateCalledResourceType string
upgradeResourceStateCalledResourceType string

// read resource request
readResourceCurrentStateValue tftypes.Value
Expand Down Expand Up @@ -635,14 +633,16 @@ var testServeProviderProviderType = tftypes.Object{

func (t *testServeProvider) GetResources(_ context.Context) (map[string]ResourceType, diag.Diagnostics) {
return map[string]ResourceType{
"test_one": testServeResourceTypeOne{},
"test_two": testServeResourceTypeTwo{},
"test_three": testServeResourceTypeThree{},
"test_attribute_plan_modifiers": testServeResourceTypeAttributePlanModifiers{},
"test_config_validators": testServeResourceTypeConfigValidators{},
"test_import_state": testServeResourceTypeImportState{},
"test_upgrade_state": testServeResourceTypeUpgradeState{},
"test_validate_config": testServeResourceTypeValidateConfig{},
"test_one": testServeResourceTypeOne{},
"test_two": testServeResourceTypeTwo{},
"test_three": testServeResourceTypeThree{},
"test_attribute_plan_modifiers": testServeResourceTypeAttributePlanModifiers{},
"test_config_validators": testServeResourceTypeConfigValidators{},
"test_import_state": testServeResourceTypeImportState{},
"test_upgrade_state": testServeResourceTypeUpgradeState{},
"test_upgrade_state_empty": testServeResourceTypeUpgradeStateEmpty{},
"test_upgrade_state_not_implemented": testServeResourceTypeUpgradeStateNotImplemented{},
"test_validate_config": testServeResourceTypeValidateConfig{},
}, nil
}

Expand Down
Loading

0 comments on commit 04be243

Please sign in to comment.