SDKv2 and PF Read return values for fully computed nested properties when importing #2180

VenelinMartinov · 2024-07-12T13:58:48Z

What happened?

During pulumi/pulumi-gcp#2096 it was discovered that imports don't generate the right state for cases where there is a fully computed nested property.

We return a value which then gets picked up as an input and makes the engine generate invalid code for any resource which has a nested computed property. This impacts imports and maybe refresh since we do a preview after the Read in refresh.

This looks like a regression on #453 The code was refactored in #1812 and it seems the logic was dropped there for nested properties.

Example

See TestRefreshExtractInputsFromOutputsListOfObjects and pulumi/pulumi-gcp#2096

Output of `pulumi about`

.

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

The text was updated successfully, but these errors were encountered:

VenelinMartinov · 2024-07-12T14:03:11Z

GRPC:

{
    "method": "/pulumirpc.ResourceProvider/Read",
    "request": {
        "id": "pulumi-development/us-central1-a/my-instance",
        "urn": "urn:pulumi:dev2::gcp_2096::gcp:compute/instance:Instance::default",
        "properties": {}
    },
    "response": {
        "id": "projects/pulumi-development/zones/us-central1-a/instances/my-instance",
        "properties": {
            "__meta": "{\"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0\":{\"create\":1200000000000,\"delete\":1200000000000,\"update\":1200000000000},\"schema_version\":\"6\"}",
            "advancedMachineFeatures": null,
            "allowStoppingForUpdate": null,
            "attachedDisks": [
                {
                    "deviceName": "persistent-disk-1",
                    "diskEncryptionKeyRaw": {
                        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                        "value": ""
                    },
                    "diskEncryptionKeySha256": "",
                    "kmsKeySelfLink": "",
                    "mode": "READ_WRITE",
                    "source": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/disks/data-disk2-7bd8ac7"
                },
                {
                    "deviceName": "persistent-disk-2",
                    "diskEncryptionKeyRaw": {
                        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                        "value": ""
                    },
                    "diskEncryptionKeySha256": "",
                    "kmsKeySelfLink": "",
                    "mode": "READ_WRITE",
                    "source": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/disks/data-disk3-c3647b6"
                },
                {
                    "deviceName": "persistent-disk-3",
                    "diskEncryptionKeyRaw": {
                        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                        "value": ""
                    },
                    "diskEncryptionKeySha256": "",
                    "kmsKeySelfLink": "",
                    "mode": "READ_WRITE",
                    "source": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/disks/data-disk1-ef7fdc0"
                }
            ],
            "bootDisk": {
                "autoDelete": true,
                "deviceName": "persistent-disk-0",
                "diskEncryptionKeyRaw": {
                    "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                    "value": ""
                },
                "diskEncryptionKeySha256": "",
                "initializeParams": {
                    "enableConfidentialCompute": false,
                    "image": "https://www.googleapis.com/compute/beta/projects/debian-cloud/global/images/debian-11-bullseye-v20240709",
                    "labels": {},
                    "provisionedIops": 0,
                    "provisionedThroughput": 0,
                    "resourceManagerTags": {},
                    "size": 10,
                    "type": "pd-standard"
                },
                "kmsKeySelfLink": "",
                "mode": "READ_WRITE",
                "source": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/disks/my-instance"
            },
            "canIpForward": false,
            "confidentialInstanceConfig": null,
            "cpuPlatform": "Intel Haswell",
            "currentStatus": "RUNNING",
            "deletionProtection": false,
            "description": "",
            "desiredStatus": null,
            "effectiveLabels": {},
            "enableDisplay": false,
            "guestAccelerators": [],
            "hostname": "",
            "id": "projects/pulumi-development/zones/us-central1-a/instances/my-instance",
            "instanceId": "6242762773445260037",
            "labelFingerprint": "42WmSpB8rSM=",
            "labels": {},
            "machineType": "n1-standard-1",
            "metadata": {},
            "metadataFingerprint": "EAsrhpzp-qs=",
            "metadataStartupScript": null,
            "minCpuPlatform": "",
            "name": "my-instance",
            "networkInterfaces": [
                {
                    "accessConfigs": [],
                    "aliasIpRanges": [],
                    "internalIpv6PrefixLength": 0,
                    "ipv6AccessConfigs": [],
                    "ipv6AccessType": "",
                    "ipv6Address": "2600:2d00:4000:a5ff:a80:41:0:0",
                    "name": "nic0",
                    "network": "https://www.googleapis.com/compute/v1/projects/pulumi-development/global/networks/default",
                    "networkAttachment": "",
                    "networkIp": "10.128.0.65",
                    "nicType": "",
                    "queueCount": 0,
                    "securityPolicy": "",
                    "stackType": "IPV4_ONLY",
                    "subnetwork": "https://www.googleapis.com/compute/v1/projects/pulumi-development/regions/us-central1/subnetworks/default",
                    "subnetworkProject": "pulumi-development"
                }
            ],
            "networkPerformanceConfig": null,
            "params": null,
            "partnerMetadata": null,
            "project": "pulumi-development",
            "pulumiLabels": {},
            "reservationAffinity": null,
            "resourcePolicies": null,
            "scheduling": {
                "automaticRestart": true,
                "instanceTerminationAction": "",
                "localSsdRecoveryTimeout": null,
                "maintenanceInterval": "",
                "maxRunDuration": null,
                "minNodeCpus": 0,
                "nodeAffinities": [],
                "onHostMaintenance": "MIGRATE",
                "onInstanceStopAction": null,
                "preemptible": false,
                "provisioningModel": "STANDARD"
            },
            "scratchDisks": [],
            "selfLink": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/instances/my-instance",
            "serviceAccount": null,
            "shieldedInstanceConfig": {
                "enableIntegrityMonitoring": true,
                "enableSecureBoot": false,
                "enableVtpm": true
            },
            "tags": [],
            "tagsFingerprint": "42WmSpB8rSM=",
            "zone": "us-central1-a"
        },
        "inputs": {
            "__defaults": [],
            "attachedDisks": [
                {
                    "__defaults": [],
                    "deviceName": "persistent-disk-1",
                    "diskEncryptionKeyRaw": {
                        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                        "value": ""
                    },
                    "diskEncryptionKeySha256": "",
                    "kmsKeySelfLink": "",
                    "mode": "READ_WRITE",
                    "source": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/disks/data-disk2-7bd8ac7"
                },
                {
                    "__defaults": [],
                    "deviceName": "persistent-disk-2",
                    "diskEncryptionKeyRaw": {
                        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                        "value": ""
                    },
                    "diskEncryptionKeySha256": "",
                    "kmsKeySelfLink": "",
                    "mode": "READ_WRITE",
                    "source": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/disks/data-disk3-c3647b6"
                },
                {
                    "__defaults": [],
                    "deviceName": "persistent-disk-3",
                    "diskEncryptionKeyRaw": {
                        "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
                        "value": ""
                    },
                    "diskEncryptionKeySha256": "",
                    "kmsKeySelfLink": "",
                    "mode": "READ_WRITE",
                    "source": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/disks/data-disk1-ef7fdc0"
                }
            ],
            "bootDisk": {
                "__defaults": [],
                "deviceName": "persistent-disk-0",
                "initializeParams": {
                    "__defaults": [],
                    "image": "https://www.googleapis.com/compute/beta/projects/debian-cloud/global/images/debian-11-bullseye-v20240709",
                    "size": 10,
                    "type": "pd-standard"
                },
                "source": "https://www.googleapis.com/compute/v1/projects/pulumi-development/zones/us-central1-a/disks/my-instance"
            },
            "machineType": "n1-standard-1",
            "name": "my-instance",
            "networkInterfaces": [
                {
                    "__defaults": [],
                    "accessConfigs": [],
                    "aliasIpRanges": [],
                    "internalIpv6PrefixLength": 0,
                    "ipv6AccessConfigs": [],
                    "ipv6AccessType": "",
                    "ipv6Address": "2600:2d00:4000:a5ff:a80:41:0:0",
                    "name": "nic0",
                    "network": "https://www.googleapis.com/compute/v1/projects/pulumi-development/global/networks/default",
                    "networkAttachment": "",
                    "networkIp": "10.128.0.65",
                    "nicType": "",
                    "queueCount": 0,
                    "securityPolicy": "",
                    "stackType": "IPV4_ONLY",
                    "subnetwork": "https://www.googleapis.com/compute/v1/projects/pulumi-development/regions/us-central1/subnetworks/default",
                    "subnetworkProject": "pulumi-development"
                }
            ],
            "project": "pulumi-development",
            "scheduling": {
                "__defaults": [],
                "onHostMaintenance": "MIGRATE",
                "provisioningModel": "STANDARD"
            },
            "zone": "us-central1-a"
        }
    },
    "metadata": {
        "kind": "resource",
        "mode": "client",
        "name": "gcp"
    }
}

Schema:

"attached_disk": {
Type:        schema.TypeList,
Optional:    true,
Description: `List of disks attached to the instance`,
Elem: &schema.Resource{
	Schema: map[string]*schema.Schema{
		"source": {
			Type:             schema.TypeString,
			Required:         true,
			DiffSuppressFunc: tpgresource.CompareSelfLinkOrResourceName,
			Description:      `The name or self_link of the disk attached to this instance.`,
		},

		"device_name": {
			Type:        schema.TypeString,
			Optional:    true,
			Computed:    true,
			Description: `Name with which the attached disk is accessible under /dev/disk/by-id/`,
		},

		"mode": {
			Type:         schema.TypeString,
			Optional:     true,
			Default:      "READ_WRITE",
			ValidateFunc: validation.StringInSlice([]string{"READ_WRITE", "READ_ONLY"}, false),
			Description:  `Read/write mode for the disk. One of "READ_ONLY" or "READ_WRITE".`,
		},

		"disk_encryption_key_raw": {
			Type:        schema.TypeString,
			Optional:    true,
			Sensitive:   true,
			Description: `A 256-bit customer-supplied encryption key, encoded in RFC 4648 base64 to encrypt this disk. Only one of kms_key_self_link and disk_encryption_key_raw may be set.`,
		},

		"kms_key_self_link": {
			Type:             schema.TypeString,
			Optional:         true,
			DiffSuppressFunc: tpgresource.CompareSelfLinkRelativePaths,
			Computed:         true,
			Description:      `The self_link of the encryption key that is stored in Google Cloud KMS to encrypt this disk. Only one of kms_key_self_link and disk_encryption_key_raw may be set.`,
		},

		"disk_encryption_key_sha256": {
			Type:        schema.TypeString,
			Computed:    true,
			Description: `The RFC 4648 base64 encoded SHA-256 hash of the customer-supplied encryption key that protects this resource.`,
		},
	},
},
},

VenelinMartinov · 2024-07-12T14:22:47Z

Confirmed regression in #1812

This adds test cases for how TF types are handled by `ExtractInputsFromOutputs` in combination with `Default` and `Computed` and `MaxItems: 1` `ExtractInputsFromOutputs` is the function in `schema.go` responsible for generating the inputs from resource outputs when a resource is imported. Its output is what the engine then uses for generating the code for the import. Notably, nested computed values are wrong for both max items one and non-max items one. In #2180 we found a regression there, so this fills in some missing coverage. Extracting the tests to make the changes more apparent and to make sure we don't regress in #2181

VenelinMartinov · 2024-07-23T09:30:47Z

After adding test coverage in PF for this in #2225 it seems that PF is also affected by this issue. This means this affects all bridged resources.

This adds adds missing test coverage for `ExtractInputsFromOutputs` for various PF schemas. `ExtractInputsFromOutputs` is a shared function so we should have coverage on both sides to make sure we don't cause regressions when fixing one. Similar to #2224 but on the PF side. This is in preparation for #2181 for #2180. Seeing some more evidence of #2218 here.

Fixes an issue with the SDKv2 and PF input extraction. During import we wrongly return values for properties which are fully computed which causes invalid programs to be generated. ~The object check in `CastToTypeObject` was wrong for SDKV2 objects: https://github.com/pulumi/pulumi-terraform-bridge/blob/48fd41bd16e2f7f933bb9390ab7619d52faabb6d/pkg/tfshim/util/types.go#L36~ The `CastToTypeObject` object check is wrong for both SDKv2 and PF. For the SDKV2 only sets and lists can have a `Resource` `.Elem`. For PF it list-nested blocks are represented with a `Resource` `.Elem` in the shim layer, so the check did not catch these. PF details here: #2231 This fixes the check and adds regression tests for the broken imports which resulted from 2180. Test coverage added in #2224 for sdkv2 and #2225 for pf. fixes #2180 depends on pulumi/providertest#91 stacked on #2225

pulumi-bot · 2024-07-25T23:11:19Z

This issue has been addressed in PR #2181 and shipped in release v3.88.0.

VenelinMartinov added kind/bug Some behavior is incorrect or out of spec impact/regression Something that used to work, but is now broken and removed impact/regression Something that used to work, but is now broken labels Jul 12, 2024

VenelinMartinov added the impact/regression Something that used to work, but is now broken label Jul 12, 2024

VenelinMartinov self-assigned this Jul 12, 2024

VenelinMartinov added the p1 A bug severe enough to be the next item assigned to an engineer label Jul 12, 2024

This was referenced Jul 12, 2024

Fix nested computed read #2181

Merged

Add import to pulumitest pulumi/providertest#91

Merged

This was referenced Jul 22, 2024

Add extract schema inputs tests for sdkv2 #2224

Merged

PF extract inputs from outputs tests #2225

Merged

VenelinMartinov mentioned this issue Jul 23, 2024

Add more shim layer tests for pf #2231

Closed

VenelinMartinov changed the title ~~SDKV2 Read returns values for fully computed nested properties when importing~~ SDKv2 and PF Read return values for fully computed nested properties when importing Jul 23, 2024

VenelinMartinov closed this as completed in #2181 Jul 23, 2024

pulumi-bot added the resolution/fixed This issue was fixed label Jul 23, 2024

VenelinMartinov mentioned this issue Jul 23, 2024

Incorrect assumptions about SDK-v2 TypeMap encoding #1828

Open

mjeffryes added this to the 0.108 milestone Aug 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SDKv2 and PF Read return values for fully computed nested properties when importing #2180

SDKv2 and PF Read return values for fully computed nested properties when importing #2180

VenelinMartinov commented Jul 12, 2024 •

edited

Loading

VenelinMartinov commented Jul 12, 2024

VenelinMartinov commented Jul 12, 2024

VenelinMartinov commented Jul 23, 2024 •

edited

Loading

pulumi-bot commented Jul 25, 2024

SDKv2 and PF Read return values for fully computed nested properties when importing #2180

SDKv2 and PF Read return values for fully computed nested properties when importing #2180

Comments

VenelinMartinov commented Jul 12, 2024 • edited Loading

What happened?

Example

Output of pulumi about

Additional context

Contributing

VenelinMartinov commented Jul 12, 2024

VenelinMartinov commented Jul 12, 2024

VenelinMartinov commented Jul 23, 2024 • edited Loading

pulumi-bot commented Jul 25, 2024

VenelinMartinov commented Jul 12, 2024 •

edited

Loading

Output of `pulumi about`

VenelinMartinov commented Jul 23, 2024 •

edited

Loading