dataproc.Metastoreservice cannot be replaced on changes

[stooj]

Describe what happened

If I create a gcp.dataproc.MetastoreService and then adjust something that would trigger a replacement (like the network), the next pulumi up fails because it tries to create another resource with the same identifier:

    error:   sdk-v2/provider2.go:515: sdk.helper_schema: Error creating Service: googleapi: Error 409: Resource 'projects/pulumi-ce-team/locations/us-central1/services/metastore-srv' already exists
    Details:
    [
      {
        "@type": "type.googleapis.com/google.rpc.ResourceInfo",
        "resourceName": "projects/pulumi-ce-team/locations/us-central1/services/metastore-srv"
      }
    ]: provider=google-beta@8.16.0
    error: 1 error occurred:
        * Error creating Service: googleapi: Error 409: Resource 'projects/pulumi-ce-team/locations/us-central1/services/metastore-srv' already exists
    Details:
    [
      {
        "@type": "type.googleapis.com/google.rpc.ResourceInfo",
        "resourceName": "projects/pulumi-ce-team/locations/us-central1/services/metastore-srv"
      }
    ]

Sample program

Using the simple example in the pulumi docs and adding a network:

import pulumi
import pulumi_gcp as gcp

default = gcp.dataproc.MetastoreService(
    "default",
    network="projects/pulumi-ce-team/global/networks/my-demo-network",
    service_id="metastore-srv",
    location="us-central1",
    port=9080,
    tier="DEVELOPER",
    maintenance_window={
        "hour_of_day": 2,
        "day_of_week": "SUNDAY",
    },
    hive_metastore_config={
        "version": "2.3.6",
    },
    labels={
        "env": "test",
    },
)

1. Run pulumi up
2. Wait a few minutes because this takes a while
3. Change the network to another one you have handy
4. Run pulumi up

Log output

replace-metastore.txt

Affected Resource(s)

dataproc.Metastoreservice

Output of pulumi about

CLI
Version      3.147.0
Go Version   go1.23.5
Go Compiler  gc

Plugins
KIND      NAME    VERSION
resource  gcp     8.16.0
language  python  3.147.0

Host
OS       nixos
Version  24.11 (Vicuna)
Arch     x86_64

This project is written in python: executable='/home/stooj/code/pulumi/stooj/pulumi-support/6546/gcp-metastore_not_recreated-python/.venv/bin/python' version='3.12.7'

Current Stack: stooj/6546/dev

TYPE                                            URN
pulumi:pulumi:Stack                             urn:pulumi:dev::6546::pulumi:pulumi:Stack::6546-dev
pulumi:providers:gcp                            urn:pulumi:dev::6546::pulumi:providers:gcp::default_8_16_0
gcp:dataproc/metastoreService:MetastoreService  urn:pulumi:dev::6546::gcp:dataproc/metastoreService:MetastoreService::default


Found no pending operations associated with dev

Backend
Name           pulumi.com
URL            https://app.pulumi.com/stooj
User           stooj
Organizations  stooj, higara, team-ce, demo, stoo-pulumi-team
Token type     personal

Dependencies:
NAME                        VERSION
aiodns                      3.2.0
aiohttp                     3.10.10
async-timeout               4.0.3
awscli                      2.19.0
azure-cli                   2.67.0
azure-graphrbac             0.61.1
azure-mgmt-common           0.20.0
azure-mgmt-datalake-nspkg   3.0.1
azure-mgmt-kusto            0.3.0
azure-mgmt-managedservices  6.0.0
black                       24.8.0
Brotli                      1.1.0
brotlicffi                  1.1.0.0
poetry                      1.8.4
pulumi_gcp                  8.16.0
PySocks                     1.7.1
redis                       5.1.1
uv                          0.4.30

Pulumi locates its logs in /tmp by default

Additional context

You can work around this by using deleteBeforeReplace.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[VenelinMartinov]

Hey @stooj, thanks for reporting. You should be able to work around the issue with deleteBeforeReplace: https://www.pulumi.com/docs/iac/concepts/options/deletebeforereplace/

This resource is likely "structural" - e.g. it doesn't have its own identity but links one or more other resources. We can use the deleteBeforeReplace for such resource by default by configuring it in the provider resources.go like so:

pulumi-gcp/provider/resources.go

Line 881 in 64eda76

DeleteBeforeReplace: true,

GCP has quite a few of these types of resource and they all need a bit of special handling in the resources.go file.

Feel free to open a PR for this if you want to! Let me know if I can help.