pulumi refresh hangs forever if a Pulumi-managed GCP Storage Bucket was deleted manually

[sofokus-jali-rainio]

Hello!

• Vote on this issue by adding a 👍 reaction
• To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already)

Issue details

Steps to reproduce

I'm trying Pulumi for the first time and I can't seem to get it to refresh the state from the cloud provider when a manual change was made via the cloud console. I'm using Google Cloud as my cloud provider and the basic TypeScript starter template.

Pulumi version 3.17.0.

The commands I've used to reproduce the issue:

1. mkdir pulumi-testing
2. cd pulumi-testing
3. pulumi new gcp-typescript
4. pulumi up -y
5. Manually delete the created bucket via Cloud Console
6. pulumi refresh --logtostderr -v=9

Expected:

Pulumi notices a change has happened in the cloud and updates the local state.

Actual:

The command hangs forever. The console output suggests it tries over and over again to find the deleted bucket.

     pulumi:pulumi:Stack    issue-test-dev  running        
 ~   └─ gcp:storage:Bucket  my-bucket       refreshing     
I1110 09:44:38.134246  260271 eventsink.go:59] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/2.0 404 Not Found
I1110 09:44:38.134283  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/2.0 404 Not Found<{%reset%}>)
I1110 09:44:38.134716  260271 eventsink.go:59] Content-Length: 171
I1110 09:44:38.134738  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Content-Length: 171<{%reset%}>)
I1110 09:44:38.135191  260271 eventsink.go:59] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
I1110 09:44:38.135223  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"<{%reset%}>)
I1110 09:44:38.135570  260271 eventsink.go:59] Cache-Control: no-cache, no-store, max-age=0, must-revalidate
I1110 09:44:38.135591  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Cache-Control: no-cache, no-store, max-age=0, must-revalidate<{%reset%}>)
I1110 09:44:38.135935  260271 eventsink.go:59] Content-Type: application/json; charset=UTF-8
I1110 09:44:38.135963  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Content-Type: application/json; charset=UTF-8<{%reset%}>)
I1110 09:44:38.136333  260271 eventsink.go:59] Date: Wed, 10 Nov 2021 07:44:39 GMT
I1110 09:44:38.136362  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Date: Wed, 10 Nov 2021 07:44:39 GMT<{%reset%}>)
I1110 09:44:38.136713  260271 eventsink.go:59] Expires: Mon, 01 Jan 1990 00:00:00 GMT
I1110 09:44:38.136739  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Expires: Mon, 01 Jan 1990 00:00:00 GMT<{%reset%}>)
I1110 09:44:38.137076  260271 eventsink.go:59] Pragma: no-cache 
I1110 09:44:38.137102  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Pragma: no-cache<{%reset%}>)
I1110 09:44:38.137439  260271 eventsink.go:59] Server: UploadServer
I1110 09:44:38.137460  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Server: UploadServer<{%reset%}>)
I1110 09:44:38.137766  260271 eventsink.go:59] Vary: Origin
I1110 09:44:38.137786  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Vary: Origin<{%reset%}>)
I1110 09:44:38.138101  260271 eventsink.go:59] Vary: X-Origin
I1110 09:44:38.138127  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Vary: X-Origin<{%reset%}>)
I1110 09:44:38.138570  260271 eventsink.go:59] X-Guploader-Uploadid: ADPycdt69SfSITaLX1C3A51gwDy5RiUOjVcvqgxzujUwZ3no5oYowtPVYqEla7fchooU8TmY39_DHmVlRR-u7FarrYQ
I1110 09:44:38.138606  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>X-Guploader-Uploadid: ADPycdt69SfSITaLX1C3A51gwDy5RiUOjVcvqgxzujUwZ3no5oYowtPVYqEla7fchooU8TmY39_DHmVlRR-u7FarrYQ<{%reset%}>)
I1110 09:44:38.138947  260271 eventsink.go:59] 
I1110 09:44:38.138981  260271 eventsink.go:62] eventSink::Debug(<{%reset%}><{%reset%}>)
I1110 09:44:38.139299  260271 eventsink.go:59] {"error":{"code":404,"message":"The specified bucket does not exist.","errors":[{"message":"The specified bucket does not exist.","domain":"global","reason":"notFound"}]}}
I1110 09:44:38.139328  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>{"error":{"code":404,"message":"The specified bucket does not exist.","errors":[{"message":"The specified bucket does not exist.","domain":"global","reason":"notFound"}]}}<{%reset%}>)
I1110 09:44:38.139646  260271 eventsink.go:59] Retry Transport: Returning after 1 attempts
I1110 09:44:38.139696  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Retry Transport: Returning after 1 attempts<{%reset%}>)
I1110 09:44:38.140063  260271 eventsink.go:59] Dismissed an error as retryable. Retry 404s for bucket creation - googleapi: Error 404: The specified bucket does not exist., notFound
I1110 09:44:38.140095  260271 eventsink.go:62] eventSink::Debug(<{%reset%}>Dismissed an error as retryable. Retry 404s for bucket creation - googleapi: Error 404: The specified bucket does not exist., notFound<{%reset%}>)
     pulumi:pulumi:Stack    issue-test-dev  running.       
 ~   └─ gcp:storage:Bucket  my-bucket       refreshing.    

[leezen]

This looks like it's an upstream issue where the read is willing to take as long as the creation timeout: https://github.com/hashicorp/terraform-provider-google-beta/blob/master/google-beta/resource_storage_bucket.go#L650 -- if you do wait long enough, it will complete (I tested this and the default timeout is 20 minutes, unfortunately) and delete the resource from state.

One workaround is to set the create timeout at creation time (this value needs to persist into state to be used during refresh):

const bucket = new gcp.storage.Bucket("helloworld", {}, { 
  customTimeouts: {
    create: "1m"
  }
});

I tested this and confirmed that it does only wait for 1 minute now (instead of 20) during refresh.

[mikhailshilkov]

This has been changed upstream to use configurable timeouts, 4 minutes by default in hashicorp/terraform-provider-google-beta#3938

There is also a comment "There seems to be some eventual consistency errors in some cases, so we want to check a few times to make sure it exists before moving on".

I'll go ahead and close the issue as done/by design.