OOM issue in azurerm_storage_blob creation with local file source

[mmattb]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.1.8

AzureRM Provider Version

3.2.0

Affected Resource(s)/Data Source(s)

azurerm_storage_blob

Terraform Configuration Files

resource "azurerm_storage_blob" "img_blob" {
  name                   = "tmp-sb-${var.image_name}"
  storage_account_name   = azurerm_storage_account.img_sa.name
  storage_container_name = azurerm_storage_container.img_container.name
  type                   = "Block"
  source                 = var.src_vhd_path
}

Debug Output/Panic Output

azurerm_storage_account.img_sa: Creating...
azurerm_storage_account.img_sa: Still creating... [10s elapsed]
azurerm_storage_account.img_sa: Still creating... [20s elapsed]
azurerm_storage_account.img_sa: Creation complete after 21s [id=<redacted>]
azurerm_storage_container.img_container: Creating...
azurerm_storage_container.img_container: Creation complete after 0s [id=<redacted>]
azurerm_storage_blob.img_blob: Creating...
╷
│ Error: Plugin did not respond
│ 
│   with azurerm_storage_blob.img_blob,
│   on main.tf line 26, in resource "azurerm_storage_blob" "img_blob":
│   26: resource "azurerm_storage_blob" "img_blob" {
│ 
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ApplyResourceChange call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-azurerm_v3.2.0_x5 plugin:

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x5cf7089?, 0x400400000?})
	runtime/panic.go:992 +0x71
runtime.sysMap(0xc003800000, 0xc00089fe90?, 0xc00089fef8?)
	runtime/mem_linux.go:189 +0x11b
runtime.(*mheap).grow(0xa2496c0, 0x200001?)
	runtime/mheap.go:1404 +0x225
runtime.(*mheap).allocSpan(0xa2496c0, 0x200001, 0x0, 0x1)
	runtime/mheap.go:1170 +0x171
runtime.(*mheap).alloc.func1()
	runtime/mheap.go:912 +0x65
runtime.systemstack()
	runtime/asm_amd64.s:469 +0x49

goroutine 194 [running]:
runtime.systemstack_switch()
	runtime/asm_amd64.s:436 fp=0xc000dcfeb0 sp=0xc000dcfea8 pc=0x461c60
runtime.(*mheap).alloc(0x400002000?, 0x200001?, 0x13?)
	runtime/mheap.go:906 +0x65 fp=0xc000dcfef8 sp=0xc000dcfeb0 pc=0x426745
runtime.(*mcache).allocLarge(0x4cf491?, 0x400000200, 0x1)
	runtime/mcache.go:213 +0x85 fp=0xc000dcff48 sp=0xc000dcfef8 pc=0x416c45
runtime.mallocgc(0x400000200, 0x5284f60, 0x1)
	runtime/malloc.go:1096 +0x5a5 fp=0xc000dcffc0 sp=0xc000dcff48 pc=0x40d205
runtime.makeslice(0xc0018cf298?, 0xc000dd0048?, 0x45e5bf?)
	runtime/slice.go:103 +0x52 fp=0xc000dcffe8 sp=0xc000dcffc0 pc=0x44abb2
github.com/tombuildsstuff/giovanni/storage/2019-12-12/blob/blobs.Client.PutBlockBlobFromFile({{{0x63e0980, 0xc001b91940}, {0x63e31a0, 0xc001a189c0}, 0x0, 0x0, 0x6fc23ac00, 0xd18c2e2800, 0x3, 0x6fc23ac00, ...}, ...}, ...)
	github.com/tombuildsstuff/giovanni@v0.18.0/storage/2019-12-12/blob/blobs/put_block_blob_file.go:18 +0x10a fp=0xc000dd0158 sp=0xc000dcffe8 pc=0x3bae82a
github.com/hashicorp/terraform-provider-azurerm/internal/services/storage.BlobUpload.uploadBlockBlob({0xc001bd2460, {0xc001b55830, 0xd}, {0xc001b557e0, 0xf}, {0xc001b55840, 0xf}, {0xc001b55860, 0x5}, {0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-provider-azurerm/internal/services/storage/blobs.go:161 +0x2e6 fp=0xc000dd03c8 sp=0xc000dd0158 pc=0x4e510e6
github.com/hashicorp/terraform-provider-azurerm/internal/services/storage.BlobUpload.Create({0xc001bd2460, {0xc001b55830, 0xd}, {0xc001b557e0, 0xf}, {0xc001b55840, 0xf}, {0xc001b55860, 0x5}, {0x0, ...}, ...}, ...)
	github.com/hashicorp/terraform-provider-azurerm/internal/services/storage/blobs.go:65 +0x1c8 fp=0xc000dd04b8 sp=0xc000dd03c8 pc=0x4e50068
github.com/hashicorp/terraform-provider-azurerm/internal/services/storage.resourceStorageBlobCreate(0xc001bfe280, {0x53c1360?, 0xc00141ea80})
	github.com/hashicorp/terraform-provider-azurerm/internal/services/storage/storage_blob_resource.go:219 +0x10a8 fp=0xc000dd15f8 sp=0xc000dd04b8 pc=0x4e82a28
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).create(0xc001be4c00?, {0x63efa00?, 0xc001b90a40?}, 0x2?, {0x53c1360?, 0xc00141ea80?})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:329 +0x178 fp=0xc000dd1670 sp=0xc000dd15f8 pc=0xa4a698
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc001458c40, {0x63efa00, 0xc001b90a40}, 0xc001b77a00, 0xc001bfe080, {0x53c1360, 0xc00141ea80})
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/resource.go:467 +0x6f1 fp=0xc000dd1740 sp=0xc000dd1670 pc=0xa4b611
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ApplyResourceChange(0xc000cfadb0, {0x63efa00, 0xc001b90a40}, 0xc001f6def0)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/helper/schema/grpc_provider.go:977 +0xd8a fp=0xc000dd19d0 sp=0xc000dd1740 pc=0xa447aa
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ApplyResourceChange(0xc000e78a00, {0x63efaa8?, 0xc001b892f0?}, 0xc001bcb960)
	github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/tf5server/server.go:603 +0x2ad fp=0xc000dd1a98 sp=0xc000dd19d0 pc=0x920f4d
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ApplyResourceChange_Handler({0x5910e00?, 0xc000e78a00}, {0x63efaa8, 0xc001b892f0}, 0xc001b3b0e0, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.5.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:380 +0x170 fp=0xc000dd1af0 sp=0xc000dd1a98 pc=0x90fe90
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0015ae380, {0x6400ac0, 0xc000a46d80}, 0xc00185eea0, 0xc00122b620, 0xa21f620, 0x0)
	google.golang.org/grpc@v1.39.0/server.go:1292 +0xc6f fp=0xc000dd1e48 sp=0xc000dd1af0 pc=0x87f2ef
google.golang.org/grpc.(*Server).handleStream(0xc0015ae380, {0x6400ac0, 0xc000a46d80}, 0xc00185eea0, 0x0)
	google.golang.org/grpc@v1.39.0/server.go:1617 +0xa1b fp=0xc000dd1f68 sp=0xc000dd1e48 pc=0x88395b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.39.0/server.go:940 +0x98 fp=0xc000dd1fe0 sp=0xc000dd1f68 pc=0x87ce18
runtime.goexit()
	runtime/asm_amd64.s:1571 +0x1 fp=0xc000dd1fe8 sp=0xc000dd1fe0 pc=0x463d41
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.39.0/server.go:938 +0x28a

goroutine 1 [select]:
github.com/hashicorp/go-plugin.Serve(0xc000affe38)
	github.com/hashicorp/go-plugin@v1.4.2/server.go:469 +0x1448
github.com/hashicorp/terraform-plugin-sdk/v2/plugin.Serve(0xc000126c00)
	github.com/hashicorp/terraform-plugin-sdk/v2@v2.10.1/plugin/serve.go:147 +0x8e5
main.main()
	github.com/hashicorp/terraform-provider-azurerm/main.go:30 +0x185

goroutine 5 [select]:
github.com/hashicorp/go-plugin.(*gRPCBrokerServer).Recv(0x0?)
	github.com/hashicorp/go-plugin@v1.4.2/grpc_broker.go:121 +0x67
github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc00009c1e0)
	github.com/hashicorp/go-plugin@v1.4.2/grpc_broker.go:411 +0x44
created by github.com/hashicorp/go-plugin.(*GRPCServer).Init
	github.com/hashicorp/go-plugin@v1.4.2/grpc_server.go:85 +0x4e5

goroutine 6 [IO wait]:
internal/poll.runtime_pollWait(0x7f1491a1eeb0, 0x72)
	runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000092960?, 0xc0015b2000?, 0x1)
	internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000092960, {0xc0015b2000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:167 +0x25a
os.(*File).read(...)
	os/file_posix.go:31
os.(*File).Read(0xc00000e008, {0xc0015b2000?, 0x400?, 0x52b6e20?})
	os/file.go:119 +0x5e
bufio.(*Reader).Read(0xc000070f40, {0xc00016e400, 0x400, 0x0?})
	bufio/bufio.go:236 +0x1b4
github.com/hashicorp/go-plugin.copyChan({0x6412968, 0xc000e78480}, 0x0?, {0x63e3440?, 0xc00000e008?})
	github.com/hashicorp/go-plugin@v1.4.2/grpc_stdio.go:181 +0x1f6
created by github.com/hashicorp/go-plugin.newGRPCStdioServer
	github.com/hashicorp/go-plugin@v1.4.2/grpc_stdio.go:37 +0xf5

goroutine 7 [IO wait]:
internal/poll.runtime_pollWait(0x7f1491a1ecd0, 0x72)
	runtime/netpoll.go:302 +0x89
internal/poll.(*pollDesc).wait(0xc000092a20?, 0xc0015b3000?, 0x1)
	internal/poll/fd_poll_runtime.go:83 +0x32
internal/poll.(*pollDesc).waitRead(...)
	internal/poll/fd_poll_runtime.go:88
internal/poll.(*FD).Read(0xc000092a20, {0xc0015b3000, 0x1000, 0x1000})
	internal/poll/fd_unix.go:167 +0x25a

Error: The terraform-provider-azurerm_v3.2.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Expected Behaviour

No OOM crash

Actual Behaviour

OOM crash

Steps to Reproduce

Just a terraform apply with some variables applied at CLI. The provided file, known as var.src_vhd_path in the script, is 17GB roughly. This is a 16GB RAM Ubuntu VM where the terraform code is running. Hopefully the provider isn't attempting to load the whole file into memory :)

Important Factoids

No response

References

No response

[tombuildsstuff]

hi @mmattb

Thanks for opening this issue.

Taking a look through this appears to be a duplicate of #12925 - rather than having multiple issues open tracking the same thing I'm going to close this issue in favour of that one; would you mind subscribing to #12925 for updates?

Thanks!

[mmattb]

Don't mind at all. Thx Tom.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.