Enhancement: azurerm_databricks_workspace Private Network

[Lachlan-White]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Enhance the existing Azure Databricks resource to allow configuration of the resource within a Private Virtual Network

New or Affected Resource(s)

• azurerm_databricks_workspace

Potential Terraform Configuration

resource "azurerm_databricks_workspace" "test" {
  name                 = "databricks-test"
  resource_group_name  = "${azurerm_resource_group.test.name}"
  location             = "${azurerm_resource_group.test.location}"
  sku                  = "standard"
  virtual_network_id   = "<networkid>"
  public_subnet_name   = "public"
  public_subnet_range  = "10.0.0.0/28"
  private_subnet_name  = "private"
  private_subnet_range = "10.0.1.0/28"
}

References

• https://github.com/Azure/azure-quickstart-templates/blob/master/101-databricks-workspace-with-vnet-injection/azuredeploy.json
• https://docs.azuredatabricks.net/administration-guide/cloud-configurations/azure/vnet-inject.html

[tombuildsstuff]

@sijusamueltech unfortunately this functionality isn't currently natively supported by Terraform (which is why this issue is still open) - however updates will be posted to this issue when support for this becomes available :)

[arsenyspb]

@tombuildsstuff when planning this issue... can you please add the ability to point to a link of an ARM template, instead of only taking in input related to VNet/Subnet etc. in the variables?.. User story is -- there are couple of other parameters that might be often times specified when provisioning Workspace... some users, due to data governance, prefer to have Locally-Redundant Storage for the Storage Account created inside managed resource group for the root of DBFS... therefore, specifying "storageAccountSkuName": { "value": "Standard_LRS" } as workspace property... And maybe for future cases... Just i.e. be able to take in the same JSON reference that @a138076 mentioned (here)

[tombuildsstuff]

@arsenyspb unfortunately that approach isn't supported by the API (since we map the properties from the SDK/API) - as such we're unable to do so.

Rather than exposing an ARM Template (which could potentially fail during deployment, leaving spurious resources) - it'd be better for the API to expose a new optional field for the ID of an existing Storage Account which could be used rather than a new one being created; but AFAIK this doesn't exist at this time and would be a feature request to the API team.

[arsenyspb]

@tombuildsstuff understood... I might have misread the (5) in Deploy with the REST API -- I thought RP is able concatenate / nest the JSON in to the payload of request.
So, if we focus on the properties, can we then add at least ability to specify the said"storageAccountSkuName": { "value": "Standard_LRS" }` (could be omitted by default)., -- what do you think?..

[arsenyspb]

And oi, no, @tombuildsstuff -- I was not talking about existing Storage Account, -- no, it's the property of a newly-created storage account as part of the new Workspace deployment (the root for DBFS inside Managed Group)

[tombuildsstuff]

@arsenyspb to clarify: Terraform doesn't deploy an ARM Template (ignoring the azurerm_template_deployment resource which does 🙃) - instead we use the Azure SDK for Go (and it's structs) to interact with the API; so these properties would need to be exposed there for us to add support for this :)

[alagala]

@arsenyspb As a temporary workaround (until Terraform and Azure SDK for Go will support VNET injection and LRS storage account), I have setup a Terraform module that calls a Azure Databricks ARM template to enable both LRS and VNET injection.
If interested, check it out at https://github.com/alagala/labs/tree/master/azure/databricks

[schlbra]

Any updates on this issue or the most recent PR? @tombuildsstuff @notchairmk

[tombuildsstuff]

@schlbra whilst this is fixed in the Swagger this is now waiting on the SDK upgrade, which'll be available via #5335 - once that's done it should be possible for someone to take a look into this :)

[michaelmcmillan]

Is this now available to be used?

[schlbra]

@michaelmcmillan , it has been added to version 1.42.0 which has not been released yet.

[ghost]

This has been released in version 1.42.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 1.42.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!