Proposal - simplifying resource referencing (part 2)

[anthony-c-martin]

Proposal - simplifying resource referencing (part 2)

Part 1 / Part 2

Problem statement

Passing around / obtaining references to resources in a type-safe manner is overly complex. Rather than inventing non-type-safe mechanisms to refer to resources or resource properties, we should provide a first-class syntax for doing so, with full type-safety and editor support.

Resources as params and outputs

A new type of resource will be accepted in param and output declarations to permit passing a reference to a resource as an input or output for a module. Supplying the type string for the resource would be optional, but functionality would be greatly reduced without it.

At compile-time, Bicep will type check for reference equality - it will ensure a valid resource reference is passed to a generic resource param, and it will ensure that a valid resource reference matching the expected type string is passed to a typed resource param.

Examples

Generic

// we haven't specified a resource type here
param lockableResource resource

resource lockResource 'Microsoft.Authorization/locks@2016-09-01' = {
  scope: lockableResource
  name: 'DontDelete'
  ...
}

Input/Output

// input a resource reference
param storageAcc resource 'Microsoft.Storage/storageAccounts@2021-01-01'

var myContainer = storageAcc.child('blobServices', 'default').child('containers', 'myContainer')

// output a resource reference - note the resource type can be omitted
output myContainer resource = myContainer

Property access

param storageAcc resource 'Microsoft.Storage/storageAccounts@2021-01-01'

// list keys
var myKey = listKeys(storageAcc.id, storageAcc.apiVersion).keys[0].value

// property access
output accountTags object = storageAcc.tags

Notes

1. If the param does not specify a resource type string, functionality will be greatly reduced - limited to using the resource as a scope for an extension resource, and accessing the resource id property. We want to encourage module authors to be specific about the type they accept to provide optimal type safety.
2. API versions do not need to match across module params and outputs, but types must match if the param has specified a type.
3. We will need to be careful when passing param references to resources at a different scope to the module, as they cannot be used for certain purposes (deploying children/extensions, for example).

Out of scope

1. This proposal requires both inputs and outputs to accept a resource reference, and there is no conversion between resourceId string and resource reference. The following would not be permitted:

   module myMod './module.bicep' = {
     name: 'myMod'
     params: {
       // resourceReference is a param of type 'resource'
       // resourceId is a string containing a resourceId
       resourceReference: resourceId
     }
   } 

Codegen

The most straightforward option for JSON codegen is to generate a string parameter or output in the template JSON, with some associated metadata.

[alex-frankel]

Will we accept a literal resource ID string for foo.params.res here?

foo.bicep

param res resource

main.bicep

module foo './foo.bicep' = {
  name: ...
  params: {
    res: '/subscriptions/.../resourceGroups/.../microsoft.rp/type/...'
  }
}

[majastrz]

My expectation would be for the string to be rejected because it's not a symbolic name. However, I think we do need some sort of interop gesture to bridge templates passing resource ID strings around with this way of passing parameters.

[majastrz]

@anthony-c-martin During our last discussion, we realized that API version match/mismatch semantics differ on inputs and outputs. I think it'd be worthwhile to explain more about for the community at large to offer feedback (if any).

In the generic resource case (note number 1), would we allow name and type property access as well? Just wondering... we can start small and add more later, of course.

Regarding code gen, we should consider moving some of the type safety down to the runtime so all tempaltes can leverage it. We could still compile down to a fully qualified resource ID but we could introduce a new parameter type in the JSON with some additional settings.

[alex-frankel]

My expectation would be for the string to be rejected because it's not a symbolic name.

Right, but then this would have to work I guess? So if you need to pass in a resource ID from an external source, you would expose that as a param of type resource.

params.json

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "foo": {
      "value": "/subscriptions/.../resourceGroups/.../microsoft.rp/type/..."
    }
  }
}

main.bicep

param foo resource

module foo './foo.bicep' = {
  name: ...
  params: {
    res: foo
  }
}

az deployment group create -f ./main.bicep -p params.json

[miqm]

My expectation would be for the string to be rejected because it's not a symbolic name.

Right, but then this would have to work I guess? So if you need to pass in a resource ID from an external source, you would expose that as a param of type resource.

params.json

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "foo": {
      "value": "/subscriptions/.../resourceGroups/.../microsoft.rp/type/..."
    }
  }
}

main.bicep

param foo resource

module foo './foo.bicep' = {
  name: ...
  params: {
    res: foo
  }
}

az deployment group create -f ./main.bicep -p params.json

in some other issue I've suggested using as keyword that could be used to cast string that we expect to be a resourceid to a biceps resource. perhaps worth revisiting that?

[majastrz]

@alex-frankel Yeah passing it through parameters would have to work. We do have to answer the question whether we need the ability to turn a string ID into a symbolic name within a Bicep file without the use of parameters. One scenario I can think of comes up with referencing a JSON file as a module (when we have the feature) when the template outputs a string resource ID.

@miqm Yeah as could be an option although we have so far resisted adding type casting so far. All the type conversions currently are done through converter functions like any() or string(). Another option would be a separate function or a resource() overload that accepts a string ID

[anthony-c-martin]

My expectation would be for the string to be rejected because it's not a symbolic name.

Right, but then this would have to work I guess? So if you need to pass in a resource ID from an external source, you would expose that as a param of type resource.

This is definitely a scenario we'll need to handle, and I think we have the options of either (with some rough examples):

1. Exposing as a string parameter in the JSON - simple, but potentially error-prone if people format the id incorrectly.

   "parameters": {
     "vmResource": {
       "type": "string"
     }
   }

2. Creating a new parameter type in JSON with enhanced validation (on server side, but possibly also client-side with psh/cli)

   "parameters": {
     "vmResource": {
       "type": "resource"
     }
   }

3. (sort of an in-between) Exposing as a string parameter in the JSON, with metadata that newer CLI utilities are able to understand (so as to not make it a breaking change, and allow better validation to be added gracefully).

   "parameters": {
     "vmResource": {
       "type": "string",
       "metadata": {
         "_typeinfo": {
           "resourceType": "Microsoft.Compute/virtualMachines"
         }
       }
     }
   }

[anthony-c-martin]

@alex-frankel Yeah passing it through parameters would have to work. We do have to answer the question whether we need the ability to turn a string ID into a symbolic name within a Bicep file without the use of parameters. One scenario I can think of comes up with referencing a JSON file as a module (when we have the feature) when the template outputs a string resource ID.

It makes sense to think about this, but for the purposes of this specific proposal, I'd like to treat turning a resourceId in a string into a symbolic reference out-of-scope. Would you be OK with me creating another issue specifically for that, and adding a note to this proposal to mention as such?

[majastrz]

Should be ok to include it in part 3.

[miqm]

I like Passing type in metadata rather than introducing new type. In addition we could do object or array typing in similar way.

[anthony-c-martin]

Will we accept a literal resource ID string for foo.params.res here?

@alex-frankel, @majastrz - FYI, I added a note to explicitly mention this scenario is not covered by this proposal.

[jamesongithub]

Will this proposal cover passing a resource directly as a parameter of another resource or just for module input/outputs

something like

resource disk 'disks' {
   ...
}

resource vm 'vms' {
  osDisk: disk
}

[anthony-c-martin]

Will this proposal cover passing a resource directly as a parameter of another resource or just for module input/outputs

This proposal is just covering module inputs/outputs.

[stan-sz]

As in #2163 the output of a module can contain secrets (e.g. output of a list*) function to store it's value in a KV. Is it planed to introduce secureString or secureObject (or other way to hide secrets) from module output?

[Devvox93]

We've also run into limitations enforced by the deployment engine using resources as module outputs. That will require changes in Azure to unblock.

Is there any update or roadmap on when passing resources as module outputs would be supported by Azure?
Or is this something that will remain unsupported?

Very eager to use this as, like others have mentioned, it would greatly simplify authoring modules :)

[stan-sz]

How about (re)using the existing syntax for params:

param resource storageAcc 'Microsoft.Storage/storageAccounts@2021-01-01' existing

or

param storageAcc resource 'Microsoft.Storage/storageAccounts@2021-01-01' existing

In the first snippet, the resource and storageAcc are flipped to preserve the resource syntax. The second snippet follows the param <symbol name> <type>.
Semantically this would behave the same as resource reference, because a resource passed into a module needs to exist, so param would just be an indicator that it's a parameter.

[dazinator]

Just tried the experimental feature.
It wasn't useful for my case, as it doesn't support declaring resource paramaters of any type, and setting the scope property for a resource e.g this won't work

param scope resource // can't pass an untyped resurce which is what `scope` needs below.

var roleAssignmentName = 'some-name'
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: roleAssignmentName
  scope: resource // can't set this
  properties: {
    roleDefinitionId: roleDefinitionResourceId
    principalId: userAssignedIdentity.properties.principalId
    principalType: 'ServicePrincipal'
    description: empty(roleAssignmentDescription) ? null : ''
  }
}

as such I am unable to modularise this code, however role assignments are quite common accross our deployments, and there is value in ensuring things like the unique name of the assignment uses a guid based on principal id, scope id, and role definition id etc.

[jangelfdez]

I just hit the same situation that @dazinator mentions in his reply. Trying to modularize roleassignments into an independent module to be reused with different type of assignment levels.

Being able to pass a generic resource would be needed in this case to cover all the different scenarios.

[asdkant-bf]

After reading all the comments here, there's something I'm still not clear on:

Assuming I have a module that creates one or more resources, will I be able to pass those resources as outputs?

[alex-frankel]

After reading all the comments here, there's something I'm still not clear on:

Assuming I have a module that creates one or more resources, will I be able to pass those resources as outputs?

Yes, you will be able to write

resource foo '...' = { ... }
output myResource resource = foo

[housten]

I am also really wanting to be able to have a generic role assignment module. I want to be able to pass multiple roles so I was trying to put the creation of the roles in a module and while breaking it out it makes sense to let it fulfill multiple purposes so I can lower the published module maintenance overhead.

For my part, this need could be met by being able to set the scope of a module to the resource it should attach to. I am not so interested in being able to receive these as outputs.

@description('Name of the queue')
param QueueName string

@description('A list of rbac settings to create')
param Rbac array

resource queue 'Microsoft.ServiceBus/namespaces/queues@2021-06-01-preview' = {
  name: QueueName
  properties: {
    deadLetteringOnMessageExpiration: false
    enableBatchedOperations: true
    enablePartitioning: false
  }
}

module roleAssignment 'br:mybicepregistry.azurecr.io/role:1.0' = [for rbac in Rbac: {
  scope: queue
  name: rbac.principleId
  params: {
    roles: rbac.roles
    principalId: rbac.principalId
    principalType: rbac.principalType
    targetRoleId: queue.id  // for naming
  }
} ]

This was the first thing I tried when I broke the role out to a module but got the error that scope for a module has to be a resource group or a subscription. Syntactically I like this because it matches the syntax for creating the resource at this level (ie without a module).

The ARM json work around #5805 was a nice idea, but then each repo would have to have its own json as I am using a bicep registry for all the modules, which I assume can't hold a json template file. It presents a nightmare maintenance-wise having lots of json templates to have to fix and a technical understanding of the file that I am trying encapsulate away for the users (or blind acceptance that a file has to be in their repo and shouldn't be fiddled with).

[BartDecker]

@housten scanning your post quickly I think you might want to look at the examples given in: #7621

[housten]

@housten scanning your post quickly I think you might want to look at the examples given in: #7621

Thanks @BartDecker. In my last paragraph I mention the ARM template work around and why I didn't think it was a good fit. To avoid having to have arm templates in every repository, I would rather have specific modules for each type of role target. Unless there is a way to put arm templates into the bicep registry?

[cloudlene]

I was trying to build a generic role assignment module.

I don't understand why we need provider & the API Version when the resource passed must be a symbolic name in parent modules.

1. If we are declaring it without provider & version, restrict the property access to top level properties (such as name, id and not 'properties'). This way we can pass it around to child modules or assign them to parent & scope properties.
2. If resource declaration must include the provider & apiVersion, at the least we should have a overload of resource function that would accept a resource Id.
3. Another option could be to make the reference function return a 'resource' type that makes it assignable to 'scope' property.

Option #1: role-assignments.bicep

param principalId string
param principalType string
param roleNames string[]
param targetResourceId string

// Runs a shell script that creates a query from the roleNames passed: az role definition list --query "roleName == 'Key Vault Reader' || roleName == 'Key Vault Secrets Officer'"
module lookupRoles 'lookup-roles.bicep' = {
  name: 'lookupRoles'
  params: {
    location: location
    roleNames: roleNames
    nameSuffixShort: nameSuffixShort
  }
}

var roleIds = lookupRoles.outputs.roleIds

resource roleDefinitions 'Microsoft.Authorization/roleDefinitions@2022-05-01-preview' existing = [for roleId in roleIds : {
  scope: resource(targetResourceId) // Or Make reference(targetResourceId) return a resource type instead of object
  name: roleId
}]

resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = [for i in range(0, length(roleIds)):{
  name: guid(string(i), roleDefinitions[i].id)
  properties: {
    principalId: principalId
    roleDefinitionId: roleDefinitions[i].id
    principalType: principalType
  }
}]

// Option 2: role-assignments.bicep

param principalId string
param principalType string
param roleNames string[]
// Notice no need for type or version
param targetResource resource

// Runs a shell script that creates a query from the roleNames passed: az role definition list --query "roleName == 'Key Vault Reader' || roleName == 'Key Vault Secrets Officer'"
module lookupRoles 'lookup-roles.bicep' = {
  name: 'lookupRoles'
  params: {
    location: location
    roleNames: roleNames
    nameSuffixShort: nameSuffixShort
  }
}

var roleIds = lookupRoles.outputs.roleIds

resource roleDefinitions 'Microsoft.Authorization/roleDefinitions@2022-05-01-preview' existing = [for roleId in roleIds : {
  scope: targetResource 
  name: roleId
}]

resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = [for i in range(0, length(roleIds)):{
  name: guid(string(i), roleDefinitions[i].id)
  properties: {
    principalId: principalId
    roleDefinitionId: roleDefinitions[i].id
    principalType: principalType
  }
}]

When it's already allowed that any resource can be passed to the scope property of the roleDefinitions, it's only logical to expect a parameter that can accept ANY resource not just ones that are statically typed to a specific provider & version.