[Compute API] Breaking Behavioural Change - Delta updates to Tags are not detected

[tombuildsstuff]

👋

It appears there's a Breaking Behavioural Change to the Compute API when performing a Delta update where Tags are now incorrectly processed.

When performing a Delta (e.g. PATCH) update on a Virtual Machine Scale Set (and potentially other Resources within this Resource Provider, I've not checked) - when tags differ in casing for only the keys, the API no longer updates the tags as it did previously.

This is in turn the root-cause of this API issue: #8952 - to give an example using API Version 2019-07-01.

When adding tags to an existing Virtual Machine Scale Set:

PATCH /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2019-07-01

{
  ...
	"tags": {
		"Hello": "World"
	}
}

and then retrieving those - we get the correct tag returned:

GET /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2019-07-01

{
  ...
	"tags": {
		"Hello": "World"
	},
  ...
}

---

At this point if we change the casing of the tag:

PATCH /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2019-07-01

{
	"tags": {
		"hello": "World"
	}
}

and then retrieve it:

GET /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2019-07-01

{
  ...
	"tags": {
		"Hello": "World"
	},
  ...
}

Note the incorrect casing for the Tag here.

---

When updating the value for this key:

PATCH /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2019-07-01

{
	"tags": {
		"hello": "World2"
	}
}

the API then detects & correctly performs these changes:

GET /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2019-07-01

{
  ...
	"tags": {
		"hello": "World2"
	},
  ...
}

The tags are also correctly updated when adding a secondary tag:

PATCH /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2019-07-01

{
  ...
	"tags": {
		"Hello": "World2",
		"Upper": "Case"
	}
}

and subsequently retrieving it:

GET /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.Compute/virtualMachineScaleSets/example-vmss?api-version=2019-07-01

{
	"tags": {
		"Hello": "World2",
		"Upper": "Case"
	},
}

---

This appears to be a bug regarding the Delta updates of Tags present in at least the Virtual Machine Scale Set API - but presumably this logic is shared across the Compute Resource Provider and so also is present in these other Resources.

Best we can tell this is a breaking behavioural change in the API - since we've used key-sensitivity for tag updates in the past - as many customers have case-sensitive scripts for tags.

Since this affects version 2019-07-01 of the Compute API - presumably this behaviour is present in every version of the API since too, and potentially in other Resources within this Resource Provider?

Thanks!

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @mjconnection, @Drewm3, @avirishuv.

[Drewm3]

Let me loop in @vladimirjoanovic on this issue. I believe ARM handles the updating of the tags instead of the RPs. So if that is correct, then this needs to be investigated by the ARM team.

[hamidtirmizi]

Most of the time, ARM is responsible for informing the RP about changes to tags (so if ARM doesn’t see a change, the RP is unlikely to see a change), so ARM plays a key role.

Tag values are case sensitive.
Tag names are case insensitive, i.e. lookup does not care about the case. So,

• When updating the case for just the tag name, ARM treats it as a no-op, and you see no effect.
• When updating the tag name case AND the tag value, it is no longer a no-op, ARM replaces the pair in the dictionary.

[tombuildsstuff]

@hamidtirmizi unfortunately that differs from every Azure API I've used, taking Resource Groups as an example - when updating just the casing of the key of a Tag (e.g. hello -> Hello) this is detected and updated.

At this point in time every language I can think of (.net, Java, Objective-C, Swift, Golang, JS, Python..) has a map/dictionary implementation which is case-sensitive for keys (I can't think of one that isn't) - particularly given systems aren't generally written to account for dictionary's being case-insensitive.

As such I believe this wants updating to detect changes to casing on keys (as it detects changes to values)?

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure.

[Drewm3]

Moving this over to the ARM team as the discussion has moved to what is the expected behavior for tags in general for any resource under ARM.

[tombuildsstuff]

@Drewm3 any update here?

[zalyoung]

Any update about this?

[jennyhunter-msft]

The behavior Hamid shared is the current expected behavior.
However, we will track this as a feature request to update our handling of tags so that the tag name is case sensitive. We don't have a timeline yet for when this will be completed, but we are currently undertaking a larger tag modernization project and will investigate it as part of that effort.

[jennyhunter-msft]

We don't seem to have any open feature requests for this on UserVoice. If you would like to create a request there to try and gain traction/priority for this work, please visit our UserVoice.

[admincasper]

This is still not fixed?

[rthorn17]

This is still not fixed?

Hi @admincasper - As mentioned above, by @jennyhunter-msft, this is the current expected behavior. The thing I would like to note this is not an issue with Tag keys only. All properties keys on ARM Resources are case insensitive. When a PATCH call is received by ARM that is only changing the casing on the property key, it is considered a no-op and is not forwarded to the Resource Provider. At this time, there are no plans to change this behavior but suggest you add this to the Azure Feedback site for tracking. feedback.azure.com