AKS: bug in the Node Pools API

[tombuildsstuff]

👋🏻

It appears there's a bug in the Node Pools API in versions 2019-11-01 -> 2020-02-01 (inclusive) - this may be applicable to newer versions of the API too, I'm unable to check at this time.

Previously we were using API version 2019-10-01 for AKS - and using the Node Pools API we were able to set Tags for a Node Pool in upper-case. Upon updating to API version 2019-11-01 (and subsequently 2020-02-01) - whilst we submit Title Case for the key of the Tags for the Node Pool - the API returns them in lower-case.

When creating/updating a Node Pool for an AKS Cluster - with the following HTTP Request:

PUT https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/example-resources/providers/Microsoft.ContainerService/managedClusters/tom-devaks999/agentPools/internal?api-version=2020-02-01

{
	"properties": {
		"count": 1,
		"enableAutoScaling": true,
		"enableNodePublicIP": false,
		"maxCount": 2,
		"maxPods": 110,
		"minCount": 1,
		"orchestratorVersion": "1.15.10",
		"osDiskSizeGB": 100,
		"osType": "Linux",
		"tags": {
			"Environment": "Staging"
		},
		"type": "VirtualMachineScaleSets",
		"vmSize": "Standard_DS2_v2"
	}
}

returns the following HTTP Response:

{
	"id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/example-resources/providers/Microsoft.ContainerService/managedClusters/tom-devaks999/agentPools/internal",
	"name": "internal",
	"type": "Microsoft.ContainerService/managedClusters/agentPools",
	"properties": {
		"count": 1,
		"vmSize": "Standard_DS2_v2",
		"osDiskSizeGB": 100,
		"maxPods": 110,
		"type": "VirtualMachineScaleSets",
		"maxCount": 2,
		"minCount": 1,
		"enableAutoScaling": true,
		"provisioningState": "Upgrading",
		"orchestratorVersion": "1.15.10",
		"enableNodePublicIP": false,
		"tags": {
			"environment": "Staging"
		},
		"osType": "Linux"
	}
}

---

When the Node Pool's been updated - retrieving the Node Pool using the GET endpoint returns:

{
	"id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/example-resources/providers/Microsoft.ContainerService/managedClusters/tom-devaks999/agentPools/internal",
	"name": "internal",
	"type": "Microsoft.ContainerService/managedClusters/agentPools",
	"properties": {
		"count": 1,
		"vmSize": "Standard_DS2_v2",
		"osDiskSizeGB": 100,
		"maxPods": 110,
		"type": "VirtualMachineScaleSets",
		"maxCount": 2,
		"minCount": 1,
		"enableAutoScaling": true,
		"provisioningState": "Succeeded",
		"orchestratorVersion": "1.15.10",
		"enableNodePublicIP": false,
		"tags": {
			"environment": "Staging"
		},
		"osType": "Linux"
	}
}

Note the value for the tags block in both resources - upon submitting the key is Environment but is returned as environment. This occurs for both new Node Pools and when updating existing ones.

Since this is a bug in the AKS Node Pools API (and not something we can work around) - can this be fixed both in the existing API's and the newer versions?

Thanks!

cc @jluk

[jluk]

@wenwu449 / @gtxistxgao for discussion

[gtxistxgao]

Thanks @tombuildsstuff for calling it out.

• Set Tags on AgentPool though agent pool API is only available for API version >= 2019-11-01.
• It does lower case the key. One thing worth to mention is that the key part of Tag is case insensitive. If add both "environment": "Staging" and "Environment": "Staging" to a VMSS, only one of them will exist. The current behavior to return lower case key is more deterministic.

For example, if the input is the following tags, user may be confused that which one will AKS pick. With the current design, we are certain that it will be the first one.

"tags": {
	"environment": "Staging",
	"Environment": "Staging"
},

[tombuildsstuff]

@gtxistxgao just to check, are you confirming this is a bug, or saying this behaviour is intentional?

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/aks-pm.

[gtxistxgao]

@tombuildsstuff the lowering case behavior does seem to be intentional at this point. The reason behinds it is that tags on VMSS is case-insensitive.

[tombuildsstuff]

@gtxistxgao thanks - whilst this issue is present in the AKS API it appears the root cause is the Compute API - I've opened #8982 to track that.

Once that issue's been resolved I'll update here to confirm if that's been fixed - but it's probably worth leaving this issue open until the bug in the Compute/AKS API's have been resolved, since it's a known issue?

Thanks!

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @armleads-azure.