Add support of Azure standard load balancer and public IP #61884

feiskyer · 2018-03-29T09:15:57Z

What this PR does / why we need it:

This PR adds a new configure loadBalancerSku and excludeMasterFromStandardLB to support Azure standard load balancer and public IP.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
First step of #60485

Special notes for your reviewer:

This is the first step of #60485, and only VMAS is supported yet. VMSS support will be added in a following PR.

Release note:

Azure cloud provider now supports standard SKU load balancer and public IP. To use it, set cloud provider config with
{
  "loadBalancerSku": "standard",
  "excludeMasterFromStandardLB": true,
}

If excludeMasterFromStandardLB is not set, it will be default to true, which means master nodes are excluded to the backend of standard LB.

Also note standard load balancer doesn't work with annotation `service.beta.kubernetes.io/azure-load-balancer-mode`. This is because all nodes (except master) are added as the LB backends.

feiskyer · 2018-03-29T09:16:37Z

/sig azure

feiskyer · 2018-03-30T01:41:46Z

/retest

karataliu · 2018-03-30T03:13:36Z

pkg/cloudprovider/providers/azure/azure_standard.go

+			matches := backendPoolIDRE.FindStringSubmatch(backendPool)
+			if len(matches) == 2 {
+				lbName := matches[1]
+				if strings.HasSuffix(lbName, InternalLoadBalancerNameSuffix) == isInternal {


It is possible to pull master nodes into the pool, if it only has public/internal LB.

Added a check before calling ensureHostInPool

feiskyer · 2018-03-30T03:40:46Z

@karataliu As discussed offline, added two configs now. PTAL

{
  "loadBalancerSku": "standard",
  "excludeMasterFromStandardLB": true,
}

If excludeMasterFromStandardLB is not set, master nodes are also added to the backend of standard LB. This is useful for small clusters, e.g. 3 nodes with HA, in which case master nodes are also running workload Pods.

karataliu · 2018-03-30T05:23:15Z

pkg/cloudprovider/providers/azure/azure.go

+	LoadBalancerSku string `json:"loadBalancerSku" yaml:"loadBalancerSku"`
+	// ExcludeMasterFromStandardLB excludes master nodes from standard load balancer.
+	ExcludeMasterFromStandardLB bool `json:"excludeMasterFromStandardLB" yaml:"excludeMasterFromStandardLB"`
+


Should assign default values to those 2 somewhere. ExcludeMasterFromStandardLB should default to true?

ExcludeMasterFromStandardLB is bool, so it is default to false. This works for all deployments

Consider using pointers thus it'll be easy to change default value

ref: Azure/acs-engine#1584 (review)

karataliu · 2018-03-30T05:30:33Z

pkg/cloudprovider/providers/azure/azure_wrap.go

@@ -242,3 +243,11 @@ func (az *Cloud) newRouteTableCache() (*timedCache, error) {

 	return newTimedcache(rtCacheTTL, getter)
 }
+
+func (az *Cloud) useStandardLoadBalancer() bool {
+	if strings.ToLower(az.LoadBalancerSku) == loadBalancerSkuStandard {


can return strings.EqualFold(az.LoadBalancerSku,loadBalancerSkuStandard) directly

karataliu · 2018-03-30T05:40:49Z

pkg/cloudprovider/providers/azure/azure_standard.go

@@ -123,7 +130,7 @@ func (az *Cloud) mapLoadBalancerNameToVMSet(lbName string, clusterName string) (
 // This would be the name for Azure LoadBalancer resource.
 func (az *Cloud) getLoadBalancerName(clusterName string, vmSetName string, isInternal bool) string {
 	lbNamePrefix := vmSetName
-	if strings.EqualFold(vmSetName, az.vmSet.GetPrimaryVMSetName()) {
+	if strings.EqualFold(vmSetName, az.vmSet.GetPrimaryVMSetName()) || az.useStandardLoadBalancer() {


Anywhere to add hint that selectLoadBalancer does not work for StandardLoadBalancer?

Good catch, will add this check in getLoadBalancer()

karataliu · 2018-03-30T05:41:34Z

pkg/cloudprovider/providers/azure/azure_standard_test.go

+		},
+		{
+			description:   "non-default standard external LB should map to empty string",
+			lbName:        "azuretest-internal",


external LB ?

karataliu · 2018-03-30T05:42:53Z

pkg/cloudprovider/providers/azure/azure_standard_test.go

 	}

 	for _, c := range cases {
+		if c.useStandardLB {
+			az.Config.LoadBalancerSku = loadBalancerSkuStandard
+		}


should set LoadBalancerSku back for !useStandardLB. This works now since the Standard SKU cases are in the end.

karataliu · 2018-03-30T05:43:56Z

pkg/cloudprovider/providers/azure/azure_standard_test.go

+	for _, c := range cases {
+		if c.useStandardLB {
+			az.Config.LoadBalancerSku = loadBalancerSkuStandard
+		}


Also, switch back for !useStandardLB

karataliu · 2018-03-30T05:44:26Z

pkg/cloudprovider/providers/azure/azure_standard_test.go

+		if c.useStandardLB {
+			az.Config.LoadBalancerSku = loadBalancerSkuStandard
+		}
+		vmset := az.getLoadBalancerName(c.clusterName, c.vmSet, c.isInternal)


better 'loadbalancerName := '

feiskyer · 2018-03-30T06:41:48Z

@karataliu addressed comments. PTAL

feiskyer · 2018-03-30T08:06:09Z

/retest

karataliu · 2018-03-30T08:26:54Z

pkg/cloudprovider/providers/azure/azure_loadbalancer.go

@@ -273,6 +284,11 @@ func (az *Cloud) selectLoadBalancer(clusterName string, service *v1.Service, exi
 				Location:                     &az.Location,
 				LoadBalancerPropertiesFormat: &network.LoadBalancerPropertiesFormat{},
 			}
+			if az.useStandardLoadBalancer() {


This will not be true now due to ' wantLb && !az.useStandardLoadBalancer()'

karataliu · 2018-03-30T08:29:27Z

pkg/cloudprovider/providers/azure/azure_standard.go

@@ -623,6 +630,22 @@ func (as *availabilitySet) ensureHostInPool(serviceName string, nodeName types.N
 		}
 	}
 	if !foundPool {
+		if as.useStandardLoadBalancer() && len(newBackendPools) > 0 {
+			// Although standard load balancer supports backends from multiple availability
+			// sets, the same interface couldn't reference more than one load balancer of


better to use full name 'interface' -> 'network interface', i got a bit confused when looking at the comment.

so are you referring go interface? that make sense

karataliu · 2018-03-30T08:57:38Z

/lgtm

k8s-ci-robot · 2018-03-30T08:57:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, karataliu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/cloudprovider/providers/azure/OWNERS~~ [feiskyer,karataliu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-github-robot · 2018-03-30T10:12:01Z

/test all [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2018-03-30T11:05:24Z

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions here.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 29, 2018

feiskyer requested review from karataliu and brendandburns and removed request for karataliu March 29, 2018 09:16

k8s-ci-robot added the sig/azure label Mar 29, 2018

k8s-ci-robot requested review from andyzhangx and khenidak March 29, 2018 09:16

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 29, 2018

feiskyer mentioned this pull request Mar 30, 2018

Added support for Standard LoadBlanacer Sku #61788

Closed

karataliu reviewed Mar 30, 2018

View reviewed changes

feiskyer force-pushed the standard branch from 934537e to 892d608 Compare March 30, 2018 03:34

karataliu reviewed Mar 30, 2018

View reviewed changes

feiskyer force-pushed the standard branch from 892d608 to d4e95a8 Compare March 30, 2018 06:41

karataliu reviewed Mar 30, 2018

View reviewed changes

feiskyer added 2 commits March 30, 2018 16:52

Add support of Azure standard load balancer and public IP

094dbde

some updates

7ec0315

feiskyer force-pushed the standard branch from d4e95a8 to 7ec0315 Compare March 30, 2018 08:52

k8s-ci-robot assigned karataliu Mar 30, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 30, 2018

k8s-github-robot merged commit 3cf80b1 into kubernetes:master Mar 30, 2018

feiskyer deleted the standard branch March 30, 2018 13:58

feiskyer mentioned this pull request Jul 12, 2018

Allow users to configure standard load balancer for k8s Azure/acs-engine#3468

Closed

tgarlot mentioned this pull request Sep 11, 2018

GA support of Standard Load Balancer support for user deployed services (was: Be able to configure Load Balancer SKU) Azure/AKS#643

Closed

dtrefilov-piterbyte mentioned this pull request Jul 2, 2019

Batch changed the SKU of load balancers provisioned in the user subscription. How to switch back? Azure/Batch#66

Closed

rfranzke mentioned this pull request Jan 30, 2020

Should we switch to standard SKU load balancers in Azure? gardener/gardener-extension-provider-azure#15

Closed

mainred mentioned this pull request Dec 10, 2020

chore: support Windows CSE and Customdata Azure/AgentBaker#396

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support of Azure standard load balancer and public IP #61884

Add support of Azure standard load balancer and public IP #61884

feiskyer commented Mar 29, 2018 •

edited

Loading

feiskyer commented Mar 29, 2018

feiskyer commented Mar 30, 2018

karataliu Mar 30, 2018

feiskyer Mar 30, 2018

feiskyer commented Mar 30, 2018

karataliu Mar 30, 2018

feiskyer Mar 30, 2018

karataliu Mar 30, 2018

karataliu Mar 30, 2018

karataliu Mar 30, 2018

feiskyer Mar 30, 2018

karataliu Mar 30, 2018

karataliu Mar 30, 2018

karataliu Mar 30, 2018

karataliu Mar 30, 2018

feiskyer commented Mar 30, 2018

feiskyer commented Mar 30, 2018

karataliu Mar 30, 2018

feiskyer Mar 30, 2018

karataliu Mar 30, 2018

feiskyer Mar 30, 2018

karataliu commented Mar 30, 2018

k8s-ci-robot commented Mar 30, 2018

k8s-github-robot commented Mar 30, 2018

k8s-github-robot commented Mar 30, 2018

Add support of Azure standard load balancer and public IP #61884

Add support of Azure standard load balancer and public IP #61884

Conversation

feiskyer commented Mar 29, 2018 • edited Loading

feiskyer commented Mar 29, 2018

feiskyer commented Mar 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feiskyer commented Mar 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feiskyer commented Mar 30, 2018

feiskyer commented Mar 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karataliu commented Mar 30, 2018

k8s-ci-robot commented Mar 30, 2018

k8s-github-robot commented Mar 30, 2018

k8s-github-robot commented Mar 30, 2018

feiskyer commented Mar 29, 2018 •

edited

Loading