Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for GKE dual-stack clusters #7459

Merged
merged 6 commits into from
Mar 23, 2023
Merged

Support for GKE dual-stack clusters #7459

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3ba8693
Support for GKE dual-stack clusters and bump Go api version.
600lyy Mar 15, 2023
e152a27
Merge branch 'main' into fix-container-cluster-13842
600lyy Mar 17, 2023
9ea6ce4
Merge branch 'main' into fix-container-cluster-13842
600lyy Mar 18, 2023
90f7ba7
forceNew on stack_type.
600lyy Mar 21, 2023
2eedf22
Remove the support for stack type update.
600lyy Mar 23, 2023
6919c06
Revert go.sum.
600lyy Mar 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions mmv1/third_party/terraform/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKV
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go v0.107.0 h1:qkj22L7bgkl6vIeZDlOY2po43Mx/TIa2Wsa7VR+PEww=
cloud.google.com/go v0.107.0/go.mod h1:wpc2eNrD7hXUTy8EKS10jkxpZBjASrORK7goS+3YX2I=
cloud.google.com/go v0.105.0 h1:DNtEKRBAAzeS4KyIory52wWHuClNaXJ5x1F7xa4q+5Y=
cloud.google.com/go v0.105.0/go.mod h1:PrLgOJNe5nfE9UMxKxgXj4mD3voiP+YQ6gdt6KMFOKM=
cloud.google.com/go v0.110.0 h1:Zc8gqp3+a9/Eyph2KDmcGaPtbKRIoqq4YTlL4NMD0Ys=
cloud.google.com/go v0.110.0/go.mod h1:SJnCLqQ0FCFGSZMUNUf84MV3Aia54kn7pi8st7tMzaY=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
Expand Down Expand Up @@ -59,6 +59,8 @@ cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/GoogleCloudPlatform/declarative-resource-client-library v1.31.0 h1:4Orb9zAllU52CUtZfKu9OWYyDK6neeBt2ye6NKqAfOg=
github.com/GoogleCloudPlatform/declarative-resource-client-library v1.31.0/go.mod h1:pL2Qt5HT+x6xrTd806oMiM3awW6kNIXB/iiuClz6m6k=
github.com/GoogleCloudPlatform/declarative-resource-client-library v1.34.0 h1:o7t+hPFv+Ax5O2vxzIH7dEtvlWA7JJOlOd7mWFvMa6s=
github.com/GoogleCloudPlatform/declarative-resource-client-library v1.34.0/go.mod h1:pL2Qt5HT+x6xrTd806oMiM3awW6kNIXB/iiuClz6m6k=
github.com/Microsoft/go-winio v0.4.14/go.mod h1:qXqCSQ3Xa7+6tgxaGTIe4Kpcdsi+P8jBhyzoq1bpyYA=
Expand Down Expand Up @@ -88,6 +90,7 @@ github.com/census-instrumentation/opencensus-proto v0.3.0/go.mod h1:f6KPmirojxKA
github.com/census-instrumentation/opencensus-proto v0.4.1 h1:iKLQ0xPNFxR/2hzXZMrBo8f1j86j5WHzznCCQxV/b8g=
github.com/census-instrumentation/opencensus-proto v0.4.1/go.mod h1:4T9NM4+4Vw91VeyqjLS6ao50K5bOcLKN6Q42XnYaRYw=
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
Expand All @@ -107,7 +110,6 @@ github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWH
github.com/cncf/xds/go v0.0.0-20220314180256-7f1daf1720fc/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cncf/xds/go v0.0.0-20230105202645-06c439db220b h1:ACGZRIr7HsgBKHsueQ1yM4WaVaXh21ynwqsF8M8tXhA=
github.com/cncf/xds/go v0.0.0-20230105202645-06c439db220b/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/creachadair/staticfile v0.1.2 h1:QG0u27/Ietu0UVOk1aMbF6jrWrEzPIdZP4ju3c1PPfY=
github.com/creachadair/staticfile v0.1.2/go.mod h1:a3qySzCIXEprDGxk6tSxSI+dBBdLzqeBOMhZ+o2d3pM=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
Expand All @@ -127,6 +129,7 @@ github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1 h1:
github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1/go.mod h1:KJwIaB5Mv44NWtYuAOFCVOjcI94vtpEz2JU/D2v6IjE=
github.com/envoyproxy/go-control-plane v0.10.3 h1:xdCVXxEe0Y3FQith+0cj2irwZudqGYvecuLB1HtdexY=
github.com/envoyproxy/go-control-plane v0.10.3/go.mod h1:fJJn/j26vwOu972OllsvAgJJM//w9BV6Fxbg2LuVd34=
github.com/envoyproxy/protoc-gen-validate v0.1.0 h1:EQciDnbrYxy13PgWoY8AqoxGiPrpgBZ1R8UNe3ddc+A=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/envoyproxy/protoc-gen-validate v0.6.7/go.mod h1:dyJXwwfPK2VSqiB9Klm1J6romD608Ba7Hij42vrOBCo=
github.com/envoyproxy/protoc-gen-validate v0.9.1 h1:PS7VIOgmSVhWUEeZwTe7z7zouA22Cr590PzXKbZHOVY=
Expand Down Expand Up @@ -566,6 +569,7 @@ golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211 h1:JGgROgKl9N8DuW20oFS5gxc+lE67/N3FcwmBPMe7ArY=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
Expand Down
58 changes: 55 additions & 3 deletions mmv1/third_party/terraform/resources/resource_container_cluster.go.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1458,6 +1458,14 @@ func ResourceContainerCluster() *schema.Resource {
ConflictsWith: ipAllocationCidrBlockFields,
Description: `The name of the existing secondary range in the cluster's subnetwork to use for service ClusterIPs. Alternatively, services_ipv4_cidr_block can be used to automatically create a GKE-managed one.`,
},

"stack_type": {
Type: schema.TypeString,
Optional: true,
Default: "IPV4",
ValidateFunc: validation.StringInSlice([]string{"IPV4", "IPV4_IPV6"}, false),
Description: `The IP Stack type of the cluster. Choose between IPV4 and IPV4_IPV6. Default type is IPV4 Only if not set`,
},
},
},
},
Expand Down Expand Up @@ -1970,7 +1978,9 @@ func resourceContainerClusterCreate(d *schema.ResourceData, meta interface{}) er

clusterName := d.Get("name").(string)

ipAllocationBlock, err := expandIPAllocationPolicy(d.Get("ip_allocation_policy"), d.Get("networking_mode").(string))
ipAllocationBlock, err := expandIPAllocationPolicy(
d.Get("ip_allocation_policy"), d.Get("networking_mode").(string), d.Get("datapath_provider").(string),
d.Get("private_ipv6_google_access").(string), d.Get("enable_l4_ilb_subsetting").(bool))
if err != nil {
return err
}
Expand Down Expand Up @@ -2858,6 +2868,27 @@ func resourceContainerClusterUpdate(d *schema.ResourceData, meta interface{}) er
log.Printf("[INFO] GKE cluster %s Private IPv6 Google Access has been updated", d.Id())
}

if d.HasChange("ip_allocation_policy") {
// Stack type change is not supported in GKE at the moment
600lyy marked this conversation as resolved.
Show resolved Hide resolved
policy := d.Get("ip_allocation_policy").([]interface{})
config := policy[0].(map[string]interface{})
stackType := config["stack_type"].(string)

req := &container.UpdateClusterRequest{
Update: &container.ClusterUpdate{
DesiredStackType: stackType,
},
}
updateF:= updateFunc(req, "updating Cluster stack type")

// Call update serially.
if err := lockedCall(lockKey, updateF); err != nil {
return err
}

log.Printf("[INFO] GKE cluster %s stack type has been updated to %s", d.Id(), stackType)
}

if d.HasChange("enable_l4_ilb_subsetting") {
// This field can be changed from false to true but not from false to true. CustomizeDiff handles that check.
enabled := d.Get("enable_l4_ilb_subsetting").(bool)
Expand Down Expand Up @@ -3963,7 +3994,10 @@ func expandClusterAddonsConfig(configured interface{}) *container.AddonsConfig {
return ac
}

func expandIPAllocationPolicy(configured interface{}, networkingMode string) (*container.IPAllocationPolicy, error) {
func expandIPAllocationPolicy(
configured interface{}, networkingMode string, datapathProvider string,
privateIpv6GoogleAccess string, enableL4ilbSubnetting bool) (*container.IPAllocationPolicy, error) {

l := configured.([]interface{})
if len(l) == 0 || l[0] == nil {
if networkingMode == "VPC_NATIVE" {
Expand All @@ -3972,20 +4006,35 @@ func expandIPAllocationPolicy(configured interface{}, networkingMode string) (*c
return &container.IPAllocationPolicy{
UseIpAliases: false,
UseRoutes: true,
StackType: "IPV4",
ForceSendFields: []string{"UseIpAliases"},
}, nil
}

config := l[0].(map[string]interface{})
stackType := config["stack_type"].(string)

if stackType == "IPV4_IPV6" {
if datapathProvider != "ADVANCED_DATAPATH" {
return nil, fmt.Errorf("Dual-stack networking is only available for VPC-native clusters with GKE DataplaneV2 enabled.")
}
if privateIpv6GoogleAccess == "PRIVATE_IPV6_GOOGLE_ACCESS_TO_GOOGLE" || privateIpv6GoogleAccess == "PRIVATE_IPV6_GOOGLE_ACCESS_BIDIRECTIONAL" {
return nil, fmt.Errorf("Dual-stack clusters doesn't support Private Google Access over IPv6.")
}
if !enableL4ilbSubnetting {
return nil, fmt.Errorf("For dual-stack clusters, GKE will enable L4 ILB Subnetting automatically. Enable L4 ILB Subnetting to avoid drift.")
}
}
600lyy marked this conversation as resolved.
Show resolved Hide resolved

return &container.IPAllocationPolicy{
UseIpAliases: networkingMode == "VPC_NATIVE" || networkingMode == "",
ClusterIpv4CidrBlock: config["cluster_ipv4_cidr_block"].(string),
ServicesIpv4CidrBlock: config["services_ipv4_cidr_block"].(string),

ClusterSecondaryRangeName: config["cluster_secondary_range_name"].(string),
ServicesSecondaryRangeName: config["services_secondary_range_name"].(string),
ForceSendFields: []string{"UseIpAliases"},
UseRoutes: networkingMode == "ROUTES",
StackType: stackType,
}, nil
}

Expand Down Expand Up @@ -5138,12 +5187,15 @@ func flattenIPAllocationPolicy(c *container.Cluster, d *schema.ResourceData, con
}

p := c.IpAllocationPolicy
log.Printf("[INFO] stack type in the configuration is %s\n", p.StackType)

return []map[string]interface{}{
{
"cluster_ipv4_cidr_block": p.ClusterIpv4CidrBlock,
"services_ipv4_cidr_block": p.ServicesIpv4CidrBlock,
"cluster_secondary_range_name": p.ClusterSecondaryRangeName,
"services_secondary_range_name": p.ServicesSecondaryRangeName,
"stack_type": p.StackType,
},
}, nil
}
Expand Down
187 changes: 187 additions & 0 deletions mmv1/third_party/terraform/tests/resource_container_cluster_test.go.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2149,6 +2149,90 @@ func TestAccContainerCluster_withIPAllocationPolicy_specificSizes(t *testing.T)
})
}

func TestAccContainerCluster_stackType_withDualStack(t *testing.T) {
t.Parallel()

clusterName := fmt.Sprintf("tf-test-cluster-%s", RandString(t, 10))
containerNetName := fmt.Sprintf("tf-test-cluster-%s", RandString(t, 10))
resourceName := "google_container_cluster.with_stack_type"

VcrTest(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Providers: TestAccProviders,
600lyy marked this conversation as resolved.
Show resolved Hide resolved
CheckDestroy: testAccCheckContainerClusterDestroyProducer(t),
Steps: []resource.TestStep{
{
Config: testAccContainerCluster_stackType_withDualStack(containerNetName, clusterName),
Check: resource.ComposeTestCheckFunc(
resource.TestCheckResourceAttr(resourceName, "ip_allocation_policy.0.stack_type", "IPV4_IPV6"),
),
},
{
ResourceName: resourceName,
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"min_master_version"},
},
},
})
}

func TestAccContainerCluster_stackType_withSingleStack(t *testing.T) {
t.Parallel()

clusterName := fmt.Sprintf("tf-test-cluster-%s", RandString(t, 10))
containerNetName := fmt.Sprintf("tf-test-cluster-%s", RandString(t, 10))
resourceName := "google_container_cluster.with_stack_type"

VcrTest(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Providers: TestAccProviders,
600lyy marked this conversation as resolved.
Show resolved Hide resolved
CheckDestroy: testAccCheckContainerClusterDestroyProducer(t),
Steps: []resource.TestStep{
{
Config: testAccContainerCluster_stackType_withSingleStack(containerNetName, clusterName),
Check: resource.ComposeTestCheckFunc(
resource.TestCheckResourceAttr(resourceName, "ip_allocation_policy.0.stack_type", "IPV4"),
),
},
{
ResourceName: resourceName,
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"min_master_version"},
},
},
})
}

func TestAccContainerCluster_stackType_withDefaultStack(t *testing.T) {
600lyy marked this conversation as resolved.
Show resolved Hide resolved
t.Parallel()

clusterName := fmt.Sprintf("tf-test-cluster-%s", RandString(t, 10))
containerNetName := fmt.Sprintf("tf-test-cluster-%s", RandString(t, 10))
resourceName := "google_container_cluster.with_stack_type"

VcrTest(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
Providers: TestAccProviders,
600lyy marked this conversation as resolved.
Show resolved Hide resolved
CheckDestroy: testAccCheckContainerClusterDestroyProducer(t),
Steps: []resource.TestStep{
{
Config: testAccContainerCluster_stackType_withDefaultStack(containerNetName, clusterName),
Check: resource.ComposeTestCheckFunc(
resource.TestCheckResourceAttr(resourceName, "ip_allocation_policy.0.stack_type", "IPV4"),
),
},
{
ResourceName: resourceName,
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"min_master_version"},
},
},
})
}

func TestAccContainerCluster_nodeAutoprovisioning(t *testing.T) {
t.Parallel()

Expand Down Expand Up @@ -6252,6 +6336,109 @@ resource "google_container_cluster" "with_ip_allocation_policy" {
`, containerNetName, clusterName)
}

func testAccContainerCluster_stackType_withDualStack(containerNetName string, clusterName string) string {
return fmt.Sprintf(`
resource "google_compute_network" "container_network" {
name = "%s"
auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "container_subnetwork" {
name = google_compute_network.container_network.name
network = google_compute_network.container_network.name
region = "us-central1"

ip_cidr_range = "10.2.0.0/16"
stack_type = "IPV4_IPV6"
ipv6_access_type = "EXTERNAL"
}
600lyy marked this conversation as resolved.
Show resolved Hide resolved

resource "google_container_cluster" "with_stack_type" {
name = "%s"
location = "us-central1-a"
network = google_compute_network.container_network.name
subnetwork = google_compute_subnetwork.container_subnetwork.name

min_master_version = "1.25"
initial_node_count = 1
datapath_provider = "ADVANCED_DATAPATH"
enable_l4_ilb_subsetting = true

ip_allocation_policy {
cluster_ipv4_cidr_block = "10.0.0.0/16"
services_ipv4_cidr_block = "10.1.0.0/16"
stack_type = "IPV4_IPV6"
}
}
`, containerNetName, clusterName)
}

func testAccContainerCluster_stackType_withSingleStack(containerNetName string, clusterName string) string {
return fmt.Sprintf(`
resource "google_compute_network" "container_network" {
name = "%s"
auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "container_subnetwork" {
name = google_compute_network.container_network.name
network = google_compute_network.container_network.name
region = "us-central1"

ip_cidr_range = "10.2.0.0/16"
}

resource "google_container_cluster" "with_stack_type" {
name = "%s"
location = "us-central1-a"
network = google_compute_network.container_network.name
subnetwork = google_compute_subnetwork.container_subnetwork.name

min_master_version = "1.25"
initial_node_count = 1
enable_l4_ilb_subsetting = true

ip_allocation_policy {
cluster_ipv4_cidr_block = "10.0.0.0/16"
services_ipv4_cidr_block = "10.1.0.0/16"
stack_type = "IPV4"
}
}
`, containerNetName, clusterName)
}

func testAccContainerCluster_stackType_withDefaultStack(containerNetName string, clusterName string) string {
return fmt.Sprintf(`
resource "google_compute_network" "container_network" {
name = "%s"
auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "container_subnetwork" {
name = google_compute_network.container_network.name
network = google_compute_network.container_network.name
region = "us-central1"

ip_cidr_range = "10.2.0.0/16"
}

resource "google_container_cluster" "with_stack_type" {
name = "%s"
location = "us-central1-a"
network = google_compute_network.container_network.name
subnetwork = google_compute_subnetwork.container_subnetwork.name

min_master_version = "1.25"
initial_node_count = 1

ip_allocation_policy {
cluster_ipv4_cidr_block = "10.0.0.0/16"
services_ipv4_cidr_block = "10.1.0.0/16"
}
}
`, containerNetName, clusterName)
}

func testAccContainerCluster_withResourceUsageExportConfig(clusterName, datasetId, enableMetering string) string {
return fmt.Sprintf(`
provider "google" {
Expand Down