fix: firewall rules for autopilot clusters are ineffective. add cluster_network_tag to autopilot cluster network_tags if firewalls are toggled on #1817

GorginZ · 2023-12-14T06:13:23Z

close Firewall rules for Autopilot clusters are ineffective #1230
add cluster_network_tag to autopilot clusters' node_pool_auto_config.network_tags if any of : add_cluster_firewall_rules, add_master_webhook_firewall_rules or add_shadow_firewall_rules are toggled true.
assign network tags if network_tags are provided or need to be assigned - to maintain existing behavior of not assigning any/empty if not required
add example/test
fix: add cluster_network_tag to autopilot cluster network_tags if firewalls are toggled on #1655 pull went stale because I was busy, issue still open so opening a freshy.

…firewall rules toggled on

…s a fw's target tag

…rrection to readme

…tag value

…work-tags

apeabody · 2024-06-11T16:27:42Z

This is almost certainly unrelated to this change, perhaps recent change in the API. We'll address in a seperate PR.

        	Error:      	Not equal: 
        	            	expected: "{\n    \"state\": \"DECRYPTED\"\n  }"
        	            	actual  : "{\n    \"currentState\": \"CURRENT_STATE_DECRYPTED\",\n    \"state\": \"DECRYPTED\"\n  }"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,3 @@
        	            	 {
        	            	+    "currentState": "CURRENT_STATE_DECRYPTED",
        	            	     "state": "DECRYPTED"
        	Test:       	TestSimpleRegional
        	Messages:   	expected databaseEncryption to match fixture {
        	            	    "state": "DECRYPTED"
        	            	  }

apeabody · 2024-06-11T20:14:23Z

/gcbrun

(#1974)

GorginZ · 2024-06-11T22:41:53Z

/gcbrun

still red I see.

apeabody · 2024-06-11T23:49:06Z

OK, these finding appear to be relevant:

Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:45Z command.go:185: "cloud-foundation-cicd"
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:45Z command.go:100: Running command gcloud with args [compute firewall-rules --project ci-gke-19fdc5db-f0aj describe gke-autopilot-private-firewal-intra-cluster-egress --format json]
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:46Z command.go:185: ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:46Z command.go:185:  - The resource 'projects/ci-gke-19fdc5db-f0aj/global/firewalls/gke-autopilot-private-firewal-intra-cluster-egress' was not found
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:46Z command.go:185: 
Step #90 - "verify autopilot-private-firewalls":     gcloud.go:84: error while running command: exit status 1; ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
Step #90 - "verify autopilot-private-firewalls":

GorginZ · 2024-06-12T09:14:31Z

OK, these finding appear to be relevant:

Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:45Z command.go:185: "cloud-foundation-cicd"
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:45Z command.go:100: Running command gcloud with args [compute firewall-rules --project ci-gke-19fdc5db-f0aj describe gke-autopilot-private-firewal-intra-cluster-egress --format json]
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:46Z command.go:185: ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:46Z command.go:185:  - The resource 'projects/ci-gke-19fdc5db-f0aj/global/firewalls/gke-autopilot-private-firewal-intra-cluster-egress' was not found
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-11T20:59:46Z command.go:185: 
Step #90 - "verify autopilot-private-firewalls":     gcloud.go:84: error while running command: exit status 1; ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
Step #90 - "verify autopilot-private-firewalls":

Well I wouldn't expect a fw named gke-autopilot-private-firewal-intra-cluster-egress but I would expect one named gke-autopilot-private-firewalls-intra-cluster-egress

Hmm will look now. All tests did used to be green, I did the whole deployment in my org.
https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/blob/master/modules/beta-autopilot-private-cluster/firewall.tf#L29

But... it should fail given this:

terraform-google-kubernetes-engine/test/integration/autopilot_private_firewalls/autopilot_private_firewalls_test.go

Line 39 in a4a56ff

n := fmt.Sprintf(fw, clusterName[:25])

Anyway that'll be it and I'll have to fix it regardless of what I may or may not remember. Maybe I renamed something.

GorginZ · 2024-06-13T10:19:11Z

@apeabody can we try once more :)

apeabody · 2024-06-13T15:08:59Z

/gcbrun

apeabody · 2024-06-13T15:48:36Z

/gcbrun

test/integration/autopilot_private_firewalls/autopilot_private_firewalls_test.go

apeabody · 2024-06-13T16:44:23Z

Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:34Z command.go:100: Running command gcloud with args [config get-value project --format json]
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:34Z command.go:185: "cloud-foundation-cicd"
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:34Z command.go:100: Running command gcloud with args [config get-value project --format json]
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:35Z command.go:185: "cloud-foundation-cicd"
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:35Z command.go:100: Running command gcloud with args [compute firewall-rules --project ci-gke-b70466df-s4fn describe gke-autopilot-private-firewalls-intra-cluster-egress --format json]
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:36Z command.go:185: ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:36Z command.go:185:  - The resource 'projects/ci-gke-b70466df-s4fn/global/firewalls/gke-autopilot-private-firewalls-intra-cluster-egress' was not found
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:36Z command.go:185: 
Step #90 - "verify autopilot-private-firewalls":     gcloud.go:84: error while running command: exit status 1; ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
Step #90 - "verify autopilot-private-firewalls":          - The resource 'projects/ci-gke-b70466df-s4fn/global/firewalls/gke-autopilot-private-firewalls-intra-cluster-egress' was not found
Step #90 - "verify autopilot-private-firewalls":         
Step #90 - "verify autopilot-private-firewalls": 2024/06/13 16:33:36 RUN_STAGE env var set to verify
Step #90 - "verify autopilot-private-firewalls": 2024/06/13 16:33:36 Skipping stage teardown
Step #90 - "verify autopilot-private-firewalls": --- FAIL: TestAutopilotPrivateFirewalls (11.34s)

I do see a gke-autopilot-private-firewalls-cluster-intra-cluster-egress firewall

GorginZ · 2024-06-15T05:54:59Z

Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:34Z command.go:100: Running command gcloud with args [config get-value project --format json]
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:34Z command.go:185: "cloud-foundation-cicd"
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:34Z command.go:100: Running command gcloud with args [config get-value project --format json]
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:35Z command.go:185: "cloud-foundation-cicd"
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:35Z command.go:100: Running command gcloud with args [compute firewall-rules --project ci-gke-b70466df-s4fn describe gke-autopilot-private-firewalls-intra-cluster-egress --format json]
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:36Z command.go:185: ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:36Z command.go:185:  - The resource 'projects/ci-gke-b70466df-s4fn/global/firewalls/gke-autopilot-private-firewalls-intra-cluster-egress' was not found
Step #90 - "verify autopilot-private-firewalls": TestAutopilotPrivateFirewalls 2024-06-13T16:33:36Z command.go:185: 
Step #90 - "verify autopilot-private-firewalls":     gcloud.go:84: error while running command: exit status 1; ERROR: (gcloud.compute.firewall-rules.describe) Could not fetch resource:
Step #90 - "verify autopilot-private-firewalls":          - The resource 'projects/ci-gke-b70466df-s4fn/global/firewalls/gke-autopilot-private-firewalls-intra-cluster-egress' was not found
Step #90 - "verify autopilot-private-firewalls":         
Step #90 - "verify autopilot-private-firewalls": 2024/06/13 16:33:36 RUN_STAGE env var set to verify
Step #90 - "verify autopilot-private-firewalls": 2024/06/13 16:33:36 Skipping stage teardown
Step #90 - "verify autopilot-private-firewalls": --- FAIL: TestAutopilotPrivateFirewalls (11.34s)

I do see a gke-autopilot-private-firewalls-cluster-intra-cluster-egress firewall

Ah thanks @apeabody for posting this, lead me to notice my test code was trimming the -cluster suffix from the clusterName, no idea why I did that. Have pushed fix.

apeabody · 2024-06-17T15:06:40Z

/gcbrun

apeabody · 2024-06-17T16:00:36Z

Step #91 - "destroy autopilot-private-firewalls":         	Error:      	Received unexpected error:
Step #91 - "destroy autopilot-private-firewalls":         	            	FatalError{Underlying: error while running command: exit status 1; 
Step #91 - "destroy autopilot-private-firewalls":         	            	Error: Cannot destroy cluster because deletion_protection is set to true. Set it to false to proceed with cluster deletion.
Step #91 - "destroy autopilot-private-firewalls":         	            	}
Step #91 - "destroy autopilot-private-firewalls":         	Test:       	TestAutopilotPrivateFirewalls

examples/autopilot_private_firewalls/main.tf

set deletion_protection to false Co-authored-by: Andrew Peabody <andrewpeabody@google.com>

apeabody · 2024-06-17T22:53:18Z

/gcbrun

GorginZ · 2024-06-19T02:52:38Z

@apeabody need anything else from my end?

apeabody

Thanks for the contribution @GorginZ!

…er_network_tag to autopilot cluster network_tags if firewalls are toggled on (terraform-google-modules#1817) Co-authored-by: Andrew Peabody <andrewpeabody@google.com>

GorginZ and others added 28 commits May 12, 2023 08:53

use var network_tags for egress firewal rules

2a36867

include service ranges in intra-egress rule

70b461b

get the cidr service range, not the name of the secondary range

4a103be

add cluster_network_tag to node_pool_auto_config.network_tags if any …

b70cfff

…firewall rules toggled on

add example test for autopilot private with firewall rules

c05ff1b

tf fmt

b81943a

readme

2982500

clusterNetworkTag var to assert cluster network tags are same value a…

82fb83a

…s a fw's target tag

readme notes

28071b1

rm some json tests, mostly interested in FW rules and private

c07eb65

test data

993f750

mm dont think I need kitchen

27a45e3

readme

3810721

tidy up comments on fw rules and tf-example foundation

8376ea2

Merge branch 'master' into gke-autopilot-network-tags

8aeb962

fw rules in base shared vpc not heirarchical just normal fw rules, co…

d43348a

…rrection to readme

test all expected fw rules created and target tag is cluster_network_…

a7af50c

…tag value

.kitchen.yml

416484b

linter

6c09110

Merge branch 'master' into gke-autopilot-network-tags

66772d4

Merge branch 'master' into gke-autopilot-network-tags

e25121d

Merge branch 'master' into gke-autopilot-network-tags

34e50bd

Merge branch 'master' into gke-autopilot-network-tags

7987177

Merge branch 'master' into gke-autopilot-network-tags

8684a25

Merge branch 'master' into gke-autopilot-network-tags

5b8456f

Merge branch 'master' into gke-autopilot-network-tags

d5f3ae8

Merge branch 'terraform-google-modules:master' into gke-autopilot-net…

51cb7d6

…work-tags

Merge branch 'terraform-google-modules:master' into gke-autopilot-net…

505c774

…work-tags

GorginZ requested review from ericyz and a team as code owners December 14, 2023 06:13

Merge branch 'master' into gke-autopilot-network-tags

a4a56ff

GorginZ and others added 2 commits June 13, 2024 20:14

increase clustername fmt limit in test

b68b4b3

Merge branch 'master' into gke-autopilot-network-tags

3961a73

Merge branch 'master' into gke-autopilot-network-tags

684b763

apeabody reviewed Jun 13, 2024

View reviewed changes

test/integration/autopilot_private_firewalls/autopilot_private_firewalls_test.go Show resolved Hide resolved

GorginZ and others added 2 commits June 15, 2024 15:24

Merge branch 'master' into gke-autopilot-network-tags

4b4c47c

rm char limit on clusterName in test

8eb4e8d

GorginZ requested a review from apeabody June 15, 2024 22:40

apeabody reviewed Jun 17, 2024

View reviewed changes

examples/autopilot_private_firewalls/main.tf Outdated Show resolved Hide resolved

GorginZ and others added 2 commits June 18, 2024 08:37

Update examples/autopilot_private_firewalls/main.tf

bc77424

set deletion_protection to false Co-authored-by: Andrew Peabody <andrewpeabody@google.com>

terraform fmt

f836567

GorginZ requested a review from apeabody June 17, 2024 22:40

apeabody approved these changes Jun 20, 2024

View reviewed changes

apeabody merged commit e7b20cd into terraform-google-modules:master Jun 20, 2024
4 checks passed

release-please bot mentioned this pull request Jun 20, 2024

chore(master): release 31.1.0 #1961

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: firewall rules for autopilot clusters are ineffective. add cluster_network_tag to autopilot cluster network_tags if firewalls are toggled on #1817

fix: firewall rules for autopilot clusters are ineffective. add cluster_network_tag to autopilot cluster network_tags if firewalls are toggled on #1817

GorginZ commented Dec 14, 2023

apeabody commented Jun 11, 2024

apeabody commented Jun 11, 2024 •

edited

Loading

GorginZ commented Jun 11, 2024

apeabody commented Jun 11, 2024

GorginZ commented Jun 12, 2024

GorginZ commented Jun 13, 2024

apeabody commented Jun 13, 2024

apeabody commented Jun 13, 2024

apeabody commented Jun 13, 2024

GorginZ commented Jun 15, 2024 •

edited

Loading

apeabody commented Jun 17, 2024

apeabody commented Jun 17, 2024

apeabody commented Jun 17, 2024

GorginZ commented Jun 19, 2024

apeabody left a comment

fix: firewall rules for autopilot clusters are ineffective. add cluster_network_tag to autopilot cluster network_tags if firewalls are toggled on #1817

fix: firewall rules for autopilot clusters are ineffective. add cluster_network_tag to autopilot cluster network_tags if firewalls are toggled on #1817

Conversation

GorginZ commented Dec 14, 2023

apeabody commented Jun 11, 2024

apeabody commented Jun 11, 2024 • edited Loading

GorginZ commented Jun 11, 2024

apeabody commented Jun 11, 2024

GorginZ commented Jun 12, 2024

GorginZ commented Jun 13, 2024

apeabody commented Jun 13, 2024

apeabody commented Jun 13, 2024

apeabody commented Jun 13, 2024

GorginZ commented Jun 15, 2024 • edited Loading

apeabody commented Jun 17, 2024

apeabody commented Jun 17, 2024

apeabody commented Jun 17, 2024

GorginZ commented Jun 19, 2024

apeabody left a comment

Choose a reason for hiding this comment

apeabody commented Jun 11, 2024 •

edited

Loading

GorginZ commented Jun 15, 2024 •

edited

Loading