Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add metadata_fields support #467

Merged
merged 1 commit into from
Jul 19, 2023

Conversation

d-costa
Copy link
Contributor

@d-costa d-costa commented Jun 29, 2023

Note:

  • Had to change subnets variable type from list(map(string)) to list(object{...}), to allow heterogeneous types.

@d-costa d-costa requested review from a team and imrannayer as code owners June 29, 2023 09:25
@google-cla
Copy link

google-cla bot commented Jun 29, 2023

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@imrannayer
Copy link
Collaborator

/gcbrun

@imrannayer
Copy link
Collaborator

@d-costa can u plz add the changes in subnets-beta module

@d-costa d-costa force-pushed the master branch 2 times, most recently from 8056932 to 1a76c93 Compare July 7, 2023 16:47
@imrannayer
Copy link
Collaborator

/gcbrun

@d-costa
Copy link
Contributor Author

d-costa commented Jul 7, 2023

@imrannayer what's causing the linting error?

Error: The following tests have failed: check_documentation

@imrannayer
Copy link
Collaborator

@d-costa plz follow this doc
You are missing

make generate_docs

@imrannayer
Copy link
Collaborator

@d-costa metadata_fields is list of strings. You are using string.

@imrannayer
Copy link
Collaborator

/gcbrun

@imrannayer
Copy link
Collaborator

@d-costa Integration test is failing with following error:

Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: Error: Missing required argument
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: "log_config.0.aggregation_interval": one of
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: `log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: must be specified
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: Error: Missing required argument
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: "log_config.0.filter_expr": one of
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: `log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: must be specified
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: Error: Missing required argument
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: "log_config.0.flow_sampling": one of
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: `log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: must be specified
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: Error: Missing required argument
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185:   28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: 
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: "log_config.0.metadata": one of
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: `log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:43Z command.go:185: must be specified
Step #27 - "converge private-service-connect": TestPrivateServiceConnect 2023-07-10T14:58:45Z retry.go:99: Returning due to fatal error: FatalError{Underlying: error while running command: exit status 1; 
Step #27 - "converge private-service-connect": Error: Missing required argument
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect":   with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect":   on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect":   28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect": "log_config.0.aggregation_interval": one of
Step #27 - "converge private-service-connect": `log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect": must be specified
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect": Error: Missing required argument
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect":   with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect":   on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect":   28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect": "log_config.0.filter_expr": one of
Step #27 - "converge private-service-connect": `log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect": must be specified
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect": Error: Missing required argument
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect":   with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect":   on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect":   28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect": "log_config.0.flow_sampling": one of
Step #27 - "converge private-service-connect": `log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect": must be specified
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect": Error: Missing required argument
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect":   with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect":   on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect":   28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect": 
Step #27 - "converge private-service-connect": "log_config.0.metadata": one of
Step #27 - "converge private-service-connect": `log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect": must be specified}
Step #27 - "converge private-service-connect":     apply.go:34: 
Step #27 - "converge private-service-connect":         	Error Trace:	/builder/home/go/pkg/mod/github.com/gruntwork-io/terratest@v0.43.0/modules/terraform/apply.go:34
Step #27 - "converge private-service-connect":         	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/tft/terraform.go:393
Step #27 - "converge private-service-connect":         	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/tft/terraform.go:403
Step #27 - "converge private-service-connect":         	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/tft/terraform.go:427
Step #27 - "converge private-service-connect":         	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/utils/stages.go:31
Step #27 - "converge private-service-connect":         	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/tft/terraform.go:427
Step #27 - "converge private-service-connect":         	Error:      	Received unexpected error:
Step #27 - "converge private-service-connect":         	            	FatalError{Underlying: error while running command: exit status 1; 
Step #27 - "converge private-service-connect":         	            	Error: Missing required argument
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	  with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect":         	            	  on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect":         	            	  28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	"log_config.0.aggregation_interval": one of
Step #27 - "converge private-service-connect":         	            	`log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect":         	            	must be specified
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	Error: Missing required argument
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	  with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect":         	            	  on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect":         	            	  28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	"log_config.0.filter_expr": one of
Step #27 - "converge private-service-connect":         	            	`log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect":         	            	must be specified
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	Error: Missing required argument
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	  with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect":         	            	  on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect":         	            	  28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	"log_config.0.flow_sampling": one of
Step #27 - "converge private-service-connect":         	            	`log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect":         	            	must be specified
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	Error: Missing required argument
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	  with module.simple_vpc.module.subnets.google_compute_subnetwork.subnetwork["us-west1/my-subnetwork"],
Step #27 - "converge private-service-connect":         	            	  on ../../modules/subnets/main.tf line 28, in resource "google_compute_subnetwork" "subnetwork":
Step #27 - "converge private-service-connect":         	            	  28: resource "google_compute_subnetwork" "subnetwork" {
Step #27 - "converge private-service-connect":         	            	
Step #27 - "converge private-service-connect":         	            	"log_config.0.metadata": one of
Step #27 - "converge private-service-connect":         	            	`log_config.0.aggregation_interval,log_config.0.filter_expr,log_config.0.flow_sampling,log_config.0.metadata`
Step #27 - "converge private-service-connect":         	            	must be specified}

@d-costa
Copy link
Contributor Author

d-costa commented Jul 15, 2023

make docker_test_prepare was successful, but I was not able to run the tests:

david@[host] in terraform-google-network on master $ make docker_test_integration
docker run --rm -it \
        -e SERVICE_ACCOUNT_JSON \
        -v /home/david/code/terraform-google-network:/workspace \
        gcr.io/cloud-foundation-cicd/cft/developer-tools:1.13 \
        /usr/local/bin/test_integration.sh
Activated service account credentials for: [myserviceaccount.iam.gserviceaccount.com]
Automatically setting inputs from outputs of test/setup
Activated service account credentials for: [ci-network@ci-network-0224.iam.gserviceaccount.com]
-----> Starting Test Kitchen (v3.5.0)
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::UserError
>>>>>> Message: Kitchen YAML file /workspace/kitchen.yml does not exist.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

-----> Starting Test Kitchen (v3.5.0)
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::UserError
>>>>>> Message: Kitchen YAML file /workspace/kitchen.yml does not exist.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

make: *** [Makefile:58: docker_test_integration] Error 20

Kitchen's log contains: Message: Kitchen YAML file /workspace/kitchen.yml does not exist.

@imrannayer
Copy link
Collaborator

/gcbrun

@imrannayer
Copy link
Collaborator

@d-costa Integration test is failing

TestSimpleProject 2023-07-15T23:04:40Z command.go:185: }
    simple_project_test.go:52: 
        	Error Trace:	/workspace/test/integration/simple_project/simple_project_test.go:52
        	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/tft/terraform.go:408
        	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/tft/terraform.go:428
        	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/utils/stages.go:31
        	            				/builder/home/go/pkg/mod/github.com/!google!cloud!platform/cloud-foundation-toolkit/infra/blueprint-test@v0.6.0/pkg/tft/terraform.go:428
        	Error:      	Not equal: 
        	            	expected: map[string]interface {}{"aggregationInterval":"INTERVAL_10_MIN", "enable":true, "filterExpr":"false", "flowSampling":0.7, "metadata":"INCLUDE_ALL_METADATA", "metadata_fields":[]interface {}{}}
        	            	actual  : map[string]interface {}{"aggregationInterval":"INTERVAL_10_MIN", "enable":true, "filterExpr":"false", "flowSampling":0.7, "metadata":"INCLUDE_ALL_METADATA"}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,2 +1,2 @@
        	            	-(map[string]interface {}) (len=6) {
        	            	+(map[string]interface {}) (len=5) {
        	            	  (string) (len=19) "aggregationInterval": (string) (len=15) "INTERVAL_10_MIN",
        	            	@@ -5,5 +5,3 @@
        	            	  (string) (len=12) "flowSampling": (float64) 0.7,
        	            	- (string) (len=8) "metadata": (string) (len=20) "INCLUDE_ALL_METADATA",
        	            	- (string) (len=15) "metadata_fields": ([]interface {}) {
        	            	- }
        	            	+ (string) (len=8) "metadata": (string) (len=20) "INCLUDE_ALL_METADATA"
        	            	 }
        	Test:       	TestSimpleProject
        	Messages:   	log config should be correct

@imrannayer
Copy link
Collaborator

@d-costa you can execute test locally as follows

make docker_run

Generate list of all the tests

cft test list

Run a test TestPrivateServiceConnect

cft test run TestPrivateServiceConnect --verbose 

@imrannayer
Copy link
Collaborator

/gcbrun

2 similar comments
@imrannayer
Copy link
Collaborator

/gcbrun

@imrannayer
Copy link
Collaborator

/gcbrun

@imrannayer imrannayer merged commit f311cba into terraform-google-modules:master Jul 19, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants