Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Secure Web Proxy fields gatewaySecurityPolicy and certificateUrls supports updates. #10549

Merged
merged 8 commits into from
May 10, 2024

Conversation

Samir-Cit
Copy link
Contributor

@Samir-Cit Samir-Cit commented Apr 29, 2024

This PR change the fields gatewaySecurityPolicy and certificateUrls on Secure Web Proxy (google_network_services_gateway) resource so both can be updatable.

Fixes: hashicorp/terraform-provider-google#14600

  • It was needed a new pre-update code to be able to update those fields.
  • Also changed the beta version of the product to use v1beta1.
  • Tests were updated to match the new version.
  • Other small indentation (nit) fixes on this PR.

Release Note Template for Downstream PRs (will be copied)

networkservices: fixed `gateway_security_policy` and `certificate_urls` fields in `google_network_services_gateway` to be updatable

@modular-magician modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label Apr 29, 2024
@Samir-Cit Samir-Cit marked this pull request as ready for review April 30, 2024 14:44
@github-actions github-actions bot requested a review from roaks3 April 30, 2024 14:44
Copy link

Hello! I am a robot. Tests will require approval from a repository maintainer to run.

@roaks3, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

@modular-magician modular-magician added service/networkservices-traffic-director service/networkservices-media-cdn and removed awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests labels May 3, 2024
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 237 insertions(+), 219 deletions(-))
google-beta provider: Diff ( 13 files changed, 258 insertions(+), 240 deletions(-))
terraform-google-conversion: Diff ( 11 files changed, 22 insertions(+), 22 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3638
Passed tests: 3222
Skipped tests: 369
Affected tests: 47

Click here to see the affected service packages
all service packages are affected

Action taken

Found 47 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccNetworkServicesEdgeCacheKeyset_networkServicesEdgeCacheKeysetBasicExample|TestAccNetworkServicesEdgeCacheKeyset_networkServicesEdgeCacheKeysetDualTokenExample|TestAccNetworkServicesEdgeCacheKeyset_update|TestAccNetworkServicesEdgeCacheOrigin_networkServicesEdgeCacheOriginAdvancedExample|TestAccNetworkServicesEdgeCacheOrigin_networkServicesEdgeCacheOriginBasicExample|TestAccNetworkServicesEdgeCacheOrigin_networkServicesEdgeCacheOriginV4authExample|TestAccNetworkServicesEdgeCacheOrigin_updateAndImport|TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceAdvancedExample|TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceBasicExample|TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceDualTokenExample|TestAccNetworkServicesEdgeCacheService_updateAndImport|TestAccNetworkServicesEndpointPolicy_networkServicesEndpointPolicyBasicExample|TestAccNetworkServicesEndpointPolicy_networkServicesEndpointPolicyEmptyMatchExample|TestAccNetworkServicesEndpointPolicy_update|TestAccNetworkServicesGateway_minimalSwp|TestAccNetworkServicesGateway_multipleSwpGatewaysDifferentNetwork|TestAccNetworkServicesGateway_multipleSwpGatewaysDifferentSubnetwork|TestAccNetworkServicesGateway_networkServicesGatewayAdvancedExample|TestAccNetworkServicesGateway_networkServicesGatewayBasicExample|TestAccNetworkServicesGateway_networkServicesGatewayMultipleSwpSameNetworkExample|TestAccNetworkServicesGateway_networkServicesGatewaySecureWebProxyExample|TestAccNetworkServicesGateway_networkServicesGatewaySecureWebProxyWithoutAddresses|TestAccNetworkServicesGateway_update|TestAccNetworkServicesGateway_updateSwp|TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteActionsExample|TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteBasicExample|TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteMatchesAndActionsExample|TestAccNetworkServicesGrpcRoute_update|TestAccNetworkServicesHttpRoute_networkServicesHttpRouteActionsExample|TestAccNetworkServicesHttpRoute_networkServicesHttpRouteBasicExample|TestAccNetworkServicesHttpRoute_networkServicesHttpRouteMatchesAndActionsExample|TestAccNetworkServicesHttpRoute_networkServicesHttpRouteMeshBasicExample|TestAccNetworkServicesHttpRoute_update|TestAccNetworkServicesMesh_networkServicesMeshBasicExample|TestAccNetworkServicesMesh_networkServicesMeshNoPortExample|TestAccNetworkServicesMesh_update|TestAccNetworkServicesServiceBinding_networkServicesServiceBindingBasicExample|TestAccNetworkServicesServiceBinding_update|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteActionsExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteBasicExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteGatewayBasicExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteMeshBasicExample|TestAccNetworkServicesTcpRoute_update|TestAccNetworkServicesTlsRoute_networkServicesTlsRouteBasicExample|TestAccNetworkServicesTlsRoute_networkServicesTlsRouteGatewayBasicExample|TestAccNetworkServicesTlsRoute_networkServicesTlsRouteMeshBasicExample|TestAccNetworkServicesTlsRoute_update

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccNetworkServicesEndpointPolicy_networkServicesEndpointPolicyBasicExample[Debug log]
TestAccNetworkServicesEndpointPolicy_networkServicesEndpointPolicyEmptyMatchExample[Debug log]
TestAccNetworkServicesEndpointPolicy_update[Debug log]
TestAccNetworkServicesGateway_minimalSwp[Debug log]
TestAccNetworkServicesGateway_multipleSwpGatewaysDifferentNetwork[Debug log]
TestAccNetworkServicesGateway_multipleSwpGatewaysDifferentSubnetwork[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewayAdvancedExample[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewayBasicExample[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewayMultipleSwpSameNetworkExample[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewaySecureWebProxyExample[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewaySecureWebProxyWithoutAddresses[Debug log]
TestAccNetworkServicesGateway_update[Debug log]
TestAccNetworkServicesGateway_updateSwp[Debug log]
TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteActionsExample[Debug log]
TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteBasicExample[Debug log]
TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteMatchesAndActionsExample[Debug log]
TestAccNetworkServicesGrpcRoute_update[Debug log]
TestAccNetworkServicesHttpRoute_networkServicesHttpRouteActionsExample[Debug log]
TestAccNetworkServicesHttpRoute_networkServicesHttpRouteBasicExample[Debug log]
TestAccNetworkServicesHttpRoute_networkServicesHttpRouteMatchesAndActionsExample[Debug log]
TestAccNetworkServicesHttpRoute_networkServicesHttpRouteMeshBasicExample[Debug log]
TestAccNetworkServicesHttpRoute_update[Debug log]
TestAccNetworkServicesMesh_networkServicesMeshBasicExample[Debug log]
TestAccNetworkServicesMesh_networkServicesMeshNoPortExample[Debug log]
TestAccNetworkServicesMesh_update[Debug log]
TestAccNetworkServicesServiceBinding_networkServicesServiceBindingBasicExample[Debug log]
TestAccNetworkServicesServiceBinding_update[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteActionsExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteBasicExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteGatewayBasicExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteMeshBasicExample[Debug log]
TestAccNetworkServicesTcpRoute_update[Debug log]
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteBasicExample[Debug log]
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteMeshBasicExample[Debug log]
TestAccNetworkServicesTlsRoute_update[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccNetworkServicesEdgeCacheKeyset_networkServicesEdgeCacheKeysetBasicExample[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheKeyset_networkServicesEdgeCacheKeysetDualTokenExample[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheKeyset_update[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheOrigin_networkServicesEdgeCacheOriginAdvancedExample[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheOrigin_networkServicesEdgeCacheOriginBasicExample[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheOrigin_networkServicesEdgeCacheOriginV4authExample[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheOrigin_updateAndImport[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceAdvancedExample[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceBasicExample[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheService_networkServicesEdgeCacheServiceDualTokenExample[Error message] [Debug log]
TestAccNetworkServicesEdgeCacheService_updateAndImport[Error message] [Debug log]
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteGatewayBasicExample[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

Copy link
Contributor

@roaks3 roaks3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't given a full review yet, but it looks like there are a few different changes to existing behavior, and we would need to make sure this doesn't break users.

mmv1/products/networkservices/product.yaml Outdated Show resolved Hide resolved
@github-actions github-actions bot requested a review from roaks3 May 7, 2024 01:20
@modular-magician modular-magician added the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label May 7, 2024
@modular-magician modular-magician removed the awaiting-approval Pull requests that needs reviewer's approval to run presubmit tests label May 7, 2024
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 237 insertions(+), 219 deletions(-))
google-beta provider: Diff ( 2 files changed, 237 insertions(+), 219 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 47
Passed tests: 12
Skipped tests: 0
Affected tests: 35

Click here to see the affected service packages
  • networkservices

Action taken

Found 35 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccNetworkServicesEndpointPolicy_networkServicesEndpointPolicyBasicExample|TestAccNetworkServicesEndpointPolicy_networkServicesEndpointPolicyEmptyMatchExample|TestAccNetworkServicesEndpointPolicy_update|TestAccNetworkServicesGateway_minimalSwp|TestAccNetworkServicesGateway_multipleSwpGatewaysDifferentNetwork|TestAccNetworkServicesGateway_multipleSwpGatewaysDifferentSubnetwork|TestAccNetworkServicesGateway_networkServicesGatewayAdvancedExample|TestAccNetworkServicesGateway_networkServicesGatewayBasicExample|TestAccNetworkServicesGateway_networkServicesGatewayMultipleSwpSameNetworkExample|TestAccNetworkServicesGateway_networkServicesGatewaySecureWebProxyExample|TestAccNetworkServicesGateway_networkServicesGatewaySecureWebProxyWithoutAddresses|TestAccNetworkServicesGateway_update|TestAccNetworkServicesGateway_updateSwp|TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteActionsExample|TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteBasicExample|TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteMatchesAndActionsExample|TestAccNetworkServicesGrpcRoute_update|TestAccNetworkServicesHttpRoute_networkServicesHttpRouteActionsExample|TestAccNetworkServicesHttpRoute_networkServicesHttpRouteBasicExample|TestAccNetworkServicesHttpRoute_networkServicesHttpRouteMatchesAndActionsExample|TestAccNetworkServicesHttpRoute_networkServicesHttpRouteMeshBasicExample|TestAccNetworkServicesHttpRoute_update|TestAccNetworkServicesMesh_networkServicesMeshBasicExample|TestAccNetworkServicesMesh_networkServicesMeshNoPortExample|TestAccNetworkServicesMesh_update|TestAccNetworkServicesServiceBinding_networkServicesServiceBindingBasicExample|TestAccNetworkServicesServiceBinding_update|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteActionsExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteBasicExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteGatewayBasicExample|TestAccNetworkServicesTcpRoute_networkServicesTcpRouteMeshBasicExample|TestAccNetworkServicesTcpRoute_update|TestAccNetworkServicesTlsRoute_networkServicesTlsRouteBasicExample|TestAccNetworkServicesTlsRoute_networkServicesTlsRouteMeshBasicExample|TestAccNetworkServicesTlsRoute_update

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccNetworkServicesEndpointPolicy_networkServicesEndpointPolicyBasicExample[Debug log]
TestAccNetworkServicesEndpointPolicy_networkServicesEndpointPolicyEmptyMatchExample[Debug log]
TestAccNetworkServicesEndpointPolicy_update[Debug log]
TestAccNetworkServicesGateway_minimalSwp[Debug log]
TestAccNetworkServicesGateway_multipleSwpGatewaysDifferentNetwork[Debug log]
TestAccNetworkServicesGateway_multipleSwpGatewaysDifferentSubnetwork[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewayAdvancedExample[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewayBasicExample[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewayMultipleSwpSameNetworkExample[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewaySecureWebProxyExample[Debug log]
TestAccNetworkServicesGateway_networkServicesGatewaySecureWebProxyWithoutAddresses[Debug log]
TestAccNetworkServicesGateway_update[Debug log]
TestAccNetworkServicesGateway_updateSwp[Debug log]
TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteActionsExample[Debug log]
TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteBasicExample[Debug log]
TestAccNetworkServicesGrpcRoute_networkServicesGrpcRouteMatchesAndActionsExample[Debug log]
TestAccNetworkServicesGrpcRoute_update[Debug log]
TestAccNetworkServicesHttpRoute_networkServicesHttpRouteActionsExample[Debug log]
TestAccNetworkServicesHttpRoute_networkServicesHttpRouteBasicExample[Debug log]
TestAccNetworkServicesHttpRoute_networkServicesHttpRouteMatchesAndActionsExample[Debug log]
TestAccNetworkServicesHttpRoute_networkServicesHttpRouteMeshBasicExample[Debug log]
TestAccNetworkServicesHttpRoute_update[Debug log]
TestAccNetworkServicesMesh_networkServicesMeshBasicExample[Debug log]
TestAccNetworkServicesMesh_networkServicesMeshNoPortExample[Debug log]
TestAccNetworkServicesMesh_update[Debug log]
TestAccNetworkServicesServiceBinding_networkServicesServiceBindingBasicExample[Debug log]
TestAccNetworkServicesServiceBinding_update[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteActionsExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteBasicExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteGatewayBasicExample[Debug log]
TestAccNetworkServicesTcpRoute_networkServicesTcpRouteMeshBasicExample[Debug log]
TestAccNetworkServicesTcpRoute_update[Debug log]
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteBasicExample[Debug log]
TestAccNetworkServicesTlsRoute_networkServicesTlsRouteMeshBasicExample[Debug log]
TestAccNetworkServicesTlsRoute_update[Debug log]

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test

Copy link
Contributor

@roaks3 roaks3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, I'm just wondering about the pre_update, so we have a clear understanding of the previous behavior and the new behavior.

Also, I don't know if this addresses everything in the issue, and I don't know if we want this PR to close it. Specifically, the updates to field descriptions. Are those no longer needed, or perhaps coming in another PR?

@@ -0,0 +1,4 @@
if d.Get("type") == "SECURE_WEB_GATEWAY" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain what this is doing and why it was added? From what I can tell, it was previously possible for a user to have a google_network_services_gateway with type = "SECURE_WEB_GATEWAY", and then change a field like server_tls_policy or description, which would trigger this condition, and presumably produce different results than before. Why do these 2 fields need to be set, and what happened before when they weren't set?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just double checking with a terraform code using the actual provider and when trying to update the description of google_network_services_gateway with type = SECURE_WEB_GATEWAY it throw the following error:
Error: Error waiting for Updating Gateway: Error code 13, message: an internal error has occurred
The update send the name, the ID and the updated description fields only:

# google_network_services_gateway.foobar will be updated in-place
  ~ resource "google_network_services_gateway" "foobar" {
      ~ description                          = "my description" -> "new description"
        id                                   = "projects/{PROJECT}/locations/us-east1/gateways/{NAME}"
        name                                 = {NAME}
        # (18 unchanged attributes hidden)
    }

When updating through GCP console, the request sends all the fields that the Gateway contains:

gateway: {
  addresses: [1]
  certificate_urls: [1]
  description: "new description"
  gateway_security_policy: {POLICY}
  name: "projects/{PROJECT}/locations/us-east1/gateways/{NAME}"
  network: {NETWORK}
  ports: [1]
  subnetwork: {SUBNETWORK}
  type: "SECURE_WEB_GATEWAY"
}
update_mask: {
  paths: [
    0: "certificate_urls"
    1: "description"
    2: "gateway_security_policy"
]}

Also, GCP Console uses API v1beta1

To update using the API Explorer first I used all fields and then I was removing one by one to check which fields are required and those 2 (name and type) were required to update the Gateway through API Explorer

PATCH https://networkservices.googleapis.com/v1/projects/{PROJECT}/locations/us-east1/gateways/{NAME}?updateMask=description&key=[YOUR_API_KEY] HTTP/1.1

Authorization: Bearer [YOUR_ACCESS_TOKEN]
Accept: application/json
Content-Type: application/json

{
  "description": "new description",
  "name": "{NAME}",
  "type": "SECURE_WEB_GATEWAY"
}

Conclusion: Since GCP Console sends all fields it's able to update the gateway. So this modification that I did will send the required fields (name and type) to be able to update as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm ok, that's interesting behavior, but seems to match your logic. I suppose it's safer to stick with the condition you've included, but I do wonder if it might be cleaner to always include name and type in the request.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At first I did that... But if the type is other than SECURE_WEB_GATEWAY the request fail.
That broke some tests when I was starting the development.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@roaks3 regarding the fields description, it's still tracked on our radar but it was was deprioritized together with ngprabhu@

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks for those answers!

kautikdk pushed a commit to kautikdk/magic-modules that referenced this pull request May 16, 2024
pawelJas pushed a commit to pawelJas/magic-modules that referenced this pull request May 16, 2024
pengq-google pushed a commit to pengq-google/magic-modules that referenced this pull request May 21, 2024
Cheriit pushed a commit to Cheriit/magic-modules that referenced this pull request Jun 4, 2024
@Samir-Cit Samir-Cit deleted the fix/swp-fields branch June 24, 2024 16:57
pcostell pushed a commit to pcostell/magic-modules that referenced this pull request Jul 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update the network service gateway resource as the api gets updated.
3 participants