Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for non-preemptible secondary dataproc workers #5686

Conversation

tsamaras
Copy link
Contributor

@tsamaras tsamaras commented Feb 7, 2022

Added the ability to set the preemptibility of secondary workers as requested in hashicorp/terraform-provider-google#7882 to the existing preemptible worker config. While it's not ideal from a semantic perspective, it doesn't break anything.

If this PR is for Terraform, I acknowledge that I have:

  • Searched through the issue tracker for an open issue that this either resolves or contributes to, commented on it to claim it, and written "fixes {url}" or "part of {url}" in this PR description. If there were no relevant open issues, I opened one and commented that I would like to work on it (not necessary for very small changes).
  • Generated Terraform, and ran make test and make lint to ensure it passes unit and linter tests.
  • Ensured that all new fields I added that can be set by a user appear in at least one example (for generated resources) or third_party test (for handwritten resources or update tests).
  • Ran relevant acceptance tests (If the acceptance tests do not yet pass or you are unable to run them, please let your reviewer know).
  • Read the Release Notes Guide before writing my release note below.

Release Note Template for Downstream PRs (will be copied)

dataproc:added `preemptibility` field to the `preemptible_worker_config` of `google_dataproc_cluster`

tsamaras and others added 25 commits January 28, 2022 16:06
…m#5649)

`settings.backup_configuration.binary_log_enabled` may be true even if `settings.backup_configuration.enabled` is false.

This behaviour as changed in GoogleCloudPlatform#4907

Co-authored-by: Matthew Barnes <matthew.simon.barnes@gmail.com>
@modular-magician
Copy link
Collaborator

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review.

Thanks for your contribution! A human will be with you soon.

@ScottSuarez, please review this PR or find an appropriate assignee.

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 82 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 82 insertions(+), 3 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@ScottSuarez
Copy link
Contributor

/gcbrun

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 82 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 82 insertions(+), 3 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician
Copy link
Collaborator

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic|TestAccApigeeEnvironmentIamPolicyGenerated|TestAccBigQueryDataset_basic|TestAccBigQueryDataset_access|TestAccCloudbuildWorkerPool_basic|TestAccComputeGlobalForwardingRule_externalCndLbWithBackendBucketExample|TestAccComputeInstanceTemplate_reservationAffinities|TestAccComputeInstance_guestAccelerator|TestAccComputeInstance_guestAcceleratorSkip|TestAccDataprocCluster_nonPreemptibleSecondary|TestAccDataprocCluster_updatable|TestAccDataprocCluster_withConfigOverrides|TestAccEventarcTrigger_BasicHandWritten|TestAccSqlDatabaseInstance_withPrivateNetwork_withAllocatedIpRange|TestAccTags You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=261055

@ScottSuarez
Copy link
Contributor

Isn't this already supported simply by supplying a preemptible_worker_config. I'm not sure what the add on here is?

@tsamaras
Copy link
Contributor Author

tsamaras commented Feb 26, 2022

It's possible I touched more than was necessary, it's my first time contributing to the repo. My intent is to extend the preemtible_worker_config so that you can configure the preemptibility of the secondary workers to be non-preemptible.

The use case I'm implementing this for is a cluster for a job that only runs periodically with a large resource requirement that is sensitive to task failures, so we want autoscaling but we do not want nodes to be preempted

@ScottSuarez
Copy link
Contributor

/gcbrun

@modular-magician
Copy link
Collaborator

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 3 files changed, 82 insertions(+), 3 deletions(-))
Terraform Beta: Diff ( 3 files changed, 82 insertions(+), 3 deletions(-))
TF Validator: Diff ( 2 files changed, 3 insertions(+), 3 deletions(-))

@modular-magician
Copy link
Collaborator

I have triggered VCR tests in RECORDING mode for the following tests that failed during VCR: TestAccDataSourceComputeRouter|TestAccDataSourceComputeRouterStatus|TestAccDatasourceGoogleServiceNetworkingPeeredDnsDomain_basic|TestAccCloudbuildWorkerPool_basic|TestAccComputeHaVpnGateway_computeHaVpnGatewayEncryptedInterconnectExample|TestAccComputeInterconnectAttachment_interconnectAttachmentBasicExample|TestAccComputeInterconnectAttachment_computeInterconnectAttachmentIpsecEncryptionExample|TestAccComputeRouter_routerBasicExample|TestAccComputeRouter_computeRouterEncryptedInterconnectExample|TestAccComputeRouterNat_withManualIpAndSubnetConfiguration|TestAccComputeRouterNat_withDisabledIndependentEndpointMapping|TestAccComputeRouter_basic|TestAccComputeRouter_noRegion|TestAccComputeRouter_full|TestAccComputeRouter_update|TestAccComputeRouterPeer_enable|TestAccComputeRouterInterface_basic|TestAccComputeRouterInterface_withTunnel|TestAccComputeRouter_updateAddRemoveBGP|TestAccComputeRouterPeer_basic|TestAccComputeRouterPeer_advertiseMode|TestAccComputeRouterPeer_bfd|TestAccComputeVpnTunnel_router|TestAccContainerCluster_withAuthenticatorGroupsConfig|TestAccServiceNetworkingPeeredDNSDomain_basic|TestAccServiceUsageConsumerQuotaOverride_consumerQuotaOverrideExample|TestAccServiceUsageConsumerQuotaOverride_consumerQuotaOverrideZeroValueExample|TestAccServiceUsageConsumerQuotaOverride_regionConsumerQuotaOverrideExample You can view the result here: https://ci-oss.hashicorp.engineering/viewQueued.html?itemId=264835

Copy link
Contributor

@ScottSuarez ScottSuarez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Truly a strange field. It looks good though

@ScottSuarez ScottSuarez merged commit a8ac0e6 into GoogleCloudPlatform:main Mar 7, 2022
lcaggio pushed a commit to lcaggio/magic-modules that referenced this pull request Mar 16, 2022
…udPlatform#5686)

Co-authored-by: Matthew Barnes <matthew.simon.barnes@gmail.com>
Co-authored-by: Stephen Lewis (Burrows) <stephenrlewis@google.com>
Co-authored-by: Riley Karson <rileykarson@google.com>
Co-authored-by: Scott Suarez <ScottMSuarez@gmail.com>
Co-authored-by: Sampath Kumar <sampathm@google.com>
Co-authored-by: Sam Levenick <slevenick@google.com>
Co-authored-by: John Pellman <pellman.john@gmail.com>
Co-authored-by: Jacek Kikiewicz <jaceq@users.noreply.github.com>
Co-authored-by: Alex Ellis <alexellis@google.com>
Co-authored-by: megan07 <mbang@hashicorp.com>
Co-authored-by: Daniel Randell <drandell@users.noreply.github.com>
Co-authored-by: Iris Chen <10179943+iyabchen@users.noreply.github.com>
Co-authored-by: prateek2408 <prateek.khushalani@gmail.com>
lcaggio pushed a commit to lcaggio/magic-modules that referenced this pull request Mar 17, 2022
…udPlatform#5686)

Co-authored-by: Matthew Barnes <matthew.simon.barnes@gmail.com>
Co-authored-by: Stephen Lewis (Burrows) <stephenrlewis@google.com>
Co-authored-by: Riley Karson <rileykarson@google.com>
Co-authored-by: Scott Suarez <ScottMSuarez@gmail.com>
Co-authored-by: Sampath Kumar <sampathm@google.com>
Co-authored-by: Sam Levenick <slevenick@google.com>
Co-authored-by: John Pellman <pellman.john@gmail.com>
Co-authored-by: Jacek Kikiewicz <jaceq@users.noreply.github.com>
Co-authored-by: Alex Ellis <alexellis@google.com>
Co-authored-by: megan07 <mbang@hashicorp.com>
Co-authored-by: Daniel Randell <drandell@users.noreply.github.com>
Co-authored-by: Iris Chen <10179943+iyabchen@users.noreply.github.com>
Co-authored-by: prateek2408 <prateek.khushalani@gmail.com>
betsy-lichtenberg pushed a commit to betsy-lichtenberg/magic-modules that referenced this pull request Apr 25, 2022
…udPlatform#5686)

Co-authored-by: Matthew Barnes <matthew.simon.barnes@gmail.com>
Co-authored-by: Stephen Lewis (Burrows) <stephenrlewis@google.com>
Co-authored-by: Riley Karson <rileykarson@google.com>
Co-authored-by: Scott Suarez <ScottMSuarez@gmail.com>
Co-authored-by: Sampath Kumar <sampathm@google.com>
Co-authored-by: Sam Levenick <slevenick@google.com>
Co-authored-by: John Pellman <pellman.john@gmail.com>
Co-authored-by: Jacek Kikiewicz <jaceq@users.noreply.github.com>
Co-authored-by: Alex Ellis <alexellis@google.com>
Co-authored-by: megan07 <mbang@hashicorp.com>
Co-authored-by: Daniel Randell <drandell@users.noreply.github.com>
Co-authored-by: Iris Chen <10179943+iyabchen@users.noreply.github.com>
Co-authored-by: prateek2408 <prateek.khushalani@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.