Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beta support concurrent nodepool CRUD Operations #1

Merged
merged 2 commits into from
Oct 27, 2022
Merged

Conversation

danielvegamyhre
Copy link
Owner

@danielvegamyhre danielvegamyhre commented Oct 27, 2022

The purpose of these changes is to implement support in the beta provider for concurrent nodepool CRUD operations on a single cluster. The GA provider should be unchanged.

The changes to the beta provider include:

  • Updating the global mutex store to use sync.RWMutex instead of sync.Mutex and adding the necessary methods to the MutexKV struct to support acquiring shared/read locks.
  • Removing the polling for cluster "ready" status, since with support for concurrent operations on the same cluster we no longer need to wait for the cluster to have no operations running on it before proceeding.
  • For NP CRUD operations, instead of acquiring an exclusive/write lock on the cluster, we acquire a read/shared lock on the cluster and an exclusive/write lock on the nodepool. This ensures cluster-wide operations (e.g. UpdateCluster) still will block NP level operations, but NP level operations on different NPs won't block eachother. These NP-level mutexes use cluster hash + nodepool name to guarantee lock key uniqueness.
  • Add retry logic to NP CRUD operations to retry while it receives an "incompatible operation" error (which has the FAILED_PRECONDITION canonical code), to safely retry concurrent operations blocked by a lock conflict with another operation.

@danielvegamyhre danielvegamyhre merged commit dad7696 into main Oct 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant