Add cluster module (cloudera-labs#224)

* Rename  cm_cluster to cluster
* Rename cm_cluster_info to cluster_info
* Add state parameter to cluster
* Add module defaults groups for general Cloudera Manager access and cluster
* Update to create services on new cluster
* Add hosts, host templates, and direct role group assignments to cluster creation.
* Add tests for base config, service, role groups, host template, and direct role
* Add YAML args fixture
* Fix cm-client Python dependency
* Add Google Analytics tag for API documentation
* Refactor parcel management logic into shared utility class
* Add parcel activation for new cluster and reorder operations
* Refactor Parcel class to use class instance for stage
* Update integration test for host template assignment
* Add Parcel and create parse_parcel_result()
* Update to use Parcel class in module_utils
* Parameterize parcel test inputs
* Add initial pre-commit configuration, including black linter for Python files
* Comment out duplicate (and unused/invalid) 'when' keys found by pre-commit linting
* Comment out duplicate 'endpoint' path for updating Capacity Scheduler configuration found by pre-commit linting
* Update collection from initial pre-commit formatting for YAML and whitespace/EOL rules and Python black linting
* Add pre-commit GitHub Action
* Add role overrides for cluster creation
* Allow base role group discovery
* Allow role group direct assignment to hosts
* Add mutually_exclusive configs to various suboptions in argument_spec
* Add full example of a basic cluster to tests

Signed-off-by: rsuplina <rsuplina@cloudera.com>
Signed-off-by: Webster Mudge <wmudge@cloudera.com>
Co-authored-by: rsuplina <rsuplina@cloudera.com>

.github/workflows/pre-commit.yml

@@ -0,0 +1,28 @@
+# Copyright 2024 Cloudera, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: pre-commit
+
+on:
+  pull_request:
+  push:
+    branches: [main, devel]
+
+jobs:
+  pre-commit:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - uses: actions/setup-python@v3
+    - uses: pre-commit/action@v3.0.1

.github/workflows/publish_docs.yml

@@ -19,8 +19,8 @@ name: Publish documentation
 on:
   push:
     branches:
-      - 'main' 
-  
+      - 'main'
+
   workflow_dispatch:
 
 jobs:

.github/workflows/validate_pr.yml

@@ -21,7 +21,7 @@ on:
     branches:
       - 'release/**'
       - 'devel'
-    
+
 jobs:
   validate:
     runs-on: ubuntu-latest
@@ -34,7 +34,7 @@ jobs:
         with:
           python-version: '3.9'
           cache: 'pip'
-      
+
       - name: Set up Ansible and Ansible collections and roles
         run: |
           sudo update-alternatives --install /usr/bin/python python $(which python3) 1
@@ -49,16 +49,16 @@ jobs:
           ansible --version
           ansible-galaxy collection list
           ansible-galaxy role list
-      
+
       - name: Set up Ansible collection dependencies
         run: |
           ansible-builder introspect --write-pip final_python.txt --write-bindep final_bindep.txt /usr/share/ansible/collections
           [[ -f final_python.txt ]] && pip install -r final_python.txt || echo "No Python dependencies found."
           [[ -f final_bindep.txt ]] && bindep --file final_bindep.txt || echo "No system dependencies found."
-      
+
       - name: Report installed Python dependencies
         run: pip freeze
-      
+
       - name: Validate collection
         run: |
           pushd /usr/share/ansible/collections/ansible_collections/cloudera/cluster
@@ -75,7 +75,7 @@ jobs:
         run: |
           mkdir -p ./pr
           echo $PR_NUMBER > ./pr/pr_number
-      
+
       - name: Upload the PR number
         uses: actions/upload-artifact@v3
         with:

.github/workflows/validate_pr_docs.yml

@@ -21,7 +21,7 @@ on:
     branches:
       - 'release/**'
       - 'devel'
-    
+
   workflow_dispatch:
 
 jobs:
@@ -31,4 +31,4 @@ jobs:
     with:
       antsibull-log-upload: true
       collection-namespace: cloudera
-      collection-name: cluster
+      collection-name: cluster

.pre-commit-config.yaml

@@ -0,0 +1,26 @@
+# Copyright 2024 Cloudera, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.2.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+  - repo: https://github.com/psf/black
+    rev: 22.10.0
+    hooks:
+      - id: black

CONTRIBUTING.md

@@ -4,7 +4,7 @@ Thank you for considering contributions to the `cloudera.cluster` Ansible collec
 
 ## Submitting a pull request
 
-You can start work on issues that are not yet part of a [Milestone](https://github.com/cloudera-labs/cloudera.cluster/milestones) -- anything in our issue tracker that isn't assigned to a Milestone is considered the [backlog](https://github.com/cloudera-labs/cloudera.cluster/issues?q=is%3Aopen+is%3Aissue+no%3Amilestone). 
+You can start work on issues that are not yet part of a [Milestone](https://github.com/cloudera-labs/cloudera.cluster/milestones) -- anything in our issue tracker that isn't assigned to a Milestone is considered the [backlog](https://github.com/cloudera-labs/cloudera.cluster/issues?q=is%3Aopen+is%3Aissue+no%3Amilestone).
 
 Before you start working, please announce that you want to do so by commenting on the issue. _([Create an issue](https://github.com/cloudera-labs/cloudera.cluster/issues/new?labels=enhancement) if there isn't one yet, and you can also check out our [Discussions](https://github.com/cloudera-labs/cloudera.cluster/discussions) for ideas.)_ We try to ensure that all active work is assigned to a Milestone in order to keep our backlog accurate.
 

README.md

@@ -18,7 +18,7 @@ If you have any questions, want to chat about the collection's capabilities and
 
 ## API
 
-See the [API documentation](https://cloudera-labs.github.io/cloudera.cluster/) for details for each plugin and role within the collection. 
+See the [API documentation](https://cloudera-labs.github.io/cloudera.cluster/) for details for each plugin and role within the collection.
 
 ## Roadmap
 
@@ -105,7 +105,7 @@ The collection also requires the following Python libraries to operate its modul
 
 The collection's Python dependencies alone, _not_ the required Python libraries of its collection dependencies, are in `requirements.txt`.
 
-All collection dependencies, required and optional, can be found in `requirements.yml`; only the _required_ dependencies are in `galaxy.yml`. `ansible-galaxy` will install only the _required_ collection dependencies; you will need to add the _optional_ collection dependencies as needed (see above). 
+All collection dependencies, required and optional, can be found in `requirements.yml`; only the _required_ dependencies are in `galaxy.yml`. `ansible-galaxy` will install only the _required_ collection dependencies; you will need to add the _optional_ collection dependencies as needed (see above).
 
 `ansible-builder` can discover and install all Python dependencies - current collection and dependencies - if you wish to use that application to construct your environment. Otherwise, you will need to read each collection and role dependency and follow its installation instructions.
 
@@ -154,7 +154,7 @@ For example, here we use the
 To create a local collection tarball, run:
 
 ```bash
-ansible-galaxy collection build 
+ansible-galaxy collection build
 ```
 
 ## Building the API Documentation

builder/requirements.yml

@@ -23,4 +23,4 @@ roles:
 
   # geerlingguy.mysql with fix for issue #332
   - src: https://github.com/dbeech/ansible-role-mysql
-    version: master
+    version: master

docs/inventories.md

@@ -13,7 +13,7 @@ host-1.example.com
 
 ### Cluster Nodes
 
-A group named `cluster` is **required**. This is the set of nodes which will have a Cloudera Manager agent and receive pre-requisite configurations like OS tuning, database client and JDK installation and Kerberos configs. 
+A group named `cluster` is **required**. This is the set of nodes which will have a Cloudera Manager agent and receive pre-requisite configurations like OS tuning, database client and JDK installation and Kerberos configs.
 
 Usually, `cluster` will be composed of child groups like this:
 
@@ -40,7 +40,7 @@ host-5.example.com
 host-6.example.com
 ```
 
-The names of the cluster sub-groups are arbitrary. These are only for convenience when assigning host template names and to make the inventory easier to read and understand. 
+The names of the cluster sub-groups are arbitrary. These are only for convenience when assigning host template names and to make the inventory easier to read and understand.
 
 #### Assigning Host Templates
 
@@ -114,9 +114,9 @@ cdsw
 
 ### HDFS Encryption (KMS / Key Trustee Server)
 
-Configuring HDFS encryption requires two extra groups `kts_active` and `kms_servers` with a third, optional (but **recommended**) group `kts_passive` to enable Key Trustee Server high availability. 
+Configuring HDFS encryption requires two extra groups `kts_active` and `kms_servers` with a third, optional (but **recommended**) group `kts_passive` to enable Key Trustee Server high availability.
 
-The `kts_active` and `kts_passive` groups must contain a single node each. The KMS group `kms_servers` must have at least one host but can have as many as desired. 
+The `kts_active` and `kts_passive` groups must contain a single node each. The KMS group `kms_servers` must have at least one host but can have as many as desired.
 
 ```ini
 [kms_servers]
@@ -181,10 +181,10 @@ host-10.example.com
 
 ## Multiple Clusters
 
-It is possible to define multiple clusters in the inventory. The key point is that the `cluster` group must contain **all** servers which will be under Cloudera Manager's control, regardless of which cluster they belong to. 
+It is possible to define multiple clusters in the inventory. The key point is that the `cluster` group must contain **all** servers which will be under Cloudera Manager's control, regardless of which cluster they belong to.
 
 
-The inventory group names `cluster1` and `cluster2` are arbitrary. This is just a convenience to make the inventory easier to understand. 
+The inventory group names `cluster1` and `cluster2` are arbitrary. This is just a convenience to make the inventory easier to understand.
 
 ```ini
 [cluster1]
@@ -202,4 +202,4 @@ host-9.example.com  host_template=Cluster2-Worker
 [cluster:children]
 cluster1
 cluster2
-```
+```

docs/security.md

@@ -43,7 +43,7 @@ tls=True
 ca-server-1.example.com
 ```
 
-3b) If you wish to manually sign certificates against an 
+3b) If you wish to manually sign certificates against an
 external CA, like Active Directory, add the path where signed certificates will be stored and root CA certificate details in `defintion.yml`
 
 ```yaml
@@ -103,4 +103,4 @@ kms-3.example.com
 
 If you do not want Key Trustee Server to be highly available, you can omit the `kts_passive` group (this is **not** recommended).
 
-The KMS group `kms_servers` must have at least one host but can have as many as desired. Unlike KTS, KMS nodes are all active and load balanced.
+The KMS group `kms_servers` must have at least one host but can have as many as desired. Unlike KTS, KMS nodes are all active and load balanced.

docs/troubleshooting.md

@@ -12,9 +12,9 @@ or
 fatal: [host.example.com]: UNREACHABLE! => {"changed": false, "msg": "SSH password was requested, but none specified", "unreachable": true}
 ```
 
-**Problem**: 
+**Problem**:
 
-  - Passwordless SSH connectivity is not correctly set up between your Ansible server and the remote servers in the inventory. 
+  - Passwordless SSH connectivity is not correctly set up between your Ansible server and the remote servers in the inventory.
 
 **Solutions**:
 
@@ -32,7 +32,7 @@ fatal: [host.example.com]: UNREACHABLE! => {"changed": false, "msg": "SSH passwo
 fatal: [host.example.com]: UNREACHABLE! => {"changed": false, "msg": "Host key checking is enabled, and SSH reported an unrecognized or mismatching host key.", "unreachable": true}
 ```
 
-**Problem**: 
+**Problem**:
 
   - SSH host key checking is enabled (Ansible default) but the servers you are trying to connect to are not present in `known_hosts` (or their key has changed)
 
@@ -57,38 +57,38 @@ fatal: [host.example.com]: UNREACHABLE! => {"changed": false, "msg": "Host key c
 
 The cause is likely to be one of the following:
 
-1) Inconsistent hostname resolution. The Cloudera Manager is reporting itself with a different hostname to that contained in your inventory file. 
+1) Inconsistent hostname resolution. The Cloudera Manager is reporting itself with a different hostname to that contained in your inventory file.
 
 2) Cloudera Manager agent(s) are not heartbeating correctly.
 
 **Solution**:
 
 By this stage of the playbook execution, Cloudera Manager server will be running. Log into Cloudera Manager and view the **Hosts** page:
 
-- If hosts appear, check the list to ensure that the hostnames shown match your inventory file. If they do not match, either update your inventory file with these hostnames or update the cluster DNS so that the same names can be resolved consistently everywhere. 
+- If hosts appear, check the list to ensure that the hostnames shown match your inventory file. If they do not match, either update your inventory file with these hostnames or update the cluster DNS so that the same names can be resolved consistently everywhere.
 
 - If no hosts appear, log into the server indicated in the error message, and:
 
-  - Check that the `cloudera-manager-agent` service is running. 
+  - Check that the `cloudera-manager-agent` service is running.
 
-  - Check the Cloudera Manager agent log file `/var/log/cloudera-scm-agent/cloudera-scm-agent.log`. Any error message there should give a clue as to why communication with the Cloudera Manager server is failing. 
+  - Check the Cloudera Manager agent log file `/var/log/cloudera-scm-agent/cloudera-scm-agent.log`. Any error message there should give a clue as to why communication with the Cloudera Manager server is failing.
 
 ## Common issue #4
 
 ```
 ERROR! couldn't resolve module/action 'cm_api'. This often indicates a misspelling, missing collection, or incorrect module path.
 ```
 
-**Problem**: 
+**Problem**:
 
 - The `cm_api` action is not available in a custom role.
 
-**Solution**: 
+**Solution**:
 
 - Add the following into the role's `meta/main.yml` file.
 
   ```yaml
   ---
   dependencies:
     - role: cloudera_manager/api_client
-  ```
+  ```

docsbuild/.gitignore

@@ -3,4 +3,4 @@
 
 /temp-rst
 /build
-/rst
+/rst

docsbuild/antsibull-docs.cfg

@@ -47,4 +47,4 @@ logging_cfg = {
       output_name = stderr
     }
   }
-}
+}

docsbuild/cloudera.css

@@ -1,16 +1,15 @@
 /**
  * Copyright 2024 Cloudera, Inc.
- * 
+ *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
  * You may obtain a copy of the License at
- * 
+ *
  *     http://www.apache.org/licenses/LICENSE-2.0
- * 
+ *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-