Update/v2 migration

CONTRIBUTING.md

@@ -0,0 +1,82 @@
+# Contributing to cloudera-deploy
+
+Thank you for considering contributions to the `cloudera-deploy` project!
+
+# Submitting a pull request
+
+You can start work on issues that are not yet part of a [Milestone](https://github.com/cloudera-labs/cloudera-deploy/milestones) -- anything in our issue tracker that isn't assigned to a Milestone is considered the [backlog](https://github.com/cloudera-labs/cloudera-deploy/issues?q=is%3Aopen+is%3Aissue+no%3Amilestone). 
+
+Before you start working, please announce that you want to do so by commenting on the issue. _([Create an issue](https://github.com/cloudera-labs/cloudera-deploy/issues/new?labels=enhancement) if there isn't one yet, and you can also check out our [Discussions](https://github.com/cloudera-labs/cloudera-deploy/discussions) for ideas.)_ We try to ensure that all active work is assigned to a Milestone in order to keep our backlog accurate.
+
+**When your work is ready for review, create a branch in your own forked repository from the `devel` branch and submit a pull request against `devel`, referencing your the issue.**
+
+As a _best practice_, you can prefix your branches with:
+
+|prefix|Description|Example|
+|------|-----------|-------|
+|`feature/`|A new feature or changes existing to existing code or documentation|`feature/update-some-params`|
+|`fix/`|A non-urgent bug fix|`fix/refactor-some-params`|
+|`hotfix/`|An urgent bug fix|`hotfix/patch-insecure-params`|
+
+> [!NOTE]
+> :fire_extinguisher: A **hotfix** should branch from `main`. It will then be committed to both the `main` and `devel` branches.
+
+# Signing your commits
+
+Note that we require signed commits inline with [Developer Certificate of Origin](https://developercertificate.org/) best-practices for open source collaboration.
+
+A signed commit is a simple one-liner at the end of your commit message that states that you wrote the patch or otherwise have the right to pass the change into open source.  Signing your commits means you agree to:
+
+```
+Developer Certificate of Origin
+Version 1.1
+
+Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
+660 York Street, Suite 102,
+San Francisco, CA 94110 USA
+
+Everyone is permitted to copy and distribute verbatim copies of this
+license document, but changing it is not allowed.
+
+
+Developer's Certificate of Origin 1.1
+
+By making a contribution to this project, I certify that:
+
+(a) The contribution was created in whole or in part by me and I
+    have the right to submit it under the open source license
+    indicated in the file; or
+
+(b) The contribution is based upon previous work that, to the best
+    of my knowledge, is covered under an appropriate open source
+    license and I have the right under that license to submit that
+    work with modifications, whether created in whole or in part
+    by me, under the same open source license (unless I am
+    permitted to submit under a different license), as indicated
+    in the file; or
+
+(c) The contribution was provided directly to me by some other
+    person who certified (a), (b) or (c) and I have not modified
+    it.
+
+(d) I understand and agree that this project and the contribution
+    are public and that a record of the contribution (including all
+    personal information I submit with it, including my sign-off) is
+    maintained indefinitely and may be redistributed consistent with
+    this project or the open source license(s) involved.
+```
+
+(See [developercertificate.org](https://developercertificate.org/))
+
+To agree, make sure to add line at the end of every git commit message, like this:
+
+```
+Signed-off-by: John Doe <jdoe@example.com>
+```
+
+> [!NOTE]
+> :rocket: TIP! Add the sign-off automatically when creating the commit via the `-s` flag, e.g. `git commit -s`.
+
+# Have questions? Opinions? Comments?
+
+Come find us on our [Discussions](https://github.com/cloudera-labs/cloudera-deploy/discussions)!

FAQ.md

@@ -1,48 +1,29 @@
 # Frequently Asked Questions
 
+Be sure to check out the [Discussions > Help](https://github.com/cloudera-labs/cloudera-deploy/discussions/categories/help) category for the latest answers. 
+
 ## Where did everything go?
 
 The project undertook some serious remodeling, but rest assured, your definitions will still work as they did in the previous version of `cloudera-deploy`.  
 
 Okay, but where did everything go? Well...
 
-1. The `quickstart.sh` migrated to `ansible-navigator`. Both of these applications use a container based on `ansible-runner`, i.e. [`cldr-runner`](https://github.com/cloudera-labs/cldr-runner), to execute the playbooks, yet `ansible-navigator` is configuration-driven and better aligned with how AWX runs Ansible in containers. Also, `ansible-navigator` brings a nifty UI and the ease of use to handle different execution modes.
+1. The `quickstart.sh` migrated to `ansible-navigator`. Both of these applications use a container based on `ansible-runner`, i.e. [`cldr-runner`](https://github.com/cloudera-labs/cldr-runner), to execute the playbooks, yet `ansible-navigator` is configuration-driven and better aligned with how AWX runs Ansible in containers. Also, `ansible-navigator` brings a nifty text-based UI (TUI) and the ease of use to handle different execution modes.
 
     We also migrated `cldr-runner` to use `ansible-builder`, but you can read more about that effort at the [`cldr-runner`](https://github.com/cloudera-labs/cldr-runner) project.
 
 1. The original `cloudera-deploy` playbooks moved into `cloudera.exe`. Starting with Ansible `2.11`, [collections can contain playbooks](https://docs.ansible.com/ansible/latest/collections_guide/collections_using_playbooks.html#using-a-playbook-from-a-collection). We call the playbooks using `import_playbook` like roles.
 
-    PLEASE NOTE, if you are developing your own project playbooks, you must first set up your `cloudera-deploy` variables _before_ calling the playbooks by running the `cloudera.exe.init_deployment` role on `localhost`.
+    > [!IMPORTANT]
+    > If you are developing your own project playbooks, you must first set up your `cloudera-deploy` variables _before_ calling the playbooks by running the `cloudera.exe.init_deployment` role.
 
-1. The _run-levels_ still remain; you can still use `-t infra` for example. However, the playbooks themselves are more granular and overall set up and tear down processes are now separate playbooks.
+1. The _runlevels_ still remain; you can still use `-t infra` for example. However, the playbooks themselves are more granular and overall set up and tear down processes are now separate playbooks.
 
     This change promotes composibility and reusability, and we are going to continue to break apart the functions and operations within `cloudera-deploy` and -- most importantly -- the collections that drive this application. We fully expect that you will want to adapt and create your own "deploy" application, one that caters to _your_ needs and operating parameters. Switching to a more granular, more modular approach is key to this objective.
 
-## How to I add _extra variables_ and tags to `ansible-navigator`?
-
-If you want to run a playbook with a given tag, e.g. `-t infra`, then simply add it as a parameter to the `ansible-navigator` commandline. For example, `ansible-navigator run playbook.yml -t infra`. 
-
-Like tags, so you can pass _extra variables_ to `ansible-navigator` and the underlying Ansible command. For example, `ansible-navigator run playbook.yml -e @some_config.yml -e some_var=yes`.
-
-## How do I tell `ansible-navigator` where to find collections and roles?
-
-By default, `cloudera-deploy` expects to use the collections, roles, and libraries within the _execution environment_ container, that is `cldr-runner`. Make sure you do _not_ have `ANSIBLE_COLLECTIONS_PATH` or `ANSIBLE_ROLES_PATH` set or `ansible-navigator` will pick up these environment variables and pass them to the running container. The underlying `ansible` application, like `ansible-playbook` will then pick up these environment variables and attempt to use them if set! This behavior is great if you want to use host-based collections, e.g. local development, but you need to ensure that you update the `ansible-navigator.yml` configuration file to mount the host collection and/or role directories into the execution environment container.
-
-## `ansible-navigator` hangs when I run my playbook. What is going on?
-
-`ansible-navigator` does not handle user prompts when running in the `curses` UI, so actions in your playbook like:
+## How do I run my `cloudera-deploy` V1 playbooks in `ansible-navigator`?
 
-* Vault passwords
-* SSH passphrases
-* Debugger statements
-
-will not work out-of-the-box. You can enable `ansible-navigator` to run with prompts, but doing so will also disable the UI and instead run its operations using `stdout`.  Try adding:
-
-```bash
-ansible-navigator run --enable-prompts ...
-```
-
-to your execution.
+See the [Migration V1](MIGRATION_V1.md) document for details.
 
 ## How can I view a previous `ansible-navigator` run to debug an issue?
 
@@ -54,14 +35,6 @@ ansible-navigator replay runs/<playbook name>-<timestamp>.json
 
 Then you can use the UI to review the plays, tasks, and inventory for the previous run!
 
-## How can I enable the playbook debugger?
-
-The [playbook debugger](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_debugger.html) is enabled in `ansible-navigator` by setting the debugger and then enabling prompts. For example,
-
-```bash
-ANSIBLE_ENABLE_TASK_DEBUGGER=True ansible-navigator run --enable-prompts main.yml
-```
-
 ## How can I select just a single subnet using `subnet_filter`, say for a CDE definition?
 
 The various `filters`, like `subnet_filter`, `loadbalancer_subnets_filter`, etc., use [JMESPath](https://jmespath.org/) expressions against a list of subnet objects. Using expression like:
@@ -114,22 +87,3 @@ You can [test sample filters](https://play.jmespath.org/?u=45e4d839-15f9-4569-94
   }
 ]
 ```
-
-## How to I configure SSH to avoid a "Failed to connect to new control master" error?
-
-When running connecting to a host via SSH while running `ansible-navigator`, in particular when you are working with Terraform inventory managed by the `cloud.terraform` inventory plugin, you might encounter the following error:
-
-```
-Failed to connect to the host via ssh: Control socket connect(/runner/.ansible/cp/b44b170fff): Connection refused
-Failed to connect to new control master
-```
-
-To resolve, be sure to add the following variable to your `ansible-navigator.yml` configuration file:
-
-```yaml
-ansible-navigator:
-  execution-environment:
-    environment-variables:
-      set:
-        ANSIBLE_SSH_CONTROL_PATH: "/dev/shm/cp%%h-%%p-%%r"
-```

MIGRATION_V1.md

@@ -0,0 +1,187 @@
+# Migrating from V1 to V2 of `cloudera-deploy`
+
+## In Summary
+
+1. Don't change your `definition.yml` or `cluster.yml` files.
+2. Create a playbook within your project to run your setup. You can start by referencing the following:
+   * [Public Cloud](public-cloud/aws/datalake/main.yml)
+   * Private Cloud (coming soon!)
+3. Create an `ansible-navigator.yml` configuration in your project. You can start by referencing the following:
+   * [Public Cloud](public-cloud/aws/datalake/ansible-navigator.yml)
+   * Private Cloud (coming soon!)
+4. Run your playbook by using `ansible-navigator` vs. `ansible-playbook`.
+   * All other arguments apply, so continue to use `-e` and `-t` as needed, e.g. `ansible-navigator run your_playbook.yml -e key=value -t infra,plat,another_tag`
+
+## In Detail
+
+So, you may ask yourself, "How do I run my `cloudera-deploy` V1 playbooks in `ansible-navigator`?" <cue [The Talking Heads](https://youtu.be/5IsSpAOD6K8?si=K4vEs-b3kvZimM5X&t=49)>
+
+Previously, you would execute the `quickstart.sh` script to bootstrap the `cldr-runner` image into a shell and then run your scripts _from the container shell_, e.g. `ansible-playbook /opt/cloudera-deploy/main.yml -e "definition_path=examples/sandbox" -t run,default_cluster -vvv`. While this mode is still certainly possible, the introduction of `ansible-navigator` simplifies these action.
+
+**The most significant change**: the legacy definitions only contain configuration files -- the `definition.yml`, `cluster.yml`, `application.yml`, and `inventory_*` files -- and the legacy `cloudera-deploy` has local playbooks that orchestrated the whole run by calling a "sequence" role in `cloudera.exe`... No longer!
+
+So, what to do? First off...
+
+**Your existing platform configurations -- `definition.yml` and `cluster.yml`, specifically -- remain as they are. No changes are needed.**
+
+What does need to change?
+
+**You need to provide an entrypoint playbook.**
+
+Your project now needs a playbook, ala `main.yml`, to coordinate execution. This change allows for considerable flexibility as to how and when infrastructure and platform runlevels execute - frankly, how and when _any_ tasks, runlevel or otherwise, are run. 
+
+In short, we have moved the responsiblity of managing key sections of the "runlevel" from the `cloudera_deploy` _application_ to the project _itself_. This allows you, on a per-project basis, to define _exactly_ what you want, when you need it. Yet, you still can call on the common, shared order-of-operations for installing Cloudera Manager or spinning up a CDP Public Cloud Datalake that the legacy `cloudera-deploy` once had, rather forced you to have. A simple `ansible.builtin.import_playbook` pragma will include these _collection playbooks_ from the updated `cloudera.exe` collection.
+
+Here is an example. The previous `main.yml` file eventually calls the `cloudera.exe.sequence` role, which in turn calls the _runlevel_ roles.
+
+```yaml
+# cloudera.exe.sequence/tasks/main.yml
+
+- name: Validate Infrastructure Configuration
+  ansible.builtin.include_role:
+    name: cloudera.exe.infrastructure
+    tasks_from: validate
+    # Truncated for clarity
+
+- name: Validate Platform Configuration
+  ansible.builtin.include_role:
+    name: cloudera.exe.platform
+    tasks_from: validate
+    # Truncated for clarity
+
+- name: Validate Runtime Configuration
+  ansible.builtin.include_role:
+    name: cloudera.exe.runtime
+    tasks_from: validate
+    # Truncated for clarity
+```
+
+([See this file in its entirety.](https://github.com/cloudera-labs/cloudera.exe/blob/v1.7.5/roles/sequence/tasks/main.yml))
+
+The _v2.x_ of `cloudera.exe` (and via proxy, `cloudera-deploy`) moves this code from the role _into_ a playbook within `cloudera.exe`.
+
+Here is a _v2.x_ entrypoint playbook. It assumes that you want to handle infrastructure - say, for a sandbox install - as well as the CDP Public Cloud setup. (There is an explicit playbook to teardown.)
+
+```yaml
+# cloudera-deploy/public-cloud/aws/datalake/main.yml
+
+- name: Set up the cloudera-deploy variables
+  hosts: localhost
+  connection: local
+  gather_facts: yes
+  tasks:
+    - name: Read definition variables
+      ansible.builtin.include_role:
+        name: cloudera.exe.init_deployment
+        public: yes
+      when: init__completed is undefined
+  tags:
+    - always
+
+- name: Set up CDP Public Cloud infrastructure (Ansible-based)
+  ansible.builtin.import_playbook: cloudera.exe.pbc_infra_setup.yml
+
+- name: Set up CDP Public Cloud (Env and DL example)
+  ansible.builtin.import_playbook: cloudera.exe.pbc_setup.yml
+```
+
+And the new `cloudera.exe` playbooks?
+
+```yaml
+# cloudera.exe/playbooks/pbc_infra_setup.yml
+
+- name: Set up CDP Public Cloud infrastructure (Ansible-based)
+  hosts: "{{ target | default('localhost') }}"
+  environment: "{{ globals.env_vars }}"
+  gather_facts: yes
+  tasks:
+    - name: Validate CDP Public Cloud infrastructure configuration
+      ansible.builtin.import_role:
+        name: cloudera.exe.infrastructure
+        tasks_from: validate
+      tags:
+        - validate
+        - initialize
+        - infra
+
+    - name: Initialize CDP Public Cloud infrastructure setup
+      ansible.builtin.import_role:
+        name: cloudera.exe.infrastructure
+        tasks_from: initialize_setup
+      tags:
+        - initialize
+        - infra
+
+    - name: Set up CDP Public Cloud infrastructure
+      ansible.builtin.import_role:
+        name: cloudera.exe.infrastructure
+        tasks_from: setup
+      tags:
+        - infra
+```
+
+```yaml
+# cloudera.exe/playbooks/pbc_setup.yml
+
+- name: Set up CDP Public Cloud
+  hosts: "{{ target | default('localhost') }}"
+  environment: "{{ globals.env_vars }}"
+  gather_facts: yes
+  tasks:
+    - name: Validate Platform configuration
+      ansible.builtin.import_role:
+        name: cloudera.exe.platform
+        tasks_from: validate
+      tags:
+        - validate
+        - initialize
+        - plat
+        - run
+
+    - name: Validate Data Services configuration
+      ansible.builtin.import_role:
+        name: cloudera.exe.runtime
+        tasks_from: validate
+      tags:
+        - validate
+        - initialize
+        - run
+
+    - name: Initialize Platform setup
+      ansible.builtin.import_role:
+        name: cloudera.exe.platform
+        tasks_from: initialize_setup
+      tags:
+        - initialize
+        - plat
+        - run
+
+    - name: Set up Platform
+      ansible.builtin.import_role:
+        name: cloudera.exe.platform
+        tasks_from: setup
+      tags:
+        - plat
+        - run
+
+    - name: Initialize Data Services setup
+      ansible.builtin.import_role:
+        name: cloudera.exe.runtime
+        tasks_from: initialize_setup
+      tags:
+        - initialize
+        - run
+
+    - name: Set up Data Services
+      ansible.builtin.import_role:
+        name: cloudera.exe.runtime
+        tasks_from: setup
+      tags:
+        - run
+```
+
+You can see that instead of calling the role and passing Ansible tags, you call the playbook, which now has _the very same code_ but without the need for some of the tags or the intermediate role, `cloudera.exe.sequence`. In fact, the playbooks in `cloudera.exe` have become the `cloudera.exe.sequence` role.
+
+You don't want to use the infrastructure playbook because you have your own process for establishing infrastructure? Great! Remove the `import_playbook` and call whatever is necessary! So long as you have run `cloudera.exe.init_deployment` in your project's playbook(s) _prior_ to importing any of the _collection playbooks_, you can use the collection playbooks anytime in your project playbooks.
+
+Need to discuss this further? Stop by the [Discussions > Help](https://github.com/cloudera-labs/cloudera.exe/discussions/categories/help)!