feat: adds markdown generation to the rules transform entrypoint (com…

…plytime#282)

* feat: adds markdown generation to the rules transform entrypoint

BREAKING CHANGE: Modifies the existing behavior of the rules transform
entrypoint

Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

* fix: adds required flag to rules-transform entrypoint

Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

* feat: adds traceback string to handle exception

Logging the traceback string in debug mode helps
with development without displaying to users by default.

Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

* test: adds basic testing for rules transformation CLI

Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

* test(rules-transform): adds assertions to check md directory creation

Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

* docs: updates actions docs to reflect workflow changes

Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

---------

Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>

actions/README.md

@@ -25,7 +25,8 @@ For more details, consult the [GitHub Actions documentation](https://docs.github
 
 ## Examples
 
-Here are examples of workflow snippets that demonstrate how to use these actions in the `trestle-bot` project.
+Here are examples of workflow snippets that demonstrate how `trestle-bot` actions can be used for authoring. For the examples below, the OSCAL Component Definition authoring workflow is explored.
+
 See each action README for more details about the inputs and outputs.
 
 ### Create a New Component Definition
@@ -57,64 +58,120 @@ jobs:
           github_token: ${{ secrets.GITHUB_TOKEN }}
 ```
 
-## Transform Rules
+## Transform and Autosync
 
-```yaml
-name: transform
+Review human-friendly formats in code reviews and apply OSCAL JSON changes after PR merge.
+The `autosync` action can be used with any supported model for authoring with Trestle. The 
+`transform` action is only supported for component definitions.
 
+```yaml
+name: Push to main
 on:
   push:
-    branches-ignore:
-    - main
-  paths:
-    - 'rules/**'
+    branches:
+      - main
+    paths:
+      - 'profiles/**'
+      - 'catalogs/**'
+      - 'component-definitions/**'
+      - 'md_comp/**'
+      - 'rules/**'
+
+concurrency:
+  group: ${{ github.ref }}-${{ github.workflow }}
+  cancel-in-progress: true
 
 jobs:
-  transform-rules:
-    name: Transform rules content
+  transform-and-sync:
+    name: Automatically Sync Content
     runs-on: ubuntu-latest
-    permissions:
-      content: write
     steps:
-      - uses: actions/checkout@v4
-      - uses: RedHatProductSecurity/trestle-bot/actions/rules-transform@main
-
+      - name: Clone
+        uses: actions/checkout@v4
+
+      # Update JSON with any markdown edits. Markdown will also be regenerated to
+      # follow the generate-edit-assemble workflow. At this stage, we are on the
+      # edit step. So autosync runs assemble then generate.
+      - name: AutoSync
+        id: autosync
+        uses: RedHatProductSecurity/trestle-bot/actions/autosync@main
+        with:
+          markdown_path: "md_comp"
+          oscal_model: "compdef"
+          commit_message: "Autosync component definition content [skip ci]"
+      # Rule transformation is not idempotent, so you may only want to run this
+      # if your rules directly has changes to avoid UUID regeneration.
+      - uses: dorny/paths-filter@v3
+        id: changes
+        with:
+          filters: |
+            rules:
+              - 'rules/**'
+      # Transformation of rules will updates the OSCAL JSON. These changes will 
+      # then be propagated to Markdown. Transformation and regeneration are run together
+      # to ensure the Markdown has the most up to date rule information.
+      - name: Transform
+        if: steps.changes.outputs.rules == 'true'
+        id: transform
+        uses: RedHatProductSecurity/trestle-bot/actions/rules-transform@main
+        with:
+          markdown_path: "md_comp"
+          commit_message: "Auto-transform rules [skip ci]" 
 ```
 
-## Autosync Markdown and JSON
+## Verify Changes
 
-```yaml
-name: autosync
+Run actions in dry run mode on pull requests to ensure content
+can be transformed and assembled on merge without errors.
 
+```yaml
+name: Validate PR with CI
 on:
-  pull-request:
+  pull_request:
     branches:
-      - 'main'
-  paths:
-    - 'component-definitions/**'
-    - 'markdown/components/**'
+      - main
+    paths:
+      - 'profiles/**'
+      - 'catalogs/**'
+      - 'component-definitions/**'
+      - 'md_comp/**'
+      - 'rules/**'
 
 jobs:
-  autosync:
-    name: Sync Markdown and JSON
+  transform-and-regenerate:
+    name: Rules Transform and Content Syncing
     runs-on: ubuntu-latest
-    permissions:
-      contents: write
     steps:
-      - uses: actions/checkout@v4
-      - uses: RedHatProductSecurity/trestle-bot/actions/autosync@main
+      - name: Clone
+        uses: actions/checkout@v4
+      - name: AutoSync
+        id: autosync
+        uses: RedHatProductSecurity/trestle-bot/actions/autosync@main
         with:
-          markdown_path: "markdown/components"
+          markdown_path: "md_comp"
           oscal_model: "compdef"
-          branch: ${{ github.head_ref }}
+          dry_run: true
+      - uses: dorny/paths-filter@v3
+        id: changes
+        with:
+          filters: |
+            rules:
+              - 'rules/**'
+      - name: Transform
+        if: steps.changes.outputs.rules == 'true'
+        id: transform
+        uses: RedHatProductSecurity/trestle-bot/actions/rules-transform@main
+        with:
+          markdown_path: "md_comp"
+          dry_run: true
 ```
 
 ## Propagate changes from upstream sources
 
-### Storing and syncing upstream content
-
 > Note: The upstream repo must be a valid trestle workspace.
 
+This example demonstrates how to use outputs by labeling pull requests.
+
 ```yaml
 name: Sync Upstream
 
@@ -131,55 +188,81 @@ jobs:
       pull-requests: write
     steps:
     - uses: actions/checkout@v4
-    - name: Run trestlebot
+    - name: Sync content
       id: trestlebot
       uses: RedHatProductSecurity/trestle-bot/actions/sync-upstreams@main
       with:
         branch: "sync-upstream-${{ github.run_id }}"
+        # We set the target branch here to create a pull request
+        # for review
         target_branch: "main"
         github_token: ${{ secrets.GITHUB_TOKEN }}
         sources: |
           https://github.com/myorg/myprofiles@main
+    - uses: actions/labeler@v4
+      if: steps.trestlebot.outputs.changes == 'true'
+      with:   
+        pr-number: |
+            ${{ steps.trestlebot.outputs.pr_number }} 
+    # Regenerate Markdown for an easier to control diff review and
+    # to understand change impact.
+    - name: Regenerate markdown (optionally)
+      if: steps.trestlebot.outputs.changes == 'true'
+      uses: RedHatProductSecurity/trestle-bot/actions/autosync@main
+      with:
+        markdown_path: "markdown/components"
+        oscal_model: "compdef"
+        branch: "sync-upstream-${{ github.run_id }}"
+        skip_assemble: true
+        github_token: ${{ secrets.GITHUB_TOKEN }}
 ```
 
-### Component Definition Regeneration
 
-This example demonstrates how to use outputs and also includes labeling pull requests.
+## Release
+
+Below is an example release workflow using the `version` on the `autosync` action set the
+component definition version in the OSCAL metadata. 
 
 ```yaml
-name: Regenerate Markdown
+name: Release
 
 on:
-  push:
-    branches:
-      - 'main'
-  paths:
-    paths:
-      - 'profiles/**'
-      - 'catalogs/**'
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Release version'
+        required: true
 
 jobs:
-  regenerate-content:
-    name: Regenerate the component definition
+  release:
     runs-on: ubuntu-latest
     permissions:
       contents: write
-      pull-requests: write
     steps:
-    - uses: actions/checkout@v4
-    - name: Run trestlebot
-      id: trestlebot
-      uses: RedHatProductSecurity/trestle-bot/actions/autosync@main
-      with:
-        markdown_path: "markdown/components"
-        oscal_model: "compdef"
-        branch: "autoupdate-${{ github.run_id }}"
-        target_branch: "main"
-        skip_assemble: true
-        github_token: ${{ secrets.GITHUB_TOKEN }}
-    - uses: actions/labeler@v4
-      if: steps.trestlebot.outputs.changes == 'true'
-      with:   
-        pr-number: |
-            ${{ steps.trestlebot.outputs.pr_number }} 
+      - name: Clone
+        uses: actions/checkout@v4
+      - name: Autosync
+        uses: RedHatProductSecurity/trestle-bot/actions/autosync@main
+        with:
+          markdown_path: "md_comp"
+          oscal_model: "compdef"
+          commit_message: "Update content for release [skip ci]"
+          version: ${{ github.event.inputs.version }}
+      - name: Create and push tags
+        env:
+          VERSION: ${{ github.event.inputs.version }}
+        run: |
+            git tag "${VERSION}"
+            git push origin "${VERSION}"
+      - name: Create Release
+        uses: actions/github-script@v7
+        with:
+          script: |
+            await github.rest.repos.createRelease({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              tag_name: '${{ github.event.inputs.version }}',
+              name: 'Release v${{ github.event.inputs.version }}',
+              generate_release_notes: true,
+            })
 ```

actions/rules-transform/README.md

@@ -13,6 +13,9 @@ name: Example Workflow
       - name: Run trestlebot
         id: trestlebot
         uses: RedHatProductSecurity/trestle-bot/actions/rules-transform@main
+        with:
+          markdown_path: "markdown/components"
+
 ```
 
 With custom rules directory:
@@ -24,6 +27,7 @@ With custom rules directory:
       id: trestlebot
       uses: RedHatProductSecurity/trestle-bot/actions/rules-transform@main
       with:
+        markdown_path: "markdown/components"
         rules_view_path: "custom-rules-dir/"
   ```
 
@@ -32,6 +36,7 @@ With custom rules directory:
 <!-- START_ACTION_INPUTS -->
 | Name | Description | Default | Required |
 | --- | --- | --- | --- |
+| markdown_path | Path relative to the repository path to create markdown files. See action README.md for more information. | None | True |
 | rules_view_path | Path relative to the repository path where the Trestle rules view files are located. Defaults to `rules/`. | rules/ | False |
 | dry_run | Runs tasks without pushing changes to the repository. | false | False |
 | github_token | "GitHub token used to make authenticated API requests. Note: You should use a defined secret like "secrets.GITHUB_TOKEN" in your workflow file, do not hardcode the token." | None | False |
@@ -63,7 +68,7 @@ With custom rules directory:
 
 ## Action Behavior
 
-The purpose of this action is to sync the rules view data in YAML to OSCAL with `compliance-trestle` and commit changes back to the branch or submit a pull request (if desired). Below are the main use-cases/workflows available:
+The purpose of this action is to sync the rules view data in YAML to OSCAL with `compliance-trestle` and generation corresponding Markdown and commit changes back to the branch or submit a pull request (if desired). Below are the main use-cases/workflows available:
 
 - The default behavior of this action is to run the rules transformation and commit the changes back to the branch the workflow ran from ( `github.ref_name` ). The branch can be changed by setting the field `branch`. If no changes exist or the changes do not exist with the file pattern set, no changes will be made and the action will exit successfully.
 

actions/rules-transform/action.yml

@@ -1,8 +1,11 @@
 name: "trestle-bot-rules-transform"
 author: "Red Hat Product Security"
-description: "A rules transform action to convert trestle rules in YAML format to OSCAL"
+description: "A rules transform action to convert trestle rules in YAML format to OSCAL and propagates changes to Markdown."
 
 inputs:
+  markdown_path: 
+    description: Path relative to the repository path to create markdown files. See action README.md for more information.
+    required: true
   rules_view_path: 
     description: Path relative to the repository path where the Trestle rules view files are located. Defaults to `rules/`.
     required: false

actions/rules-transform/rules-transform-entrypoint.sh

@@ -9,6 +9,7 @@ set_git_safe_directory
 
 # Initialize the command variable
 command="trestlebot-rules-transform \
+        --markdown-path=\"${INPUT_MARKDOWN_PATH}\" \
         --rules-view-path=\"${INPUT_RULES_VIEW_PATH}\" \
         --commit-message=\"${INPUT_COMMIT_MESSAGE}\" \
         --pull-request-title=\"${INPUT_PULL_REQUEST_TITLE}\" \

tests/data/yaml/test_complete_rule.yaml

@@ -16,7 +16,7 @@ x-trestle-rule-info:
     description: My check description
   profile:
     description: Simple NIST Profile
-    href: profiles/simplified_nist_profile/profile.json
+    href: trestle://profiles/simplified_nist_profile/profile.json
     include-controls:
       - id: ac-1
 x-trestle-component-info:

tests/data/yaml/test_complete_rule_multiple_controls.yaml

@@ -15,7 +15,7 @@ x-trestle-rule-info:
     description: My check description
   profile:
     description: Simple NIST Profile
-    href: profiles/simplified_nist_profile/profile.json
+    href: trestle://profiles/simplified_nist_profile/profile.json
     include-controls:
       - id: ac-1
       - id: ac-1_smt.a

tests/data/yaml/test_complete_rule_no_params.yaml

@@ -3,7 +3,7 @@ x-trestle-rule-info:
   description: My rule description for example rule 2
   profile:
     description: Simple NIST Profile
-    href: profiles/simplified_nist_profile/profile.json
+    href: trestle://profiles/simplified_nist_profile/profile.json
     include-controls:
       - id: ac-1
 x-trestle-component-info:

tests/e2e/test_e2e_compdef.py

@@ -36,6 +36,7 @@
             "success/happy path",
             {
                 "branch": "test",
+                "markdown-path": "md_comp",
                 "rules-view-path": RULES_VIEW_DIR,
                 "committer-name": "test",
                 "committer-email": "test@email.com",
@@ -46,6 +47,7 @@
             {
                 "branch": "test",
                 "rules-view-path": RULES_VIEW_DIR,
+                "markdown-path": "md_comp",
                 "committer-name": "test",
                 "committer-email": "test",
                 "skip-items": test_comp_name,
@@ -82,6 +84,7 @@ def test_rules_transform_e2e(
             tmp_repo_path, test_comp_name, ComponentDefinition, FileContentType.JSON
         )
         assert comp_path.exists()
+        assert tmp_repo_path.joinpath("md_comp").exists()
         assert f"input: {test_comp_name}.csv" in response_stdout
     branch = command_args["branch"]
     assert f"Changes pushed to {branch} successfully." in response_stdout