chore: docs and config maintenance (complytime#105)

* chore: update semgrep version in pre-commit and add linting check Bumps semgrep version to 1.51.0 and adds a local check for `make lint` Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com> * chore: removes warning linters from megalinter configs To ensure warnings/violations are not missed, remove the linter with disable errors and move to enabled. Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com> * chore: adds pre-commit setup in Dev Container configuration Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com> * docs: updates contributing guide with information about development tools Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com> * docs: adds a table of contents to CONTRIBUTING.md Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com> --------- Signed-off-by: Jennifer Power <barnabei.jennifer@gmail.com>
jpower432 · Jan 3, 2024 · ee5cf32 · ee5cf32
1 parent 23f7979
commit ee5cf32
Show file tree

Hide file tree

Showing 5 changed files with 75 additions and 20 deletions.
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -29,7 +29,7 @@
 		"POETRY_VIRTUALENVS_IN_PROJECT": "true"
 	},
 	"updateContentCommand": "python3 -m venv $VENV_PATH && source $VENV_PATH/bin/activate && poetry install --no-root --no-interaction",
-	"postCreateCommand": "echo \"source $VENV_PATH/bin/activate\" >> $HOME/.bashrc",
+	"postCreateCommand": "echo \"source $VENV_PATH/bin/activate\" >> $HOME/.bashrc && make pre-commit",
 	"workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind",
 	"workspaceFolder": "/workspace"
 }
diff --git a/.mega-linter.yml b/.mega-linter.yml
@@ -4,10 +4,6 @@ ENABLE_LINTERS:
   - REPOSITORY_KICS
   - ACTION_ACTIONLINT
   - MARKDOWN_MARKDOWNLINT
-  - MAKEFILE_CHECKMAKE
-
-DISABLE_ERRORS_LINTERS:
-  - MAKEFILE_CHECKMAKE
   - BASH_SHELLCHECK
 
 REPOSITORY_KICS_ARGUMENTS: "--fail-on high"
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
 repos:
   - repo: https://github.com/returntocorp/semgrep
-    rev: v1.42.0
+    rev: v1.51.0
     hooks:
       - id: semgrep
         name: Semgrep Python
@@ -10,4 +10,12 @@ repos:
         name: Semgrep Bandit
         types: [python]
         exclude: "^tests/.+$"
-        args: ["--config", "p/bandit", "--error", --metrics=off]
+        args: ["--config", "p/bandit", "--error", --metrics=off]
+  - repo: local
+    hooks:
+      - id: lint
+        language: script
+        name: Check formatting and lint
+        entry: make lint
+        stages: [commit]
+        types: [python]
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -6,7 +6,22 @@ Before you start contributing, please take a moment to read through the guide be
 
 > WARNING: This project is currently under initial development. APIs may be changed incompatibly from one commit to another.
 
-### How To Contribute
+- [Contributing to trestlebot](#contributing-to-trestlebot)
+  - [How To Contribute](#how-to-contribute)
+  - [Opening a Pull Request](#opening-a-pull-request)
+  - [Developer Guide](#developer-guide)
+  - [Prerequisites](#prerequisites)
+    - [How It Works](#how-it-works)
+      - [Components](#components)
+      - [Code structure](#code-structure)
+    - [Format and Styling](#format-and-styling)
+    - [Update the `actions` files](#update-the-actions-files)
+    - [Running tests](#running-tests)
+    - [Run with poetry](#run-with-poetry)
+    - [Local testing](#local-testing)
+
+
+## How To Contribute
 
 Some initial contributions could be:
 
@@ -29,35 +44,71 @@ When submitting a pull request, please follow these guidelines:
 
 - [Python](https://www.python.org/downloads/) - v3.8+
 - [Poetry](https://python-poetry.org/)
-- [Podman](https://podman.io/docs/installation)
+- [Podman](https://podman.io/docs/installation) (Optional) - For testing locally and end-to-end tests
 
-### How It Works
+## Development Environment
+
+For a reproducible development environment, we use Dev Containers. See [devcontainer.json](./.devcontainer/devcontainer.json) for more information. Note that this does not include the `podman` installation to avoid the requirement for containers with elevated privileges.
+
+## How It Works
 
 For workflow diagrams, see the [diagrams](./docs/diagrams/) under the `docs` folder.
 
-#### Components
+### Components
 
 1. CI Provider - Runs or builds and runs trestle-bot container
 2. Trestle Bot - Provides logic for managing workspace and containerized environment for use in workflows
 3. Compliance-Trestle - Upstream library that provides core logic for how OSCAL content is managed
 
-#### Code structure
+### Code structure
 
 - `actions` - Provides specific logic for `trestle-bot` tasks that are packaged as Actions. See [README.md](./actions/README.md) for more information.
 - `entrypoints` - Provides top level logic for specific user-facing tasks. These tasks are not necessarily related in any way so they are not organized into a hierarchical command structure, but they do inherit logic and flags from a base class.
 - `provider.py, github.py, and gitlab.py` - Git provider abstract class and concrete implementations for interacting with the API.
 - `tasks` - Pre-tasks can be configured before the main git logic is run. Any task that does workspace management should go here.
 - `tasks/authored` - The `authored` package contains logic for managing authoring tasks for single instances of a top-level OSCAL model. These encapsulate logic from the `compliance-trestle` library and allows loose coupling between `tasks` and `authored` types.
-- `transformers` - This contains data transformation logic; specifically for rules. 
+- `transformers` - This contains data transformation logic; specifically for rules.
+
 
-### Format and Styling
+## Format and Styling
+
+This project uses `black` and `isort` for formatting and `flake8` for linting. You can run these commands to format and lint your code.
+They are also run as part as a pre-commit hook.
 
 ```bash
 make format
-make lint
+make lint- [Contributing to trestlebot](#contributing-to-trestlebot)
+    - [How To Contribute](#how-to-contribute)
+  - [Opening a Pull Request](#opening-a-pull-request)
+  - [Developer Guide](#developer-guide)
+  - [Prerequisites](#prerequisites)
+    - [How It Works](#how-it-works)
+      - [Components](#components)
+      - [Code structure](#code-structure)
+    - [Format and Styling](#format-and-styling)
+    - [Update the `actions` files](#update-the-actions-files)
+    - [Running tests](#running-tests)
+    - [Run with poetry](#run-with-poetry)
+    - [Local testing](#local-testing)
+
 ```
 
-### Running tests
+For non-Python files, we use [Megalinter](https://github.com/oxsecurity/megalinter) to lint in a CI task. See [megalinter.yaml](./.github/megalinter.yaml) for more information.
+
+## Type Hints and Static Type Checking
+
+We encourage the use of type hints in Python code to enhance readability, maintainability, and robustness of the codebase. Type hints serve as documentation and aid in catching potential errors during development. For static type analysis, we utilize `mypy`. Running `make lint` will run `mypy` checks on the codebase.
+
+## Analysis Tools
+
+- [SonarCloud](https://sonarcloud.io/dashboard?id=rh-psce_trestle-bot) - We use SonarCloud to analyze code quality, coverage, and security. To not break GitHub security model, this will not run on a forked repository.
+- [Semgrep](https://semgrep.dev/docs/extensions/overview/#pre-commit) - Identify issues in the local development environment before committing code. These checks are also run in CI.
+
+## Running tests
+
+Run all tests with `make test` or `make test-slow` to run all tests including end-to-end.
+For information on end-to-end tests, see [README.md](./tests/e2e/README.md).
+
 ```bash
 # Run all tests
 make test
@@ -69,10 +120,10 @@ make test-e2e
 
 ### Run with poetry
 ```
-poetry shell
-poetry install
+make develop
 poetry run trestlebot-autosync
 poetry run trestlebot-rules-transform
+poetry run trestlebot-create-cd
 ```
 
 ### Local testing

diff --git a/Makefile b/Makefile
@@ -6,7 +6,7 @@ all: develop lint test
 .PHONY: all
 
 develop: pre-commit
-	@poetry install	
+	@poetry install
 	@poetry shell
 .PHONY: develop
 
@@ -44,7 +44,7 @@ test-code-cov:
 .PHONY: test-code-cov
 
 # https://github.com/python-poetry/poetry/issues/994#issuecomment-831598242
-# Check for CVEs locally. For automated/scheduled checks, use dependabot. 
+# Check for CVEs locally. For continuous dependency updates, we use dependabot.
 dep-cve-check:
 	@poetry export -f requirements.txt --without-hashes | poetry run safety check --stdin
 .PHONY: dep-cve-check