Incorporating user documentation feedback and simplifies the `Makefil…

…e` (#227)

* Ignores another pytest deprecation

* Simplifications and deprecations for the Makefile. The preferred method for installing CRM is through conda/conda-forge

* Overhauls existing README documentation

* Updates README TOC

* More README doc work

* More README improvements

* Filling more doc gaps

* Simplifies and automates the API documentaiton process

* Update README.md

Co-authored-by: Mahe Iram Khan <65779580+ForgottenProgramme@users.noreply.github.com>

* Update README.md

Co-authored-by: Mahe Iram Khan <65779580+ForgottenProgramme@users.noreply.github.com>

* Update README.md

Co-authored-by: Mahe Iram Khan <65779580+ForgottenProgramme@users.noreply.github.com>

* Adds a contribution file

* Update CONTRIBUTING.md

---------

Co-authored-by: Mahe Iram Khan <65779580+ForgottenProgramme@users.noreply.github.com>

.gitignore

@@ -70,6 +70,7 @@ instance/
 
 # Sphinx documentation
 docs/_build/
+docs/_autosummary/
 
 # PyBuilder
 .pybuilder/

.pre-commit-config.yaml

@@ -11,7 +11,7 @@ repos:
   rev: v4.6.0
   hooks:
   - id: check-added-large-files
-    exclude_types: ["json", "yaml", "pdf", "html"]
+    exclude_types: ["json", "yaml", "pdf", "html", "png"]
   - id: check-ast
   - id: fix-byte-order-marker
   - id: check-case-conflict

.pytest.ini

@@ -2,4 +2,5 @@
 [pytest]
 filterwarnings =
     ignore::DeprecationWarning:boltons.*
+    ignore::DeprecationWarning:xdist.*
 addopts = --ignore=tests/test_aux_files

CONTRIBUTING.md

@@ -0,0 +1,31 @@
+# Contributing
+Before contributing to this project, please check to see if a similar or identical GitHub issue is already open.
+If no such issue applies, please file one so we may openly discuss the requested changes. This process aims to prevent
+duplicated work and to reduce wasted time on work that may already be underway.
+
+We welcome all contribution types. However, for code contributions, please follow the guide below to ensure your
+contributions follow our expectations for engineering excellence.
+
+The maintainers of this project manage a project board
+[here](https://github.com/orgs/conda-incubator/projects/11/views/1?groupedBy%5BcolumnId%5D=Milestone). When an issue is
+filed, we aim to quickly organize it into this board.
+
+Please note we follow the `conda` organization's
+Code of Conduct](https://github.com/conda/governance/blob/main/CODE_OF_CONDUCT.md).
+
+## Code PR Requirements
+The following is a list of requirements for code contributions. Most rules (if not all) are enforced with automated
+code checks.
+- A PR that does not pass our CI workflows, will not be accepted. All automated code checks can be run locally.
+    - Our development environment (created with `make dev`) will automatically install and configure `pre-commit` to
+      enforce our code checks.
+- Unless otherwise approved/discussed, contributions must be done in Python with a minimum target version of 3.11
+- Code must be automatically formatted per our rule set (enforced by `isort` and `black`)
+- Code must comply to our linting rule set (enforced by `pylint`)
+- Code must comply to our static analyzer rule set (enforced by `mypy`)
+- Code must be documented. Modules, functions, classes, and files must use
+  [reST](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) formatting so that our automated API
+  documentation system can publish API docs for our users.
+  - Please aim to capture the "why" of a problem in in-code comments, especially if there is a lot of nuance.
+- If there is work that needs to be completed at a latter date, file a new issue, tag it to the existing, and leave a
+  `TODO` comment (where appropriate) explaining the situation.

Makefile

@@ -1,5 +1,4 @@
-# The `.ONESHELL` and setting `SHELL` allows us to run commands that require
-# `conda activate`
+# The `.ONESHELL` and setting `SHELL` allows us to run commands that require `conda activate`
 .ONESHELL:
 SHELL := /bin/bash
 # For GNU Make v4 and above, you must include the `-c` in order for `make` to find symbols from `PATH`
@@ -8,14 +7,15 @@ CONDA_ACTIVATE = source $$(conda info --base)/etc/profile.d/conda.sh ; conda act
 # Ensure that we are using the python interpretter provided by the conda environment.
 PYTHON3 := "$(CONDA_PREFIX)/bin/python3"
 
-.PHONY: clean clean-env clean-test clean-pyc clean-build clean-other help dev test test-debug test-cov pre-commit lint format format-docs analyze docs
+.PHONY: clean clean-build clean-env clean-test help dev pre-commit test test-cov lint format format-docs analyze docs
 .DEFAULT_GOAL := help
 
 CONDA_ENV_NAME ?= conda-recipe-manager
 SRC_DIR = conda_recipe_manager
 TEST_DIR = tests/
 SCRIPTS_DIR = scripts/
 
+# Auto-generated `make help` directives defined in this `Makefile`.
 define BROWSER_PYSCRIPT
 import os, webbrowser, sys
 
@@ -38,51 +38,40 @@ export PRINT_HELP_PYSCRIPT
 
 BROWSER := python -c "$$BROWSER_PYSCRIPT"
 
-clean: clean-cov clean-build clean-env clean-pyc clean-test clean-other ## remove all build, test, coverage and Python artifacts
+clean: clean-test clean-build clean-env clean-docs ## remove all build, test, coverage, and Python artifacts
 
-clean-cov:
-	rm -rf .coverage
-	rm -rf htmlcov
-	rm -rf reports/{*.html,*.png,*.js,*.css,*.json}
-	rm -rf pytest.xml
-	rm -rf pytest-coverage.txt
-
-clean-build: ## remove build artifacts
+clean-build: ## remove build and Python artifacts
 	rm -fr build/
 	rm -fr dist/
 	rm -fr .eggs/
 	find . -name '*.egg-info' -exec rm -fr {} +
 	find . -name '*.egg' -exec rm -f {} +
-
-clean-env:					## remove conda environment
-	conda remove -y -n $(CONDA_ENV_NAME) --all
-
-clean-pyc: ## remove Python file artifacts
 	find . -name '*.pyc' -exec rm -f {} +
 	find . -name '*.pyo' -exec rm -f {} +
 	find . -name '*~' -exec rm -f {} +
 	find . -name '__pycache__' -exec rm -fr {} +
 
-clean-test: ## remove test and coverage artifacts
+clean-env:					## remove conda environment
+	conda remove -y -n $(CONDA_ENV_NAME) --all
+
+clean-test: ## remove test, coverage, and profiler artifacts
 	rm -fr .tox/
 	rm -f .coverage
 	rm -fr htmlcov/
 	rm -fr .pytest_cache
-
-clean-other:
+	rm -rf reports/{*.html,*.png,*.js,*.css,*.json}
+	rm -rf pytest.xml
+	rm -rf pytest-coverage.txt
 	rm -fr *.prof
 	rm -fr profile.html profile.json
 
+clean-docs:	## Removes temporary and auto-generated static doc files
+	rm -rf docs/_autosummary
+	rm -rf docs/_build
+
 help:
 	$(PYTHON3) -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
 
-install: clean	## install the package to the active Python's site-packages
-	pip install .
-
-environment:    ## handles environment creation
-	conda env create -f environment.yaml --name $(CONDA_ENV_NAME) --yes
-	conda run --name $(CONDA_ENV_NAME) pip install .
-
 dev: clean		## install the package's development version to a fresh environment
 	conda env create -f environment.yaml --name $(CONDA_ENV_NAME) --yes
 	conda run --name $(CONDA_ENV_NAME) pip install -e .
@@ -94,9 +83,6 @@ pre-commit:     ## runs pre-commit against files. NOTE: older files are disabled
 test:			## runs test cases
 	$(PYTHON3) -m pytest -n auto --capture=no $(TEST_DIR)
 
-test-debug:		## runs test cases with debugging info enabled
-	$(PYTHON3) -m pytest -n auto -vv --capture=no $(TEST_DIR)
-
 test-cov:		## checks test coverage requirements
 	$(PYTHON3) -m pytest -n auto --cov-config=.coveragerc --cov=$(SRC_DIR) \
 		$(TEST_DIR) --cov-fail-under=80 --cov-report term-missing
@@ -108,11 +94,11 @@ format:			## runs the code auto-formatter
 	isort --profile black --line-length=120 $(SRC_DIR) $(TEST_DIR) $(SCRIPTS_DIR)
 	black --line-length=120 $(SRC_DIR) $(TEST_DIR) $(SCRIPTS_DIR)
 
-format-docs:	## runs the docstring auto-formatter. Note this requires manually installing `docconvert` with `pip`
+format-docs:	## runs the docstring auto-formatter. NOTE: this requires manually installing `docconvert` with `pip`
 	docconvert --in-place --config .docconvert.json $(SRC_DIR)
 
 analyze:		## runs static analyzer on the project
 	mypy --config-file=.mypy.ini --cache-dir=/dev/null $(SRC_DIR) $(TEST_DIR) $(SCRIPTS_DIR)
 
-docs:
+docs: clean-docs	## generates static html documentation
 	$(MAKE) -C docs html

README.md

@@ -3,84 +3,137 @@
 [release-badge]: https://img.shields.io/github/v/release/conda-incubator/conda-recipe-manager?logo=github
 
 
-# conda-recipe-manager
+# `conda-recipe-manager`
 
 [![Commit Checks][commit-checks-badge]](https://github.com/conda-incubator/conda-recipe-manager/actions/workflows/commit_checks.yaml)
 [![Integration Tests][integration-tests-badge]](https://github.com/conda-incubator/conda-recipe-manager/actions/workflows/integration_tests.yaml)
 [![GitHub Release][release-badge]](https://github.com/conda-incubator/conda-recipe-manager/releases)
 
-## Table of Contents
+# Overview
+Conda Recipe Manager (CRM) is a library and tool-set capable of managing `conda` recipe files. It is intended
+to be used by package builders and developers to automate the generation and editing of `conda` recipe files.
+
+The most mature portion of this project is the `parser` module, that allows developers to parse, render, and edit
+existing recipe files. There is also some on-going work for parsing recipe selectors and Conda Build Config files.
+
+For a more comprehensive break-down and status of the library modules, see
+[this document](./conda_recipe_manager/README.md).
+
+## Recipe Compatibility
+The latest recipe-parsing compatibility statistics can be found in the summary of our automated
+[Integration Tests](https://github.com/conda-incubator/conda-recipe-manager/actions).
+
+Our integration test data set is available [here](https://github.com/conda-incubator/conda-recipe-manager-test-data)
+and is based off of publicly available recipe files from various sources.
+
+NOTE: CRM only officially supports recipe files in the V0. There is on-going work to add full support for editing
+V1-formatted files.
+
 <!-- TOC -->
 
 - [conda-recipe-manager](#conda-recipe-manager)
-    - [Table of Contents](#table-of-contents)
 - [Overview](#overview)
+    - [Recipe Compatibility](#recipe-compatibility)
     - [History](#history)
 - [Getting Started](#getting-started)
     - [General Installation](#general-installation)
-        - [Install into your current environment](#install-into-your-current-environment)
-        - [Install into a custom environment](#install-into-a-custom-environment)
-- [Developer Notes](#developer-notes)
-        - [Running pre-commit checks](#running-pre-commit-checks)
-    - [Release process](#release-process)
+    - [CLI Usage](#cli-usage)
+    - [Developer Installation and Notes](#developer-installation-and-notes)
+        - [Developer Documentation](#developer-documentation)
+        - [Setup Troubleshooting](#setup-troubleshooting)
+        - [Making Commits](#making-commits)
+        - [Running pre-commit Checks Individually](#running-pre-commit-checks-individually)
+        - [Release process](#release-process)
 - [Special Thanks](#special-thanks)
 
 <!-- /TOC -->
-# Overview
-Conda Recipe Manager (CRM) is a library and tool set capable of parsing Conda recipe files. It is intended to be
-used by package builders and developers to automate the generation and editing of Conda recipe files.
-
-Currently only recipe files in the V0 format are supported, but there is some on-going work to add full support for
-V1-formatted files.
-
-Library documentation can be found [here](https://conda-incubator.github.io/conda-recipe-manager/index.html).
 
 ## History
 This project started out as a recipe parsing library in Anaconda's
-[percy](https://github.com/anaconda-distribution/percy) project.
+[percy](https://github.com/anaconda-distribution/percy) project. Some `git` history was lost during that transfer
+process.
+
+For those of you who come from `conda-forge`, you may associate CRM as "the tool that converts recipe files for
+`rattler-build`". Admittedly, that was the first-use case of the parsing capabilities provided by in this library. In
+the future, we aim to expand past that and offer a number of recipe automation tools and modules.
 
 # Getting Started
 
 ## General Installation
 
-### Install into your current environment
+To install the project to your current `conda` environment, run:
 ```sh
-make install
+conda install -c conda-forge conda-recipe-manager
 ```
+This will add the commands `conda-recipe-manager` and `crm` to your environment's path. Note that both of these
+commands are the same. `crm` is provided for convenience of typing.
+
+## CLI Usage
+Although CRM is a library, it does ship with a handful of command line tools. Running `crm --help` will provide a
+an up-to-date listing of all available tools. Run `crm <tool-name> --help` for usage documentation about each tool.
 
-### Install into a custom environment
+The following usage message was last updated on 2024-10-31:
 ```sh
-make environment
-conda activate conda-recipe-manager
+Usage: crm [OPTIONS] COMMAND [ARGS]...
+
+  Command line interface for conda recipe management commands.
+
+Options:
+  --help  Show this message and exit.
+
+Commands:
+  convert             Converts a `meta.yaml` formatted-recipe file to the new
+                      `recipe.yaml` format.
+  graph               Interactive CLI for examining recipe dependency graphs.
+  patch               Modify recipe files with JSON patch blobs.
+  rattler-bulk-build  Given a directory, performs a bulk rattler-build
+                      operation. Assumes rattler-build is installed.
 ```
 
-# Developer Notes
+A high-level overview of the CLI tools can be found [here](./conda_recipe_manager/commands/README.md).
+
+## Developer Installation and Notes
+The `make dev` directive will configure a `conda` environment named `conda-recipe-manager` for you with
+a development version of the tooling installed.
+
 ```sh
 make dev
 conda activate conda-recipe-manager
 ```
-The `dev` recipe will configure a `conda` environment named `conda-recipe-manager` with
-development tools installed.
 
-`pre-commit` is automatically installed and configured for you to run a number
-of automated checks on each commit.
+### Developer Documentation
+We aim for a very high bar when it comes to code documentation so that we may leverage automatic documentation
+workflows. API docs are hosted [here](https://conda-incubator.github.io/conda-recipe-manager/index.html)
+
+### Setup Troubleshooting
+- If you are currently in the `conda-recipe-manager` environment, make sure that you exit the environment with
+  `conda deactivate` before running `make dev`. There have been known issues with attempting to delete the environment
+  while an active instance is open.
+- There have been known some issues using Berkley `make` (`bmake`) to setup the environment. The `Makefile` provided
+  assumes GNU `make` is being used. This should only be an issue when running `make dev` as the `conda-recipe-manager`
+  environment installs a version of GNU `make` to the environment.
+
+### Making Commits
+`pre-commit` is automatically installed and configured for you to run a number of automated checks on each commit. These
+checks will also be strictly enforced by our automated GitHub workflows.
+
+This project uses modern Python type annotations and a strict set of `pylint` and `mypy` configurations to ensure code
+quality. We use the `black` text formatter to prevent arguments over code style. We attempt to signify if a type,
+variable, function, etc is `private`/`protected` with a single leading `_`.
 
-**NOTE:** As of writing, only a handful of files are checked by the linter and
-`pre-commit`. **ANY NEW FILES** should be added to these checks.
+### Running pre-commit Checks Individually
+The provided `Makefile` also provides a handful of convenience directives for running all or part of the `pre-commit`
+checks:
 
-### Running pre-commit checks
-The provided `Makefile` also provides a handful of convenience recipes for
-running all or part of the `pre-commit` automations:
 1. `make test`: Runs all the unit tests
-1. `make test-cov`: Reports the current test coverage percentage and indicates
-   which lines are currently untested.
-1. `make lint`: Runs our `pylint` configuration, based on Google's Python
-   standards.
+1. `make test-cov`: Reports the current test coverage percentage and indicates which lines are currently untested.
+1. `make lint`: Runs our `pylint` configuration, based on Google's Python standards.
 1. `make format`: Automatically formats code
 1. `make analyze`: Runs the static analyzer, `mypy`.
-1. `make pre-commit`: Runs all the `pre-commit` checks
+1. `make pre-commit`: Runs all the `pre-commit` checks on every file.
 
-## Release process
+### Release process
+Here is a brief overview of our current release process:
 1. Update `CHANGELOG.md`
 1. Update the version number in `pyproject.toml`, `docs/conf.py`, and `recipe/meta.yaml`
 1. Ensure `environment.yaml` is up to date with the latest dependencies

conda_recipe_manager/README.md

@@ -32,4 +32,9 @@ The primary class for reading recipe files is called the `RecipeReader` while ed
 
 ### `scanner` (WIP)
 This module provides tools for _scanning_ files and other feedstock/recipe artifacts. Unlike the parsers, full
-comprehension of the file(s) is not guaranteed.
+comprehension of the file(s) is not guaranteed. We hope to develop some advanced static project analysis here for
+multiple programming languages.
+
+### `utils`
+This module provides general utilities that have no other sensible home. These modules tend to
+be used by multiple modules throughout this project.