Command line changes for v3.0

[tarkatronic]

Usage: tartufo [OPTIONS] COMMAND [ARGS]...

Find secrets hidden in the depths of git.

Tartufo will, by default, scan the entire history of a git repository for
any text which looks like a secret, password, credential, etc. It can also
be made to work in pre-commit mode, for scanning blobs of text as a pre-
commit hook.

Options:
--json / --no-json Output in JSON format.

• Default: --no-json
• Other: --output-format json / compact / text / yaml / toml (WINNER)

--rules FILENAME Path(s) to regex rules json list file(s).

• Change to inline config

--default-regexes / --no-default-regexes Whether to include the default regex list when configuring search patterns. Only applicable if --rules is also specified. [default: True]

• Default: --default-regexes (no change)

--compact / --no-compact Enable reduced output. [default: --no-compact]

• Deprecate in favor of --output-format

--entropy / --no-entropy Enable entropy checks. [default: True]

• Default: --entropy (no change)

--regex / --no-regex Enable high signal regexes checks. [default: False]

• Default: --regex

-i, --include-paths FILENAME [DEPRECATED] Use --include-path-patterns. File with regular expressions (one per line), at least one of which must match a Git object path in order for it to be scanned; lines starting with '#' are treated as comments and are ignored. If empty or not provided (default), all Git object paths are included unless otherwise excluded via the --exclude-paths option.

• Remove

-ip, --include-path-patterns TEXT Specify a regular expression which matches Git object paths to include in the scan. This option can be specified multiple times to include multiple patterns. If not provided (default), all Git object paths are included unless otherwise excluded via the --exclude-path-patterns option.

• Config file only (table of tables)

-x, --exclude-paths FILENAME [DEPRECATED] Use --exclude-path-patterns. File with regular expressions (one per line), none of which may match a Git object path in order for it to be scanned; lines starting with '#' are treated as comments and are ignored. If empty or not provided (default), no Git object paths are excluded unless effectively excluded via the --include-paths option.

• Remove

-xp, --exclude-path-patterns TEXT. Specify a regular expression which matches Git object paths to exclude from the scan. This option can be specified multiple times to exclude multiple patterns. If not provided (default), no Git object paths are excluded unless effectively excluded via the --include-path-patterns option.

• Config file only (table of tables)

-xe, --exclude-entropy-patterns TEXT. Specify a regular expression which matches entropy strings to exclude from the scan. This option can be specified multiple times to exclude multiple patterns. If not provided (default), no entropy strings will be excluded ({path regex}::{pattern regex}).

• Config file only (table of tables)

-e, --exclude-signatures TEXT Specify signatures of matches that you explicitly want to exclude from the scan, and mark as okay. These signatures are generated during the scan process, and reported out with each individual match. This option can be specified multiple times, to exclude as many signatures as you would like.

• Config file only (table of tables)

-od, --output-dir DIRECTORY If specified, all issues will be written out as individual JSON files to a uniquely named directory under this one. This will help with keeping the results of individual runs of tartufo separated.

• No change

--git-rules-repo TEXT A file path, or git URL, pointing to a git repository containing regex rules to be used for scanning. By default, all .json files will be loaded from the root of that repository. --git-rules-files can be used to override this behavior and load specific files.

• Update help text
• Rename? --git-config-repo ?

--git-rules-files TEXT Used in conjunction with --git-rules-repo, specify glob-style patterns for files from which to load the regex rules. Can be specified multiple times.

• Remove (load from config file instead)

--config FILE Read configuration from specified file. [default: tartufo.toml]

• No change
• Maybe look at Cargo.toml as well? (Rust)

-q, --quiet / --no-quiet Quiet mode. No outputs are reported if the scan is successful and doesn't find any issues

• Default: --no-quiet (no change)

-v, --verbose Display more verbose output. Specifying this option multiple times will incrementally increase the amount of output.

• Limit to 3 levels (currently 6)
• Document levels in help text

--log-timestamps / --no-log-timestamps Enable or disable timestamps in logging messages. [default: True]

• Default: --log-timestamps (no change)

-b64, --b64-entropy-score FLOAT Modify the base64 entropy score. If a value greater than the default is specified, tartufo lists higher entropy base64 strings (longer or more randomized strings). A lower value lists lower entropy base64 strings (shorter or less randomized strings). [default: 4.5]

• Deprecate in favor of --entropy-sensitivity 1-100

-hex, --hex-entropy-score FLOAT Modify the hexadecimal entropy score. If a value greater than the default is specified, tartufo lists higher entropy hexadecimal strings (longer or more randomized strings). A lower value lists lower entropy hexadecimal strings (shorter or less randomized strings). [default: 3.0]

• Deprecate in favor of --entropy-sensitivity 1-100

-V, --version Show the version and exit.
-h, --help Show this message and exit.

Commands:
pre-commit Scan staged changes in a pre-commit hook.
scan-remote-repo Automatically clone and scan a remote git repository.
scan-folder Scan a folder.
scan-local-repo Scan a repository already cloned to your local system.

Usage: tartufo pre-commit [OPTIONS]

Scan staged changes in a pre-commit hook.

Options:
--include-submodules / --exclude-submodules Controls whether the contents of git submodules are scanned [default: False]

• Default: --exclude-submodules (no change)

-h, --help Show this message and exit.

Usage: tartufo scan-remote-repo [OPTIONS] GIT_URL

Automatically clone and scan a remote git repository.

Options:
--since-commit TEXT Only scan from a given commit hash.

• --commits asdf12342..qwert6543
• --commits asdf1234..HEAD
• --commits HEAD~5..HEAD == --max-depth 5

--max-depth INTEGER The max commit depth to go back when searching for secrets. [default: 1000000]

• Maybe merge with --since-commit / --commits ?
• --from-commit asdf1234 / --to-commit qwert6543 ?

--branch TEXT Specify a branch name to scan only that
branch.

• No change
• Add --current-branch (v3.1?)

-wd, --work-dir DIRECTORY Specify a working directory; this is where the repository will be cloned to before scanning.

• No change

--include-submodules / --exclude-submodules Controls whether the contents of git submodules are scanned [default: False]

• Default: --exclude-submodules (no change)

-h, --help Show this message and exit.

Usage: tartufo scan-local-repo [OPTIONS] REPO_PATH

Scan a repository already cloned to your local system.

Options:
--since-commit TEXT Only scan from a given commit hash.

• --commits asdf12342..qwert6543
• --commits asdf1234..HEAD
• --commits HEAD~5..HEAD == --max-depth 5

--max-depth INTEGER The max commit depth to go back when searching for secrets. [default: 1000000]

• Maybe merge with --since-commit / --commits ?
• --from-commit asdf1234 / --to-commit qwert6543 ?

--branch TEXT Specify a branch name to scan only that branch.

• No change
• Add --current-branch (v3.1?)

--fetch / --no-fetch Controls whether or not the remote repo is fetched prior to local scanning [default: True]

• Remove

--include-submodules / --exclude-submodules Controls whether the contents of git submodules are scanned [default: False]

• Default: --exclude-submodules (no change)

-h, --help Show this message and exit.

Usage: tartufo scan-folder [OPTIONS] TARGET

Scan a folder.

Options:
-h, --help Show this message and exit.

• Add --recurse / --no-recurse (default: --recurse) (v3.1?)

[rmarkins-godaddy]

If we are discussing command line changes what was the decision to do

tartufo [OPTIONS] COMMAND [ARGS]...

typically (at least most cli's I've encountered) is

<cli-name> COMMAND [OPTIONS/FlAGS] [ARGS]

I don't have hard arguments for this way other than it just feels odd to do

tartufo --config ./tartufo.toml pre-commit

Just my 2️⃣ 🪙 😄

[tarkatronic]

That's a result of the way that tartufo is split into multiple parts:

• The main top-level command tartufo
• The options for that top-level command, which apply to all use-cases
• Sub-commands, pre-commit, scan-local-repo, scan-remote-repo, and scan-folder
• Options/arguments that apply specifically to those individual sub-commands

And also a result of the way that our underlying CLI library, click, works with those sub-commands.

It would likely require swapping out to an entirely different CLI library if we wanted to have different option parsing.

[rbailey-godaddy]

Same thing in different words -- this isn't about the library but rather eliminating ambiguity from the conceptual model. Most CLIs where this issue is in play actually look like:

<cli-name> [GLOBAL-OPTIONS] COMMAND [POSITIONAL-OPTIONS] [ARGS...]

For python, both click and argparse produce CLIs that look like this. The Unix/Linux find command works this way (where options that control the search must appear before the search base(s) and options that specify filters must appear after). Everything in DEC's VMS operating system works this way (dating myself).

We could sweep the distinction under the rug by defining everything as a global option, but then we lose a huge chunk of functionality because we have to do all of the "are you allowed to specify this qualifier in this context?" validation ourselves. Bleah.

[tarkatronic]

This has all been turned into the following issues:

#240, #253, #254, #255, #256, #257, #264, #265, #267, #268