V0.4.0 (#49)

* add verbose to mask-retain added verbose information to the mask-retain mode * Add status timer add 5 minute status timer to the transformation controller to print updates * Update templates update templates and swap lists * Update SECURITY.md * Add load timer * async add * test timer * clean timer implementation * Update utils.go * Complete timer * Add memory usage * Update utils.go
JakeWnuk · Nov 9, 2024 · 7b94e5c · 7b94e5c
1 parent d5463ad
commit 7b94e5c
Show file tree

Hide file tree

Showing 17 changed files with 659 additions and 65 deletions.
diff --git a/README.md b/README.md
@@ -52,7 +52,7 @@ git clone https://github.com/JakeWnuk/ptt && cd ptt && docker build -t ptt . &&
 ---
 ### Usage:
 ```
-Usage of Password Transformation Tool (ptt) version (0.3.8):
+Usage of Password Transformation Tool (ptt) version (0.4.0):
 
 ptt [options] [...]
 Accepts standard input and/or additonal arguments.
@@ -124,7 +124,7 @@ These create or alter based on the selected mode.
         Transforms input by 'popping' tokens from character boundaries using the provided mask.
   -t mask-remove -rm [uldsb]
         Transforms input by removing characters with provided mask.
-  -t mask-retain -rm [uldsb] -tf [file]
+  -t mask-retain -rm [uldsb] -tf [file] -v
         Transforms input by creating masks that still retain strings from file.
   -t mask-swap -tf [file]
         Transforms input by swapping tokens from a mask/partial mask input and a transformation file of tokens.

diff --git a/docs/SECURITY.md b/docs/SECURITY.md
@@ -5,4 +5,5 @@
 ## Notes:
 - `pkg/utils/utils.go:9` use of `crypto/rand` over `math/rand` is not needed in this module.
 ## Last SAST Scan:
-- `10/16/2024` - `v0.3.7`
+- `11/08/2024` - `v0.4.0`
+- `10/16/2024` - `v0.3.7`
diff --git a/docs/USAGE.md b/docs/USAGE.md
@@ -1,5 +1,5 @@
 # Password Transformation Tool (PTT) Usage Guide
-> Version 0.3.8
+> Version 0.4.0
 ## Table of Contents
 - [Introduction](#introduction)
 - [Installation](#installation)
@@ -256,7 +256,7 @@ Where `<mask_characters>` can be any of the following:
 - `b`: Byte characters
 - Multiple characters can be combined to create a mask.
 
-The default value is `uldsb` for all characters. The `-v` flag is optional and, if provided, will print the length of the original string and its character complexity. The format will be `:length:complexity:mask-keyspace` appended to the end of the output. The mask keyspace is the number of possible combinations for the masked portion of the string.
+The default value is `uldsb` for all characters. The `-v` flag is optional and, if provided, will print the length of the original string, the length, the complexity, and the remaining mask keyspace. The format will be `:length:complexity:mask-keyspace` appended to the end of the output. The mask keyspace is the number of possible combinations for the masked portion of the string.
 ```
 $ echo 'HelloWorld!I<3ThePasswordTransformationToolPr0j3ct' | go run . -t mask -rm ds -v
 [*] All input loaded.
@@ -276,9 +276,9 @@ ptt -f <input_file> -t mask-remove -rm <mask_characters>
 ```
 Where `<mask_characters>` is the mask to remove from the string. The output will be the string with the characters removed.
 ### Creating Retain/Partial Masks
-Retain masks or partial masks can be created to retain only certain keywords in a string. The syntax to create a retain mask is as follows:
+Retain masks or partial masks can be created to retain only certain keywords in a string. The `-v` flag is optional and, if provided, will print the length of the original string, the length, the complexity, and the remaining mask keyspace. The syntax to create a retain mask is as follows:
 ```
-ptt -f <input_file> -t mask-retain -rm <mask_characters> -tf <keep_file>
+ptt -f <input_file> -t mask-retain -rm <mask_characters> -tf <keep_file> -v
 ```
 Where `<mask_characters>` is the mask to retain and `<keep_file>` is the file containing the keywords to retain. The output will be the mask with only the keywords retained.
 

diff --git a/main.go b/main.go
@@ -15,7 +15,7 @@ import (
 	"github.com/jakewnuk/ptt/pkg/utils"
 )
 
-var version = "0.3.8"
+var version = "0.4.0"
 var wg sync.WaitGroup
 var mutex = &sync.Mutex{}
 var retain models.FileArgumentFlag
@@ -44,29 +44,29 @@ func main() {
 		fmt.Fprintf(os.Stderr, "Transformation Modes:\n")
 		fmt.Fprintf(os.Stderr, "These create or alter based on the selected mode.\n\n")
 		modes := map[string]string{
-			"rule-append":                        "Transforms input by creating append rules.",
-			"rule-append-remove":                 "Transforms input by creating append-remove rules.",
-			"rule-prepend":                       "Transforms input by creating prepend rules.",
-			"rule-prepend-remove":                "Transforms input by creating prepend-remove rules.",
-			"rule-prepend-toggle":                "Transforms input by creating prepend-toggle rules.",
-			"rule-insert -i [index]":             "Transforms input by creating insert rules starting at index.",
-			"rule-overwrite -i [index]":          "Transforms input by creating overwrite rules starting at index.",
-			"rule-toggle -i [index]":             "Transforms input by creating toggle rules starting at index.",
-			"encode":                             "Transforms input by HTML and Unicode escape encoding.",
-			"decode":                             "Transforms input by HTML and Unicode escape decoding.",
-			"hex":                                "Transforms input by encoding strings into $HEX[...] format.",
-			"dehex":                              "Transforms input by decoding $HEX[...] formatted strings.",
-			"mask -rm [uldsb] -v":                "Transforms input by masking characters with provided mask.",
-			"mask-remove -rm [uldsb]":            "Transforms input by removing characters with provided mask.",
-			"mask-retain -rm [uldsb] -tf [file]": "Transforms input by creating masks that still retain strings from file.",
-			"mask-pop -rm [uldsbt]":              "Transforms input by 'popping' tokens from character boundaries using the provided mask.",
-			"mask-match -tf [file]":              "Transforms input by keeping only strings with matching masks from a mask file.",
-			"swap-single -tf [file]":             "Transforms input by swapping tokens once per string per replacement with exact matches from a ':' separated file.",
-			"mask-swap -tf [file]":               "Transforms input by swapping tokens from a mask/partial mask input and a transformation file of tokens.",
-			"passphrase -w [words]":              "Transforms input by generating passphrases from sentences with a given number of words.",
-			"substring -i [index]":               "Transforms input by extracting substrings starting at index and ending at index.",
-			"replace-all -tf [file]":             "Transforms input by replacing all strings with all matches from a ':' separated file.",
-			"regram -w [words]":                  "Transforms input by 'regramming' sentences into new n-grams with a given number of words.",
+			"rule-append":                           "Transforms input by creating append rules.",
+			"rule-append-remove":                    "Transforms input by creating append-remove rules.",
+			"rule-prepend":                          "Transforms input by creating prepend rules.",
+			"rule-prepend-remove":                   "Transforms input by creating prepend-remove rules.",
+			"rule-prepend-toggle":                   "Transforms input by creating prepend-toggle rules.",
+			"rule-insert -i [index]":                "Transforms input by creating insert rules starting at index.",
+			"rule-overwrite -i [index]":             "Transforms input by creating overwrite rules starting at index.",
+			"rule-toggle -i [index]":                "Transforms input by creating toggle rules starting at index.",
+			"encode":                                "Transforms input by HTML and Unicode escape encoding.",
+			"decode":                                "Transforms input by HTML and Unicode escape decoding.",
+			"hex":                                   "Transforms input by encoding strings into $HEX[...] format.",
+			"dehex":                                 "Transforms input by decoding $HEX[...] formatted strings.",
+			"mask -rm [uldsb] -v":                   "Transforms input by masking characters with provided mask.",
+			"mask-remove -rm [uldsb]":               "Transforms input by removing characters with provided mask.",
+			"mask-retain -rm [uldsb] -tf [file] -v": "Transforms input by creating masks that still retain strings from file.",
+			"mask-pop -rm [uldsbt]":                 "Transforms input by 'popping' tokens from character boundaries using the provided mask.",
+			"mask-match -tf [file]":                 "Transforms input by keeping only strings with matching masks from a mask file.",
+			"swap-single -tf [file]":                "Transforms input by swapping tokens once per string per replacement with exact matches from a ':' separated file.",
+			"mask-swap -tf [file]":                  "Transforms input by swapping tokens from a mask/partial mask input and a transformation file of tokens.",
+			"passphrase -w [words]":                 "Transforms input by generating passphrases from sentences with a given number of words.",
+			"substring -i [index]":                  "Transforms input by extracting substrings starting at index and ending at index.",
+			"replace-all -tf [file]":                "Transforms input by replacing all strings with all matches from a ':' separated file.",
+			"regram -w [words]":                     "Transforms input by 'regramming' sentences into new n-grams with a given number of words.",
 		}
 
 		// Sort and print transformation modes
@@ -110,7 +110,7 @@ func main() {
 
 	// Bypass map creation if requested
 	if *bypassMap {
-		fmt.Fprintf(os.Stderr, "[*] Bypassing map creation and using standard output as primary output. Options are disabled.\n")
+		fmt.Fprintf(os.Stderr, "[*] Bypassing map creation and using standard output as primary output. Options are disabled. This does not bypass the initial input memory usage.\n")
 	}
 
 	// Print debug information if requested
@@ -124,6 +124,8 @@ func main() {
 	var removeMap map[string]int
 	var readFilesMap map[string]int
 	var transformationFilesMap map[string]int
+	doneLoad := make(chan bool)
+	go utils.TrackLoadTime(doneLoad, "Load")
 
 	// Read files if provided
 	if retain != nil || remove != nil || readFiles != nil || transformationFiles != nil {
@@ -170,7 +172,13 @@ func main() {
 		primaryMap = utils.CombineMaps(primaryMap, readFilesMap, readURLsMap)
 	}
 
+	doneLoad <- true
+	close(doneLoad)
 	fmt.Fprintf(os.Stderr, "[*] All input loaded.\n")
+	fmt.Fprintf(os.Stderr, "[*] Starting Processing.\n")
+
+	doneProcess := make(chan bool)
+	go utils.TrackLoadTime(doneProcess, "Processing")
 
 	// Apply transformation if provided
 	if *transformation != "" && templateFiles == nil {
@@ -199,6 +207,9 @@ func main() {
 		return
 	}
 
+	doneProcess <- true
+	close(doneProcess)
+
 	// Print ignore case if provided
 	if *ignoreCase {
 		fmt.Fprintf(os.Stderr, "[*] Ignoring case when processing output.\n")

diff --git a/pkg/mask/mask.go b/pkg/mask/mask.go
@@ -130,11 +130,12 @@ func MakeMaskedMap(input map[string]int, replacementMask string, verbose bool, b
 //	retain (map[string]int): Map of keywords to retain
 //	bypass (bool): If true, the map is not used for output or filtering
 //	debug (bool): If true, print additional debug information to stderr
+//	verbose (bool): Verbose information if true
 //
 // Returns:
 //
 //	maskedMap (map[string]int): Masked retain map
-func MakeRetainMaskedMap(input map[string]int, replacementMask string, retain map[string]int, bypass bool, debug bool) map[string]int {
+func MakeRetainMaskedMap(input map[string]int, replacementMask string, retain map[string]int, bypass bool, debug bool, verbose bool) map[string]int {
 	maskedMap := make(map[string]int)
 	replacements := ConstructReplacements(replacementMask)
 	replacer := strings.NewReplacer(replacements...)
@@ -163,6 +164,10 @@ func MakeRetainMaskedMap(input map[string]int, replacementMask string, retain ma
 				continue
 			}
 
+			if verbose {
+				newKey = fmt.Sprintf("%s:%d:%d:%d", newKey, len(key), TestMaskComplexity(newKey), CalculateMaskKeyspace(newKey))
+			}
+
 			if debug {
 				fmt.Fprintf(os.Stderr, "[?] MakeRetainMaskedMap:\n")
 				fmt.Fprintf(os.Stderr, "Key: %s\n", key)

diff --git a/pkg/mask/mask_test.go b/pkg/mask/mask_test.go
@@ -115,7 +115,7 @@ func TestMakeRetainMaskedMap(t *testing.T) {
 
 	// Run test cases
 	for _, test := range tests {
-		output := MakeRetainMaskedMap(test.input, test.replacements, test.retain, false, false)
+		output := MakeRetainMaskedMap(test.input, test.replacements, test.retain, false, false, false)
 		if !reflect.DeepEqual(output, test.output) {
 			t.Errorf("Test failed: %v inputted, %v expected, %v returned", test.input, test.output, output)
 		}

diff --git a/pkg/transform/transform.go b/pkg/transform/transform.go
@@ -93,7 +93,7 @@ func TransformationController(input map[string]int, mode string, startingIndex i
 			fmt.Fprintf(os.Stderr, "[!] Retain masks require use of one or more -tf flags to specify one or more files.\n")
 			os.Exit(1)
 		}
-		output = mask.MakeRetainMaskedMap(input, replacementMask, transformationFilesMap, bypass, functionDebug)
+		output = mask.MakeRetainMaskedMap(input, replacementMask, transformationFilesMap, bypass, functionDebug, verbose)
 	case "mask-match", "match":
 		if len(transformationFilesMap) == 0 {
 			fmt.Fprintf(os.Stderr, "[!] Match masks require use of one or more -tf flags to specify one or more files.\n")

diff --git a/pkg/utils/utils.go b/pkg/utils/utils.go
@@ -12,6 +12,7 @@ import (
 	"os"
 	"path/filepath"
 	"regexp"
+	"runtime"
 	"sort"
 	"strconv"
 	"strings"
@@ -30,6 +31,38 @@ import (
 // Loading and Processing Functions
 // ----------------------------------------------------------------------------
 
+// TrackLoadTime tracks the time it takes to load the input and prints the time
+//
+// Args:
+// done (chan bool): channel to use to track tasks
+// work (string): string used in status printing
+//
+// Returns:
+// None
+func TrackLoadTime(done <-chan bool, work string) {
+	start := time.Now()
+	ticker := time.NewTicker(30 * time.Second)
+	for {
+		select {
+		case <-done:
+			ticker.Stop()
+			fmt.Fprintf(os.Stderr, "[-] Total %s Time: %02d:%02d:%02d.\n", work, int(time.Since(start).Hours()), int(time.Since(start).Minutes())%60, int(time.Since(start).Seconds())%60)
+			return
+		case t := <-ticker.C:
+			elapsed := t.Sub(start)
+			memUsage := GetMemoryUsage()
+			fmt.Fprintf(os.Stderr, "[-] Please wait. Elapsed: %02d:%02d:%02d.%03d. Memory Usage: %.2f MB.\n", int(t.Sub(start).Hours()), int(t.Sub(start).Minutes())%60, int(t.Sub(start).Seconds())%60, elapsed.Milliseconds()%1000, memUsage)
+		}
+	}
+}
+
+// GetMemoryUsage returns the current memory usage in megabytes
+func GetMemoryUsage() float64 {
+	var m runtime.MemStats
+	runtime.ReadMemStats(&m)
+	return float64(m.Alloc) / 1024 / 1024
+}
+
 // ReadFilesToMap reads the contents of the multiple files and returns a map of words
 //
 // Args:

diff --git a/templates/misc/swap-encode.json → templates/append-rule.json b/templates/misc/swap-encode.json → templates/append-rule.json
@@ -5,7 +5,7 @@
         "Verbose": false,
         "ReplacementMask": "uldsb",
         "Bypass": false,
-        "TransformationMode": "swap",
+        "TransformationMode": "rule-append",
         "WordRangeStart": 0,
         "WordRangeEnd": 0
     },
@@ -15,7 +15,7 @@
         "Verbose": false,
         "ReplacementMask": "uldsb",
         "Bypass": false,
-        "TransformationMode": "encode",
+        "TransformationMode": "rule-append-remove",
         "WordRangeStart": 0,
         "WordRangeEnd": 0
     }

diff --git a/templates/rules/insert1-6.json → templates/insert-overwrite-1-9.json b/templates/rules/insert1-6.json → templates/insert-overwrite-1-9.json
@@ -1,21 +1,21 @@
 [
     {
         "StartIndex": 1,
-        "EndIndex": 6,
+        "EndIndex": 9,
         "Verbose": false,
         "ReplacementMask": "uldsb",
         "Bypass": false,
-        "TransformationMode": "insert",
+        "TransformationMode": "rule-insert",
         "WordRangeStart": 0,
         "WordRangeEnd": 0
     },
     {
         "StartIndex": 1,
-        "EndIndex": 6,
+        "EndIndex": 9,
         "Verbose": false,
         "ReplacementMask": "uldsb",
         "Bypass": false,
-        "TransformationMode": "overwrite",
+        "TransformationMode": "rule-overwrite",
         "WordRangeStart": 0,
         "WordRangeEnd": 0
     }

diff --git a/templates/masks/mask.json → templates/mask-analysis.json b/templates/masks/mask.json → templates/mask-analysis.json
diff --git a/templates/rules/append.json → templates/prepend-rule.json b/templates/rules/append.json → templates/prepend-rule.json
@@ -5,7 +5,7 @@
         "Verbose": false,
         "ReplacementMask": "uldsb",
         "Bypass": false,
-        "TransformationMode": "append",
+        "TransformationMode": "rule-prepend",
         "WordRangeStart": 0,
         "WordRangeEnd": 0
     },
@@ -15,17 +15,7 @@
         "Verbose": false,
         "ReplacementMask": "uldsb",
         "Bypass": false,
-        "TransformationMode": "append-remove",
-        "WordRangeStart": 0,
-        "WordRangeEnd": 0
-    },
-    {
-        "StartIndex": 0,
-        "EndIndex": 0,
-        "Verbose": false,
-        "ReplacementMask": "uldsb",
-        "Bypass": false,
-        "TransformationMode": "append-shift",
+        "TransformationMode": "rule-prepend-remove",
         "WordRangeStart": 0,
         "WordRangeEnd": 0
     }