common-substr

Simple tool to extract the most common substrings from an input text. Built for password cracking. A write-up on the theory can be found at the sensepost.com blog

There are two versions, the older awk script and the newer & faster golang version. They use the same commandline switches.

Usage

Common Substring Generator by @singe
Usage: ./common-substr [-hinsp] [-t <n>] [-l <n>] [-L <n>] -f <filename>
	-h|--help This help
	-i|--insensitive Ignore case of substrings
	-L|--maxlength <n> Maximum length substring to look for. Default is 32.
	-l|--minlength <n> Minimum length substring to look for. Default is 2.
	-n|--nostats Just print the substrings, no stats. Default is to include them.
	-t|--threshold <n> Only print substrings more prevalent than <n> percent.
	-f|--file <filename> The file to extract substrings from
	-s|--suffix Only look at suffix substrings at the end of a string
	-p|--prefix Only look at prefix substrings at the beginning of a string
Default output (with stats) is tab separated: <percentage>	<count>	<substring>
Sorted from most to least common

Simple Usage Examples

Given the test file:

123
123
234

We can find the most common substrings:

./common-substr -f test
100     3 23
66.6667 2 12
66.6667 2 123

Read this output as "100% of the input file had the substring "23" which consisted of 3 instances".

Do the same, but suppress printing of the stats:

./common-substr -f test -n
23
12
123

Only include substrings that occur at least 70% of the time:

./common-substr -f test -t 70
100	3	23

The stats are tab-separated, to make cut'ing easy:

./common-substr -f test > output
cut -f 3 output
23
12
123

Only include substrings 3 characters or longer:

./common-substr -f test -l 3 
66.6667 2 123

Only include substrings 2 characters or shorter:

./common-substr -f test -L 2 
100     3 23
66.6667 2 12

Only include the start of the strings (prefix):

./common-substr -f test -p
66.6667	2	12
66.6667	2	123

Only include the end of the strings (suffix):

./common-substr -f test -s
66.6667	2	23
66.6667	2	123

Password Cracking Examples

Vanilla wordlist + substrings

An example use for password cracking. Assuming you've put already cracked clear-text passwords in a file called 'passwords':

# Limit substrings to a max length of 27 and only include those which occur
# at least 1% or more of the time
./common-substr -t 1 -l 27 -n -f passwords > substrs
sort -u passwords > uniques
hashcat -a1 hashes uniques substrs 

Basewords + substrings

It also helps to create "base words" and combine those with the substrings:

grep -oi "[a-z]*[a-z]" uniques > basewords
hashcat -a1 hashes basewords substrs

Remember to try it the other way around too:

hashcat -a1 hashes substrs basewords

All Substrings

Drop the threshold and throw the full list of substrings into combinator:

./common-substr -n -f passwords > all-substrs
hashcat -a1 hashes all-substrs all-substrs

Prefix & Suffix Substrings

Take the commons starts and ends of passwords and combine them:

./common-substr -n -p -f passwords > prefix
./common-substr -n -s -f passwords > suffix
hashcat -a1 hashes prefix suffix

Building

The golang version can be built using go build ./common-substr.go.

The awk version can be run using the common-substr.sh wrapper script. It requires awk.

I recommend the golang version.