Skip to content

Simple tool to extract the most common substrings from an input text. Built for password cracking.

License

Notifications You must be signed in to change notification settings

sensepost/common-substr

Repository files navigation

common-substr

Simple tool to extract the most common substrings from an input text. Built for password cracking. A write-up on the theory can be found at the sensepost.com blog

There are two versions, the older awk script and the newer & faster golang version. They use the same commandline switches.

Usage

Common Substring Generator by @singe
Usage: ./common-substr [-hinsp] [-t <n>] [-l <n>] [-L <n>] -f <filename>
	-h|--help This help
	-i|--insensitive Ignore case of substrings
	-L|--maxlength <n> Maximum length substring to look for. Default is 32.
	-l|--minlength <n> Minimum length substring to look for. Default is 2.
	-n|--nostats Just print the substrings, no stats. Default is to include them.
	-t|--threshold <n> Only print substrings more prevalent than <n> percent.
	-f|--file <filename> The file to extract substrings from
	-s|--suffix Only look at suffix substrings at the end of a string
	-p|--prefix Only look at prefix substrings at the beginning of a string
Default output (with stats) is tab separated: <percentage>	<count>	<substring>
Sorted from most to least common

Simple Usage Examples

Given the test file:

123
123
234

We can find the most common substrings:

./common-substr -f test
100     3 23
66.6667 2 12
66.6667 2 123

Read this output as "100% of the input file had the substring "23" which consisted of 3 instances".

Do the same, but suppress printing of the stats:

./common-substr -f test -n
23
12
123

Only include substrings that occur at least 70% of the time:

./common-substr -f test -t 70
100	3	23

The stats are tab-separated, to make cut'ing easy:

./common-substr -f test > output
cut -f 3 output
23
12
123

Only include substrings 3 characters or longer:

./common-substr -f test -l 3 
66.6667 2 123

Only include substrings 2 characters or shorter:

./common-substr -f test -L 2 
100     3 23
66.6667 2 12

Only include the start of the strings (prefix):

./common-substr -f test -p
66.6667	2	12
66.6667	2	123

Only include the end of the strings (suffix):

./common-substr -f test -s
66.6667	2	23
66.6667	2	123

Password Cracking Examples

Vanilla wordlist + substrings

An example use for password cracking. Assuming you've put already cracked clear-text passwords in a file called 'passwords':

# Limit substrings to a max length of 27 and only include those which occur
# at least 1% or more of the time
./common-substr -t 1 -l 27 -n -f passwords > substrs
sort -u passwords > uniques
hashcat -a1 hashes uniques substrs 

Basewords + substrings

It also helps to create "base words" and combine those with the substrings:

grep -oi "[a-z]*[a-z]" uniques > basewords
hashcat -a1 hashes basewords substrs

Remember to try it the other way around too:

hashcat -a1 hashes substrs basewords

All Substrings

Drop the threshold and throw the full list of substrings into combinator:

./common-substr -n -f passwords > all-substrs
hashcat -a1 hashes all-substrs all-substrs

Prefix & Suffix Substrings

Take the commons starts and ends of passwords and combine them:

./common-substr -n -p -f passwords > prefix
./common-substr -n -s -f passwords > suffix
hashcat -a1 hashes prefix suffix

Building

The golang version can be built using go build ./common-substr.go.

The awk version can be run using the common-substr.sh wrapper script. It requires awk.

I recommend the golang version.

About

Simple tool to extract the most common substrings from an input text. Built for password cracking.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published