Skip to content

Commit

Permalink
Add and improve rules (#232)
Browse files Browse the repository at this point in the history
- Add rule: `JSON Web Token Secret`
- Add rule: `Credentials in MongoDB Connection String`
- Refine `Jenkins Token or Crumb` rule for more detection
- Refine `Credentials in PostgreSQL Connection String`
- Retrain tests
- Update changelog
  • Loading branch information
bradlarsen authored Nov 19, 2024
1 parent 11c6a61 commit 8375040
Show file tree
Hide file tree
Showing 10 changed files with 242 additions and 21 deletions.
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,18 +17,23 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Lockfiles from a few languages (e.g., `Cargo.lock`, `Pipfile.lock`, `go.sum`) are now ignored by default.

- Rules have been modified:

- `Age Recipient (X25519 public key)` and `ThingsBoard Access Token` have expanded category metdata.
- `Credentials in ODBC Connection String` detects more occurrences ([#227](https://github.com/praetorian-inc/noseyparker/pull/227)).
- `Jenkins Token or Crumb` has been refined to improve detection ([#232](https://github.com/praetorian-inc/noseyparker/pull/232)).


- When using the `--copy-blobs` option, the default output format is now `parquet` (when the `parquet` feature is enabled, which it is unless you build with `--no-default-features`) ([#229](https://github.com/praetorian-inc/noseyparker/pull/229)).

### Additions

- New rules have been added:

- `Credentials in MongoDB Connection String` ([#232](https://github.com/praetorian-inc/noseyparker/pull/232))
- `Credentials in PostgreSQL Connection URI` ([#227](https://github.com/praetorian-inc/noseyparker/pull/227))
- `Django Secret Key` ([#227](https://github.com/praetorian-inc/noseyparker/pull/227))
- `Jina Search Foundation API Key`
- `JSON Web Token Secret` ([#232](https://github.com/praetorian-inc/noseyparker/pull/232))
- `HTTP Basic Authentication`
- `HTTP Bearer Token`
- `PHPMailer Credentials` ([#227](https://github.com/praetorian-inc/noseyparker/pull/227))
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Nosey Parker is a command-line tool that finds secrets and sensitive information

**Key features:**
- It natively scans files, directories, and Git repository history
- It uses regular expression matching with a set of [151 patterns](crates/noseyparker/data/default/builtin/rules) chosen for high signal-to-noise based on experience and feedback from offensive security engagements
- It uses regular expression matching with a set of [154 patterns](crates/noseyparker/data/default/builtin/rules) chosen for high signal-to-noise based on experience and feedback from offensive security engagements
- It deduplicates its findings, grouping matches together that share the same secret, which in practice can reduce review burden by 100x or more compared to other tools
- It is fast: it can scan at hundreds of megabytes per second on a single core, and is able to scan 100GB of Linux kernel source history in less than 2 minutes on an older MacBook Pro
- It scales: it has scanned inputs as large as 20TiB during security engagements
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
source: crates/noseyparker-cli/tests/rules/mod.rs
expression: stdout
---
151 rules and 3 rulesets: no issues detected
154 rules and 3 rulesets: no issues detected
Original file line number Diff line number Diff line change
Expand Up @@ -1770,16 +1770,17 @@ expression: stdout
},
{
"id": "np.jenkins.1",
"structural_id": "aa43b5efaf8c5a50caa4af166bbec9606aa2f12a",
"structural_id": "2f7fa7fff0e28da2f3138710adec2eb499e656d4",
"name": "Jenkins Token or Crumb",
"syntax": {
"name": "Jenkins Token or Crumb",
"id": "np.jenkins.1",
"pattern": "(?i)jenkins.{0,10}(?:crumb)?.{0,10}\\b([0-9a-f]{32,36})\\b",
"pattern": "(?x)\n(?i)\njenkins.{0,12}(?: (?: crumb | token ) .{0,10} )?\n\\b\n( [0-9a-f]{32,36} | [a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12} )\n(?: [^0-9a-f-] | $ )\n",
"examples": [
"jenkins_user = 'root'\n# jenkins_passwd = '116365fd86d63bf507aba962606a5c8956' Pre token\njenkins_passwd = '11811f784531053132519844d047186074' # Dev Token\njenkins_url = 'http://10.1.188.121'\n",
"export JENKINS_USER=justin-admin-edit-view\nexport JENKINS_TOKEN=11f4274ec59be12eace9a08b08ee13d54b\nexport JENKINS=jenkins-cicd.apps.sno.openshiftlabs.net\n",
"sh \"curl -X POST 'http://jenkins.lsfusion.luxsoft.by/job/${Paths.updateParentVersionsJob}/build' --user ${USERPASS} -H 'Jenkins-Crumb:440561953171ba44ace9740562d172bb'\"\n"
"sh \"curl -X POST 'http://jenkins.lsfusion.luxsoft.by/job/${Paths.updateParentVersionsJob}/build' --user ${USERPASS} -H 'Jenkins-Crumb:440561953171ba44ace9740562d172bb'\"\n",
"// Jenkins query token\nreadonly static string jenkinsToken = \"df490bc6-ef6e-4f6a-99a0-b03d26cb56f9\";\n"
],
"negative_examples": [
"1. ~~Does not play well with [Build Token Root Plugin](https://wiki.jenkins-ci.org/display/JENKINS/Build+Token+Root+Plugin) URL formats.~~ (added with [this commit](https://github.com/morficus/Parameterized-Remote-Trigger-Plugin/commit/f687dbe75d1c4f39f7e14b68220890384d7c5674) )"
Expand Down Expand Up @@ -1843,6 +1844,52 @@ expression: stdout
]
}
},
{
"id": "np.jwt.2",
"structural_id": "76dd64b6948d4d3a8324bfcad0a0a3327174bf61",
"name": "JSON Web Token Secret",
"syntax": {
"name": "JSON Web Token Secret",
"id": "np.jwt.2",
"pattern": "(?x)(?m)(?i)\n^\\s*\n(?: jwt: \\s+ secret: | jwtsecret: )\n\\s+\n\"( [^\"]{6,200} )\"\n",
"examples": [
"something_else: true\njwt:\n secret: \"acb1ca5bc8d7fb2ef9f890f1be16d964\" #some comment\n ttl: 640000 #\n",
"JWT:\n secret: \"development#product\"\n exp: \"50m\"\n",
"oauthproviders:\n jwtsecret: \"7`D'}/vYR)T;W\\5]EBrf;_&k>qX??]G&3CF=;@vyrVBp[$_hK4^?{qT/EglJe[FYjp:IPgl_t$%r\"\n"
],
"negative_examples": [],
"references": [
"https://en.wikipedia.org/wiki/JSON_Web_Token"
],
"categories": [
"fuzzy",
"secret"
]
}
},
{
"id": "np.jwt.3",
"structural_id": "0c8531cfa969e7834743191e9da4bf911be46e9b",
"name": "JSON Web Token Secret",
"syntax": {
"name": "JSON Web Token Secret",
"id": "np.jwt.3",
"pattern": "(?x)(?m)(?i)\n^\\s*\n(?: jwt: \\s+ secret: | jwtsecret: )\n\\s+\n'( [^']{6,200} )'\n",
"examples": [
"something_else: true\njwt:\n secret: 'acb1ca5bc8d7fb2ef9f890f1be16d964' #some comment\n ttl: 640000 #\n",
"JWT:\n Secret: 'development#product'\n exp: '50m'\n",
"oauthproviders:\n jwtsecret: '7`D\"}/vYR)T;W\\5]EBrf;_&k>qX??]G&3CF=;@vyrVBp[$_hK4^?{qT/EglJe[FYjp:IPgl_t$%r'\n"
],
"negative_examples": [],
"references": [
"https://en.wikipedia.org/wiki/JSON_Web_Token"
],
"categories": [
"fuzzy",
"secret"
]
}
},
{
"id": "np.krb5.asrep.23.1",
"structural_id": "0026c7bd3577be46d947892edab1bde4e1c320a0",
Expand Down Expand Up @@ -2031,6 +2078,41 @@ expression: stdout
]
}
},
{
"id": "np.mongodb.1",
"structural_id": "79fc752031d3098001d928a1246e48cdad695899",
"name": "Credentials in MongoDB Connection String",
"syntax": {
"name": "Credentials in MongoDB Connection String",
"id": "np.mongodb.1",
"pattern": "(?x)\n(?: mongodb\\+srv | mongodb ) :// (?# URI scheme )\n ([a-zA-Z0-9%;._~!$&'()*+,;=-]{3,}) (?# username)\n: ([a-zA-Z0-9%;._~!$&'()*+,;=-]{3,}) (?# password)\n@ ([a-zA-Z0-9_.-]{3,} (?: :\\d{1,5})?) (?# hostname and port)\n(?: [^a-zA-Z0-9_.-] | $ )\n",
"examples": [
"spring:\n application:\n name: Prince\n data:\n mongodb:\n uri: \"mongodb+srv://servicelogin:CPCdZH5k0F6dxQ9V@example-prince.mongodb.net/?retryWrites=true&w=majority\"\n database: prince\n",
"mongodb://myDatabaseUser:D1fficult@localhost",
"mongodb://myDatabaseUser:D1fficult@localhost/database",
"mongodb://myDatabaseUser:D1fficultP%40ssw0rd@localhost",
"mongodb://myDatabaseUser:D1fficultP%40ssw0rd@localhost/records",
"mongodb+srv://myDatabaseUser:D1fficultP%40ssw0rd@cluster0.example.mongodb.net/?retryWrites=true&w=majority",
"mongodb+srv://myDatabaseUser:D1fficultP%40ssw0rd@mongodb0.example.com/?authSource=admin&replicaSet=myRepl",
"mongodb://myDatabaseUser:D1fficultP%40ssw0rd@mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?authSource=admin&replicaSet=myRepl",
"mongodb://myDatabaseUser:D1fficultP%40ssw0rd@localhost,localhost:27018,localhost:27019/?replicaSet=test",
"mongodb://myDatabaseUser:D1fficultP%40ssw0rd@example1.com,example2.com,example3.com/?replicaSet=test&readPreference=secondary",
"mongodb://myDatabaseUser:D1fficultP%40ssw0rd@router1.example.com:27017,router2.example2.com:27017,router3.example3.com:27017/",
"mongodb://myDatabaseUser:D1fficultP%40ssw0rd@example1.com,example2.com,example3.com/?replicaSet=test&w=majority&wtimeoutMS=2000",
"var store = new MongoDBStore({\n uri: 'mongodb+srv://somethingrotten:Zxcvbnm@12345)@cluster0-cnc84.mongodb.net/e-commerce?retryWrites=true',\n collection: 'mySessions',\n});\n",
"mongodb+srv://hpapp:bria@pass135!@briaserver.7xsu17j.mongodb.net/test"
],
"negative_examples": [
"mongodb://myDatabaseUser:D1fficultP%40ssw0rd@%2Ftmp%2Fmongodb-27017.sock"
],
"references": [
"https://www.mongodb.com/docs/manual/reference/connection-string/"
],
"categories": [
"secret"
]
}
},
{
"id": "np.msteams.1",
"structural_id": "ffeb4e52ea44a01f3e630ae45cc92f7b39558446",
Expand Down Expand Up @@ -2551,20 +2633,25 @@ expression: stdout
},
{
"id": "np.postgres.1",
"structural_id": "d1b558227cf1fa521f2db96bac75c1914554d667",
"structural_id": "937c1841abe7875724f3c4100036e6f2b8e79cb9",
"name": "Credentials in PostgreSQL Connection URI",
"syntax": {
"name": "Credentials in PostgreSQL Connection URI",
"id": "np.postgres.1",
"pattern": "(?x)\n(?: postgres | postgresql ) :// (?# URI scheme )\n ([a-zA-Z0-9.-~]{3,}) (?# username)\n: ([a-zA-Z0-9.-~]{3,}) (?# password)\n@ ([a-zA-Z0-9_.-]{3,} (?: :\\d{1,5})?) (?# hostname and port)\n(/[a-zA-Z0-9_.-]{2,}) (?# database)\n(?: \\? [a-zA-Z0-9.-~]+ = [a-zA-Z0-9.-~]+\n (?: & [a-zA-Z0-9.-~]+ = [a-zA-Z0-9.-~]+ )* )? (?# query params )\n(?: [^a-zA-Z0-9.-~] | $ )\n",
"pattern": "(?x)\n(?: postgres | postgresql ) :// (?# URI scheme )\n ([a-zA-Z0-9%;._~!$&'()*+,;=-]{3,}) (?# username)\n: ([a-zA-Z0-9%;._~!$&'()*+,;=-]{3,}) (?# password)\n@ ([a-zA-Z0-9_.-]{3,} (?: :\\d{1,5})?) (?# hostname and port)\n(?: [^a-zA-Z0-9_.-] | $ )\n",
"examples": [
"\"REDSHIFT\": \"postgres://spot_app:Pseg2020@calling-mr-bones.c0qsadyxbf4k.us-east-1.redshift.amazonaws.com:5439/datalakespotprod\",",
"postgresql://user:secret@localhost",
"postgresql://user:secret@localhost/",
"postgresql://user:secret@localhost/database",
"postgresql://user:secret@localhost/otherdb?connect_timeout=10&application_name=myapp"
"postgresql://user:secret@localhost:5433",
"postgresql://user:secret@localhost:5433/",
"postgresql://user:secret@localhost:5433/database",
"\"REDSHIFT\": \"postgres://spot_app:Pseg2020@calling-mr-bones.c0qsadyxbf4k.us-east-1.redshift.amazonaws.com:5439/datalakespotprod\",",
"postgresql://user:secret@localhost/otherdb?connect_timeout=10&application_name=myapp",
"postgresql://user:secret@host1:123,user:secret@host2:456/somedb?target_session_attrs=any&application_name=myapp"
],
"negative_examples": [
"postgresql://user:secret@[2001:db8::1234]/database",
"postgresql://user:secret@host1:123,user:secret@host2:456/somedb?target_session_attrs=any&application_name=myapp",
"postgresql:///mydb?host=localhost&port=5433&user=user&password=secret"
],
"references": [
Expand Down Expand Up @@ -3648,7 +3735,7 @@ expression: stdout
{
"id": "default",
"name": "Nosey Parker default rules",
"num_rules": 130
"num_rules": 133
},
{
"id": "np.assets",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ expression: stdout
np.jenkins.1 Jenkins Token or Crumb api, fuzzy, secret
np.jina.1 Jina Search Foundation API Key api, secret
np.jwt.1 JSON Web Token (base64url-encoded) api
np.jwt.2 JSON Web Token Secret fuzzy, secret
np.jwt.3 JSON Web Token Secret fuzzy, secret
np.krb5.asrep.23.1 Password Hash (Kerberos 5, etype 23, AS-REP) hashed, secret
np.linkedin.1 LinkedIn Client ID api, fuzzy, identifier
np.linkedin.2 LinkedIn Secret Key api, fuzzy, secret
Expand All @@ -89,6 +91,7 @@ expression: stdout
np.mapbox.1 Mapbox Public Access Token api, fuzzy
np.mapbox.2 Mapbox Secret Access Token api, fuzzy, secret
np.mapbox.3 Mapbox Temporary Access Token api, fuzzy, secret
np.mongodb.1 Credentials in MongoDB Connection String secret
np.msteams.1 Microsoft Teams Webhook api, secret
np.netrc.1 netrc Credentials secret
np.newrelic.1 New Relic License Key secret
Expand Down Expand Up @@ -158,6 +161,6 @@ expression: stdout

Ruleset ID Ruleset Name Rules
─────────────────────────────────────────────────────────
default Nosey Parker default rules 130
default Nosey Parker default rules 133
np.assets Nosey Parker asset detection rules 15
np.hashes Nosey Parker password hash rules 6
11 changes: 10 additions & 1 deletion crates/noseyparker/data/default/builtin/rules/jenkins.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,13 @@ rules:
- name: Jenkins Token or Crumb
id: np.jenkins.1

pattern: '(?i)jenkins.{0,10}(?:crumb)?.{0,10}\b([0-9a-f]{32,36})\b'
pattern: |
(?x)
(?i)
jenkins.{0,12}(?: (?: crumb | token ) .{0,10} )?
\b
( [0-9a-f]{32,36} | [a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12} )
(?: [^0-9a-f-] | $ )
categories: [api, fuzzy, secret]

Expand All @@ -19,6 +25,9 @@ rules:
export JENKINS=jenkins-cicd.apps.sno.openshiftlabs.net
- |
sh "curl -X POST 'http://jenkins.lsfusion.luxsoft.by/job/${Paths.updateParentVersionsJob}/build' --user ${USERPASS} -H 'Jenkins-Crumb:440561953171ba44ace9740562d172bb'"
- |
// Jenkins query token
readonly static string jenkinsToken = "df490bc6-ef6e-4f6a-99a0-b03d26cb56f9";
negative_examples:
- '1. ~~Does not play well with [Build Token Root Plugin](https://wiki.jenkins-ci.org/display/JENKINS/Build+Token+Root+Plugin) URL formats.~~ (added with [this commit](https://github.com/morficus/Parameterized-Remote-Trigger-Plugin/commit/f687dbe75d1c4f39f7e14b68220890384d7c5674) )'
Expand Down
62 changes: 62 additions & 0 deletions crates/noseyparker/data/default/builtin/rules/jwt.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,65 @@ rules:
it "sets the relation to nil" do
eye.eyeable.should be_nill
end
# This kind of thing is often seen in spring config files and other configs
- name: JSON Web Token Secret
id: np.jwt.2

pattern: |
(?x)(?m)(?i)
^\s*
(?: jwt: \s+ secret: | jwtsecret: )
\s+
"( [^"]{6,200} )"
categories: [fuzzy, secret]

examples:
- |
something_else: true
jwt:
secret: "acb1ca5bc8d7fb2ef9f890f1be16d964" #some comment
ttl: 640000 #
- |
JWT:
secret: "development#product"
exp: "50m"
- |
oauthproviders:
jwtsecret: "7`D'}/vYR)T;W\5]EBrf;_&k>qX??]G&3CF=;@vyrVBp[$_hK4^?{qT/EglJe[FYjp:IPgl_t$%r"
references:
- https://en.wikipedia.org/wiki/JSON_Web_Token


# This kind of thing is often seen in spring config files and other configs
- name: JSON Web Token Secret
id: np.jwt.3

pattern: |
(?x)(?m)(?i)
^\s*
(?: jwt: \s+ secret: | jwtsecret: )
\s+
'( [^']{6,200} )'
categories: [fuzzy, secret]

examples:
- |
something_else: true
jwt:
secret: 'acb1ca5bc8d7fb2ef9f890f1be16d964' #some comment
ttl: 640000 #
- |
JWT:
Secret: 'development#product'
exp: '50m'
- |
oauthproviders:
jwtsecret: '7`D"}/vYR)T;W\5]EBrf;_&k>qX??]G&3CF=;@vyrVBp[$_hK4^?{qT/EglJe[FYjp:IPgl_t$%r'
references:
- https://en.wikipedia.org/wiki/JSON_Web_Token
50 changes: 50 additions & 0 deletions crates/noseyparker/data/default/builtin/rules/mongo.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
rules:

# This handles some cases of MongoDB connection strings, but not the full generality of them.
- name: Credentials in MongoDB Connection String
id: np.mongodb.1

pattern: |
(?x)
(?: mongodb\+srv | mongodb ) :// (?# URI scheme )
([a-zA-Z0-9%;._~!$&'()*+,;=-]{3,}) (?# username)
: ([a-zA-Z0-9%;._~!$&'()*+,;=-]{3,}) (?# password)
@ ([a-zA-Z0-9_.-]{3,} (?: :\d{1,5})?) (?# hostname and port)
(?: [^a-zA-Z0-9_.-] | $ )
categories: [secret]

examples:
- |
spring:
application:
name: Prince
data:
mongodb:
uri: "mongodb+srv://servicelogin:CPCdZH5k0F6dxQ9V@example-prince.mongodb.net/?retryWrites=true&w=majority"
database: prince
- 'mongodb://myDatabaseUser:D1fficult@localhost'
- 'mongodb://myDatabaseUser:D1fficult@localhost/database'
- 'mongodb://myDatabaseUser:D1fficultP%40ssw0rd@localhost'
- 'mongodb://myDatabaseUser:D1fficultP%40ssw0rd@localhost/records'
- 'mongodb+srv://myDatabaseUser:D1fficultP%40ssw0rd@cluster0.example.mongodb.net/?retryWrites=true&w=majority'
- 'mongodb+srv://myDatabaseUser:D1fficultP%40ssw0rd@mongodb0.example.com/?authSource=admin&replicaSet=myRepl'
- 'mongodb://myDatabaseUser:D1fficultP%40ssw0rd@mongodb0.example.com:27017,mongodb1.example.com:27017,mongodb2.example.com:27017/?authSource=admin&replicaSet=myRepl'
- 'mongodb://myDatabaseUser:D1fficultP%40ssw0rd@localhost,localhost:27018,localhost:27019/?replicaSet=test'
- 'mongodb://myDatabaseUser:D1fficultP%40ssw0rd@example1.com,example2.com,example3.com/?replicaSet=test&readPreference=secondary'
- 'mongodb://myDatabaseUser:D1fficultP%40ssw0rd@router1.example.com:27017,router2.example2.com:27017,router3.example3.com:27017/'
- 'mongodb://myDatabaseUser:D1fficultP%40ssw0rd@example1.com,example2.com,example3.com/?replicaSet=test&w=majority&wtimeoutMS=2000'

- |
var store = new MongoDBStore({
uri: 'mongodb+srv://somethingrotten:Zxcvbnm@12345)@cluster0-cnc84.mongodb.net/e-commerce?retryWrites=true',
collection: 'mySessions',
});
- 'mongodb+srv://hpapp:bria@pass135!@briaserver.7xsu17j.mongodb.net/test'

negative_examples:
- 'mongodb://myDatabaseUser:D1fficultP%40ssw0rd@%2Ftmp%2Fmongodb-27017.sock'

references:
- https://www.mongodb.com/docs/manual/reference/connection-string/
Loading

0 comments on commit 8375040

Please sign in to comment.