Tests to check if sorted by rarity, if correct regex format,...

[amadejpapez]

• added 3 bigger tests:
  • check if every regex has a test in test_regex_identifier.py (this requires at least one test with a full regex pattern name because previously a lot of tests asserted just one or two words to match to the name)
  • check if regexes are sorted by rarity in regex.json (fixes Add black + isort check #106)
  • check if regexes have ^(regex)$ format and fail if there are any in-between boundaries (fixes Add black + isort check #106)
  • I have also added meaningfull messages, so if one of these tests fail, users will be able to easily see why and how to fix it
• fixed typo in the Placekey name

[bee-san]

Can you write "fixes #106" that way github will auto-close the issue when this is merged? edit your original description!

[bee-san]

tests fail!

[ghost]

Can you add black and isort check?

[amadejpapez]

Can you add black and isort check?

I am not quite sure how to do that. We have to add them to the workflow file right?

[ghost]

Can you add black and isort check?

I am not quite sure how to do that. We have to add them to the workflow file right?

Yes, you can use this for isort and this one for black.

[amadejpapez]

I have reorder the database as some some API keys had rarity of 1 and they were not sorted by rarity which was the reason why my tests failed - so rarity and regex format tests work :)

Here is how regexes are sorted now and what criteria I used:

1 (matching to specific words and characters)

PGP Public Key, PGP Private Key, SSH RSA Public Key, SSH ECDSA Public Key, SSH ED25519 Public Key, Access-Control-Allow-Header, TryHackMe Flag Format, HackTheBox Flag Format, Capture The Flag (CTF) Flag, YouTube Video, Bitcoin Cash (BCH) Wallet Address, Amazon Web Services Simple Storage (AWS S3) URL, Amazon Web Services Simple Storage (AWS S3) Internal URL, Square Application Secret, Square Access Token, Stripe API Key, GitHub Access Token, Amazon Resource Name (ARN)

0.7 (very restricted regex, can still cause false positives)

Internet Protocol (IP) Address Version 6, Uniform Resource Locator (URL), Internet Protocol (IP) Address Version 4, Bitcoin (₿) Wallet Address, Latitude & Longitude Coordinates

0.5 (mostly matching to specific characters)

EUI-48 Identifier (Ethernet, WiFi, Bluetooth, etc), Placekey Universal Identifier for Physical Place, Dogecoin (DOGE) Wallet Address, Monero (XMR) Wallet Address, Email Address, Phone Number, American Social Security Number, YouTube Channel ID

0.3 (pretty broad, only a few specific characters)

Monero (XMR) Wallet Address, Litecoin (LTC) Wallet Address, Ripple (XRP) Wallet Address, American Express Card Number, BCGlobal Card Number, Carte Blanche Card Number, Diners Club Card Number, Discover Card Number, MasterCard Number, Maestro Card Number, Visa Card Number, Insta Payment Card Number, JCB Card Number, Korean Local Card Number, Laser Card Number, Solo Card Number, Switch Card Number, Ethereum (ETH) Wallet Address, Amazon Web Services EC2 Security Group identifier, Slack Token, Amazon Web Services Organization identifier, Google API Key, Google OAuth Token, Mailgun API Key, Twilio API Key, Twilio Account SID, Twilio Application SID, Google ReCaptcha API Key, Amazon Standard Identification Number (ASIN)

0.2 (very broad, no specific characters, only length limits)

JSON Web Token (JWT), Amazon Web Services Access Key, Amazon Web Services Secret Access Key, Amazon Web Services EC2 Instance identifier, YouTube Video ID, Recent Unix Timestamp, Recent Unix Millisecond Timestamp, Unix Timestamp, Unix Millisecond Timestamp

0 (madness)

Key:Value Pair

[ghost]

Looks good, but I am waiting for black and isort check 😄

[bee-san]

Looks good, but I am waiting for black and isort check

https://github.com/olance/isort-action
https://github.com/rickstaa/action-black/

For the actions :)