Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev -> Stable 2.1.1 #1885

Open
wants to merge 93 commits into
base: stable
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
126c1c6
enable manual trigger
invalid-email-address Oct 18, 2024
f2b70ce
Update nuclei
blsaccess Oct 20, 2024
2c0ebb6
Bump pytest-asyncio from 0.23.8 to 0.24.0
dependabot[bot] Oct 21, 2024
f76e688
Bump mkdocs-material from 9.5.41 to 9.5.42
dependabot[bot] Oct 21, 2024
ec67fa4
Bump xmltojson from 2.0.2 to 2.0.3
dependabot[bot] Oct 21, 2024
6a5303d
Bump mkdocstrings-python from 1.12.1 to 1.12.2
dependabot[bot] Oct 21, 2024
1b03b70
Merge pull request #1878 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Oct 21, 2024
a4b9824
Merge pull request #1882 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Oct 21, 2024
ba10b9e
Merge pull request #1881 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Oct 21, 2024
6250877
Merge pull request #1880 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Oct 21, 2024
9dc05bb
Bump psutil from 5.9.8 to 6.1.0
dependabot[bot] Oct 21, 2024
20aaf68
resolve conflicts
invalid-email-address Oct 18, 2024
f55c004
resolve conflicts
invalid-email-address Oct 18, 2024
6be3f23
simplified run_tests.sh
TheTechromancer Jun 18, 2024
c08648e
resolve conflicts
invalid-email-address Oct 18, 2024
c2e69e8
runs-on --> container: image
invalid-email-address Oct 18, 2024
6875793
fix python setup
invalid-email-address Oct 18, 2024
047b9ff
no sudo
invalid-email-address Oct 21, 2024
3c6728c
python things
invalid-email-address Oct 21, 2024
0572ce6
quotes
invalid-email-address Oct 21, 2024
5a6e68e
testing stuff
invalid-email-address Oct 21, 2024
ffe8ae1
pipx
invalid-email-address Oct 21, 2024
46f65a4
onwards
invalid-email-address Oct 21, 2024
50969eb
distro stuff
invalid-email-address Oct 21, 2024
b30cbaf
more
invalid-email-address Oct 21, 2024
262d5d7
just ubuntu things
invalid-email-address Oct 21, 2024
cc296b7
tests
invalid-email-address Oct 21, 2024
97ce06b
iterating
invalid-email-address Oct 21, 2024
3036629
bash oopsie
invalid-email-address Oct 21, 2024
687d247
distrosgit diff
invalid-email-address Oct 21, 2024
c763c1e
alpine
invalid-email-address Oct 21, 2024
16f204c
ubuntu things
invalid-email-address Oct 21, 2024
10627df
more ubuntu things
invalid-email-address Oct 21, 2024
e594c3f
Merge pull request #1879 from blacklanternsecurity/dependabot/pip/dev…
TheTechromancer Oct 21, 2024
f2cdcb8
Merge pull request #1877 from blacklanternsecurity/update-nuclei
TheTechromancer Oct 21, 2024
f3b90e5
better debian
invalid-email-address Oct 21, 2024
0a6e78b
ubuntu why?
invalid-email-address Oct 21, 2024
838c781
ubuntu why?
invalid-email-address Oct 21, 2024
782a68d
ubuntu why?
invalid-email-address Oct 21, 2024
8f3ba61
ubuntu why?
invalid-email-address Oct 21, 2024
cdfbe7e
skip dastardly
invalid-email-address Oct 21, 2024
ce31264
pyenv
invalid-email-address Oct 21, 2024
1eedaf5
pyenv
invalid-email-address Oct 21, 2024
bb451fd
curl
invalid-email-address Oct 21, 2024
60a418e
git
invalid-email-address Oct 21, 2024
32abe3b
bash
invalid-email-address Oct 21, 2024
b9d40bf
dev
invalid-email-address Oct 21, 2024
51fd792
path
invalid-email-address Oct 21, 2024
f788dad
pyenv things
invalid-email-address Oct 21, 2024
73b00f4
alpine
invalid-email-address Oct 21, 2024
cda7d50
Merge pull request #1476 from blacklanternsecurity/multi-distro-tests
TheTechromancer Oct 22, 2024
51e745a
bufferover.run module
invalid-email-address Oct 23, 2024
6821a1e
add test
invalid-email-address Oct 23, 2024
ee078b3
update docs workflow
invalid-email-address Oct 23, 2024
84cc1a2
Merge pull request #1888 from blacklanternsecurity/bufferoverrun
TheTechromancer Oct 24, 2024
e08e385
bump versions
invalid-email-address Oct 24, 2024
17160e7
continue on error
invalid-email-address Oct 24, 2024
30320b6
don't continue on error
invalid-email-address Oct 24, 2024
07a0713
silence stderr in tests
invalid-email-address Oct 26, 2024
dd7884e
upload debug logs
invalid-email-address Oct 26, 2024
7da0c63
troubleshooting ubuntu 20.04 test
invalid-email-address Oct 27, 2024
9e90572
test things
invalid-email-address Oct 27, 2024
0441155
ruby things
invalid-email-address Oct 27, 2024
5eded28
more test things
invalid-email-address Oct 27, 2024
95187e8
ansible things
invalid-email-address Oct 27, 2024
614dcd5
ubuntu why
invalid-email-address Oct 27, 2024
cfbd211
alpine
invalid-email-address Oct 27, 2024
991b531
bye ubuntu
invalid-email-address Oct 27, 2024
f44983f
all tests
invalid-email-address Oct 27, 2024
1f71c58
fix chrome in fedora
invalid-email-address Oct 27, 2024
f4bbd88
goodbye gentoo, goodbye alpine
invalid-email-address Oct 27, 2024
8a8d33e
flaked
invalid-email-address Oct 27, 2024
5360050
Merge pull request #1892 from blacklanternsecurity/better-logging
TheTechromancer Oct 28, 2024
32c4f7e
fixing pytest summary
liquidsec Oct 28, 2024
24676a6
showcapture no
invalid-email-address Oct 28, 2024
d348d96
silence live logging
invalid-email-address Oct 28, 2024
905cc04
blacked
invalid-email-address Oct 28, 2024
3f3c43b
test tests
invalid-email-address Oct 28, 2024
ffb8c09
remove thing
invalid-email-address Oct 28, 2024
38d3f49
flaked
invalid-email-address Oct 28, 2024
16e82e8
done
invalid-email-address Oct 28, 2024
dc54677
Merge pull request #1894 from blacklanternsecurity/logging-restore-su…
TheTechromancer Oct 28, 2024
0207fd0
stop logger on exit
invalid-email-address Oct 28, 2024
ad88024
better log cleanup
invalid-email-address Oct 28, 2024
d83dba4
Merge pull request #1891 from blacklanternsecurity/stable-update
TheTechromancer Oct 28, 2024
60ab921
rename tests
invalid-email-address Oct 28, 2024
e8c27f4
don't skip dastardly tests
invalid-email-address Oct 28, 2024
1837f0a
unskip dastardly tests
invalid-email-address Oct 28, 2024
52242dd
Merge branch 'dev' into excavate-intercept
TheTechromancer Oct 29, 2024
de740f6
Merge pull request #1896 from blacklanternsecurity/test-cleanup
TheTechromancer Oct 29, 2024
ae55da4
Merge pull request #1873 from blacklanternsecurity/excavate-intercept
TheTechromancer Oct 29, 2024
88a9cf9
Update trufflehog
blsaccess Oct 30, 2024
9e53c38
Merge pull request #1883 from blacklanternsecurity/update-trufflehog
TheTechromancer Oct 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
select = F,E722
ignore = F403,F405,F541
per-file-ignores =
*/__init__.py:F401,F403
*/__init__.py:F401,F403
64 changes: 64 additions & 0 deletions .github/workflows/distro_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: Tests (Linux Distros)
on:
pull_request:

concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
test-distros:
runs-on: ubuntu-latest
container:
image: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: ["ubuntu:22.04", "ubuntu:24.04", "debian", "archlinux", "fedora", "kalilinux/kali-rolling", "parrotsec/security"]
steps:
- uses: actions/checkout@v4
- name: Install Python and Poetry
run: |
if [ -f /etc/os-release ]; then
. /etc/os-release
if [ "$ID" = "ubuntu" ] || [ "$ID" = "debian" ] || [ "$ID" = "kali" ] || [ "$ID" = "parrotsec" ]; then
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get -y install curl git bash build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev
elif [ "$ID" = "alpine" ]; then
apk add --no-cache bash gcc g++ musl-dev libffi-dev curl git make openssl-dev bzip2-dev zlib-dev xz-dev sqlite-dev
elif [ "$ID" = "arch" ]; then
pacman -Syu --noconfirm curl git bash base-devel
elif [ "$ID" = "fedora" ]; then
dnf install -y curl git bash gcc make openssl-devel bzip2-devel libffi-devel zlib-devel xz-devel tk-devel gdbm-devel readline-devel sqlite-devel
elif [ "$ID" = "gentoo" ]; then
echo "media-libs/libglvnd X" >> /etc/portage/package.use/libglvnd
emerge-webrsync
emerge --update --newuse dev-vcs/git media-libs/mesa curl bash
fi
fi

# Re-run the script with bash
exec bash -c "
curl https://pyenv.run | bash
export PATH=\"$HOME/.pyenv/bin:\$PATH\"
export PATH=\"$HOME/.local/bin:\$PATH\"
eval \"\$(pyenv init --path)\"
eval \"\$(pyenv init -)\"
eval \"\$(pyenv virtualenv-init -)\"
pyenv install 3.11
pyenv global 3.11
pyenv rehash
python3.11 -m pip install --user pipx
python3.11 -m pipx ensurepath
pipx install poetry
"
- name: Run tests
run: |
export PATH="$HOME/.local/bin:$PATH"
export PATH="$HOME/.pyenv/bin:$PATH"
export PATH="$HOME/.pyenv/shims:$PATH"
export BBOT_DISTRO_TESTS=true
poetry env use python3.11
poetry install
poetry run pytest --reruns 2 -o timeout_func_only=true --timeout 1200 --disable-warnings --log-cli-level=INFO .
23 changes: 14 additions & 9 deletions .github/workflows/docs_updater.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ name: Daily Docs Update

on:
schedule:
- cron: '0 0 * * *' # Runs daily at midnight UTC
- cron: '30 2 * * *' # Runs daily at 2:30 AM UTC, a less congested time
workflow_dispatch: # Allows manual triggering of the workflow

jobs:
update_docs:
Expand All @@ -23,18 +24,22 @@ jobs:
- name: Generate docs
run: |
poetry run bbot/scripts/docs.py
- name: Commit changes
uses: EndBug/add-and-commit@v9
with:
add: '["*.md", "docs/data/chord_graph/*.json"]'
author_name: "BBOT Docs Autopublish"
author_email: info@blacklanternsecurity.com
message: "Refresh module docs"
- name: Create Pull Request
- name: Commit and Push Changes
run: |
git config user.name "BBOT Docs Autopublish"
git config user.email "info@blacklanternsecurity.com"
git checkout -b update-docs
git add "*.md" "docs/data/chord_graph/*.json"
git commit -m "Refresh module docs"
git push -u origin update-docs --force
env:
GITHUB_TOKEN: ${{ secrets.BBOT_DOCS_UPDATER_PAT }}
- name: Create or Update Pull Request
uses: peter-evans/create-pull-request@v7
with:
token: ${{ secrets.BBOT_DOCS_UPDATER_PAT }}
branch: update-docs
base: dev
title: "Daily Docs Update"
body: "This is an automated pull request to update the documentation."
update-existing: true
96 changes: 15 additions & 81 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: tests
name: Tests (Python Versions)
on:
push:
branches:
Expand All @@ -14,20 +14,20 @@ jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: psf/black@stable
with:
options: "--check"
- name: Install Python 3
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.x"
- name: Install dependencies
run: |
pip install flake8
- name: flake8
run: |
flake8 --select F,E722 --ignore F403,F405,F541 --per-file-ignores="*/__init__.py:F401,F403"
flake8
test:
needs: lint
runs-on: ubuntu-latest
Expand All @@ -37,9 +37,9 @@ jobs:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12"]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
Expand All @@ -48,95 +48,29 @@ jobs:
poetry install
- name: Run tests
run: |
poetry run pytest --exitfirst --reruns 2 -o timeout_func_only=true --timeout 1200 --disable-warnings --log-cli-level=DEBUG --cov-config=bbot/test/coverage.cfg --cov-report xml:cov.xml --cov=bbot .
poetry run pytest --exitfirst --reruns 2 -o timeout_func_only=true --timeout 1200 --disable-warnings --log-cli-level=INFO --cov-config=bbot/test/coverage.cfg --cov-report xml:cov.xml --cov=bbot .
- name: Upload Debug Logs
uses: actions/upload-artifact@v3
with:
name: pytest-debug-logs
path: pytest_debug.log
- name: Upload Code Coverage
uses: codecov/codecov-action@v3
with:
token: ${{ secrets.CODECOV_TOKEN }}
files: ./cov.xml
verbose: true
update_docs:
needs: test
runs-on: ubuntu-latest
if: github.event_name == 'push' && (github.ref != 'refs/heads/dev' && github.ref != 'refs/heads/stable')
steps:
- uses: actions/checkout@v3
with:
token: ${{ secrets.BBOT_DOCS_UPDATER_PAT }}
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.x"
- name: Install dependencies
run: |
pip install poetry
poetry install
- name: Generate docs
run: |
poetry run bbot/scripts/docs.py
- name: Commit docs
uses: EndBug/add-and-commit@v9
continue-on-error: true
with:
add: '["*.md", "docs/data/chord_graph/*.json"]'
author_name: "BBOT Docs Autopublish"
author_email: info@blacklanternsecurity.com
message: "Refresh module docs"
publish_docs:
needs: test
runs-on: ubuntu-latest
if: github.event_name == 'push' && (github.ref == 'refs/heads/stable' || github.ref == 'refs/heads/dev')
steps:
- uses: actions/checkout@v3
with:
token: ${{ secrets.BBOT_DOCS_UPDATER_PAT }}
- uses: actions/setup-python@v4
with:
python-version: "3.x"
- run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
- uses: actions/cache@v3
with:
key: mkdocs-material-${{ env.cache_id }}
path: .cache
restore-keys: |
mkdocs-material-
- name: Install dependencies
run: |
pip install poetry
poetry install --only=docs
- name: Configure Git
run: |
git config user.name github-actions
git config user.email github-actions@github.com
git fetch origin gh-pages:refs/remotes/origin/gh-pages
if git show-ref --verify --quiet refs/heads/gh-pages; then
git branch -f gh-pages origin/gh-pages
else
git branch --track gh-pages origin/gh-pages
fi
- name: Generate docs (stable branch)
if: github.ref == 'refs/heads/stable'
run: |
poetry run mike deploy Stable
- name: Generate docs (dev branch)
if: github.ref == 'refs/heads/dev'
run: |
poetry run mike deploy Dev
- name: Publish docs
run: |
git switch gh-pages
git push
publish_code:
needs: test
runs-on: ubuntu-latest
if: github.event_name == 'push' && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/stable')
continue-on-error: true
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.x"
- name: Install dependencies
Expand Down Expand Up @@ -185,7 +119,7 @@ jobs:
# runs-on: ubuntu-latest
# if: github.event_name == 'push' && github.ref == 'refs/heads/stable'
# steps:
# - uses: actions/checkout@v3
# - uses: actions/checkout@v4
# with:
# ref: ${{ github.head_ref }}
# fetch-depth: 0 # Fetch all history for all tags and branches
Expand Down
29 changes: 28 additions & 1 deletion bbot/core/config/logger.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import multiprocessing
import logging.handlers
from pathlib import Path
from contextlib import suppress

from ..helpers.misc import mkdir, error_and_exit
from ...logger import colorize, loglevel_mapping
Expand Down Expand Up @@ -71,10 +72,36 @@ def __init__(self, core):
# Start the QueueListener
self.listener = logging.handlers.QueueListener(self.queue, *self.log_handlers.values())
self.listener.start()
atexit.register(self.listener.stop)
atexit.register(self.cleanup_logging)

self.log_level = logging.INFO

def cleanup_logging(self):
# Close the queue handler
with suppress(Exception):
self.queue_handler.close()

# Clean root logger
root_logger = logging.getLogger()
for handler in list(root_logger.handlers):
with suppress(Exception):
root_logger.removeHandler(handler)
with suppress(Exception):
handler.close()

# Clean all other loggers
for logger in logging.Logger.manager.loggerDict.values():
if hasattr(logger, "handlers"): # Logger, not PlaceHolder
for handler in list(logger.handlers):
with suppress(Exception):
logger.removeHandler(handler)
with suppress(Exception):
handler.close()

# Stop queue listener
with suppress(Exception):
self.listener.stop()

def setup_queue_handler(self, logging_queue=None, log_level=logging.DEBUG):
if logging_queue is None:
logging_queue = self.queue
Expand Down
10 changes: 9 additions & 1 deletion bbot/core/helpers/depsinstaller/installer.py
Original file line number Diff line number Diff line change
Expand Up @@ -342,7 +342,15 @@ def install_core_deps(self):
# ensure tldextract data is cached
self.parent_helper.tldextract("evilcorp.co.uk")
# command: package_name
core_deps = {"unzip": "unzip", "curl": "curl"}
core_deps = {
"unzip": "unzip",
"zipinfo": "unzip",
"curl": "curl",
"git": "git",
"make": "make",
"gcc": "gcc",
"bash": "bash",
}
for command, package_name in core_deps.items():
if not self.parent_helper.which(command):
to_install.add(package_name)
Expand Down
48 changes: 48 additions & 0 deletions bbot/modules/bufferoverrun.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
from bbot.modules.templates.subdomain_enum import subdomain_enum_apikey


class BufferOverrun(subdomain_enum_apikey):
watched_events = ["DNS_NAME"]
produced_events = ["DNS_NAME"]
flags = ["subdomain-enum", "passive", "safe"]
meta = {
"description": "Query BufferOverrun's TLS API for subdomains",
"created_date": "2024-10-23",
"author": "@TheTechromancer",
"auth_required": True,
}
options = {"api_key": "", "commercial": False}
options_desc = {"api_key": "BufferOverrun API key", "commercial": "Use commercial API"}

base_url = "https://tls.bufferover.run/dns"
commercial_base_url = "https://bufferover-run-tls.p.rapidapi.com/ipv4/dns"

async def setup(self):
self.commercial = self.config.get("commercial", False)
return await super().setup()

def prepare_api_request(self, url, kwargs):
if self.commercial:
kwargs["headers"]["x-rapidapi-host"] = "bufferover-run-tls.p.rapidapi.com"
kwargs["headers"]["x-rapidapi-key"] = self.api_key
else:
kwargs["headers"]["x-api-key"] = self.api_key
return url, kwargs

async def request_url(self, query):
url = f"{self.commercial_base_url if self.commercial else self.base_url}?q=.{query}"
return await self.api_request(url)

def parse_results(self, r, query):
j = r.json()
subdomains_set = set()
if isinstance(j, dict):
results = j.get("Results", [])
for result in results:
parts = result.split(",")
if len(parts) > 4:
subdomain = parts[4].strip()
if subdomain and subdomain.endswith(f".{query}"):
subdomains_set.add(subdomain)
for subdomain in subdomains_set:
yield subdomain
2 changes: 1 addition & 1 deletion bbot/modules/deadly/nuclei.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ class nuclei(BaseModule):
}

options = {
"version": "3.3.4",
"version": "3.3.5",
"tags": "",
"templates": "",
"severity": "",
Expand Down
2 changes: 1 addition & 1 deletion bbot/modules/gowitness.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ async def setup(self):

# make sure we have a working chrome install
chrome_test_pass = False
for binary in ("chrome", "chromium", custom_chrome_path):
for binary in ("chrome", "chromium", "chromium-browser", custom_chrome_path):
binary_path = self.helpers.which(binary)
if binary_path and Path(binary_path).is_file():
chrome_test_proc = await self.run_process([binary_path, "--version"])
Expand Down
Loading
Loading