Modules not found when using BBOT in Arch Linux

[D3vil0p3r]

Describe the bug
I am packaging BBOT for BlackArch and I installed all Python dependencies as sudo pacman -S python-<modulename> as usual.

When I launch bbot, it recognizes Python modules used directly in the code, but modules used by bbot during the scan/enumeration cannot be found.

Indeed, when I launch bbot -t evilcorp.com -p subdomain-enum -v, I get:

  ______  _____   ____ _______
 |  ___ \|  __ \ / __ \__   __|
 | |___) | |__) | |  | | | |
 |  ___ <|  __ <| |  | | | |
 | |___) | |__) | |__| | | |
 |______/|_____/ \____/  |_|
 BIGHUGE BLS OSINT TOOL v2.2.0

www.blacklanternsecurity.com/bbot

[VERB] Preset : Adding module "httpx" because baddns_direct depends on it
[VERB] Preset : Adding module "social" because github_org depends on it
[VERB] Creating events from 1 targets
[VERB] 
[VERB] ### MODULES ENABLED ###
.... MODULE TABLE
[VERB] Loading word cloud from /home/athena/.bbot/scans/distilled_tiffany/wordcloud.tsv
[INFO] Scan with 52 modules seeded with 1 targets (1 in whitelist)
[INFO] Installing module dependencies. Please be patient, this may take a while.
[VERB] Installing dependencies for module "httpx"
[INFO] Running 1 Ansible tasks for httpx
[INFO] Successfully ran 1 Ansible tasks for httpx
[VERB] Installing dependencies for module "baddns_direct"
[INFO] Installing the following pip packages: baddns~=1.4.13
[VERB] run: /usr/bin/python -m pip install --upgrade baddns~=1.4.13
[WARN] Failed to install pip packages baddns~=1.4.13 (return code 1): error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

[WARN] Setup failed for module "baddns_direct"
[VERB] Installing dependencies for module "dnsbrute_mutations"
[INFO] Running 5 Ansible tasks for dnsbrute_mutations
[INFO] Successfully ran 5 Ansible tasks for dnsbrute_mutations
[VERB] Installing dependencies for module "sslcert"
[INFO] Installing the following OS packages: openssl
[WARN] Failed to install OS packages (Could not find a matching action for the "pacman" package manager.). Recommend installing the following packages manually:
[WARN]  - openssl
[INFO] Installing the following pip packages: pyOpenSSL~=24.0.0
[VERB] run: /usr/bin/python -m pip install --upgrade pyOpenSSL~=24.0.0
[WARN] Failed to install pip packages pyOpenSSL~=24.0.0 (return code 1): error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

[WARN] Setup failed for module "sslcert"
[VERB] Installing dependencies for module "postman"
[VERB] Installing dependencies for module "baddns_zone"
[INFO] Installing the following pip packages: baddns~=1.4.13
[VERB] run: /usr/bin/python -m pip install --upgrade baddns~=1.4.13
[WARN] Failed to install pip packages baddns~=1.4.13 (return code 1): error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

[WARN] Setup failed for module "baddns_zone"
[ERRR] Failed to install dependencies for 3 modules: baddns_direct,baddns_zone,sslcert (--force to run module anyway)

On Arch I installed baddns from source by the latest commit:

  __ )              |      |              
  __ \    _` |   _` |   _` |  __ \    __| 
  |   |  (   |  (   |  (   |  |   | \__ \ 
 ____/  \__,_| \__,_| \__,_| _|  _| ____/ 
                                          

Version - 1.4.0

Note: on baddns, I don't understand why it outputs 1.4.0 whereas I installed the latest commit that should be 1.4.13.r0.g0c8d6e9, I think this is a bug on baddns.

I installed BadDNS by PKGBUILD I created in Arch for packaging it and it works well.

Btw, I also installed openssl on my Arch:

openssl --version
OpenSSL 3.4.0 22 Oct 2024 (Library: OpenSSL 3.4.0 22 Oct 2024)

and python-pyopenssl (version 24.2.1-2).

Why is BBOT not able to find these modules? Is it searching on wrong library path instead of /usr/lib?

I am using Python 3.12 and there I have:

/usr/lib/python3.12/site-packages/baddns/
/usr/lib/python3.12/site-packages/OpenSSL/

Expected behavior
BBOT can find

BBOT Command
bbot -t evilcorp.com -p subdomain-enum

OS, BBOT Installation Method + Version
OS: Arch Linux
Installation method: PKGBUILD for packaging BBOT in Arch by:

build() {
  cd $pkgname

  python -m build --wheel --outdir="$startdir/dist"
}

package() {
  cd $pkgname

  pip install \
    --verbose \
    --disable-pip-version-check \
    --no-warn-script-location \
    --ignore-installed \
    --no-compile \
    --no-deps \
    --root="$pkgdir" \
    --prefix=/usr \
    --no-index \
    --find-links="file://$startdir/dist" \
    $pkgname
}

BBOT version: latest commit 95d1bc565d90f9dea92f0c6283d7976972ac11e2

BBOT Config
Attach your BBOT config (bbot --current-config).

I get bbot: error: unrecognized arguments: --current-config.

Logs
If possible, produce the bug while --debug is enabled, and attach the relevant parts of ~/.bbot/logs/bbot.debug.log

2024-11-24 22:49:50,102 [TRACE] bbot.cli logger.py:159 Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/bbot/cli.py", line 169, in _main
    await scan._prep()
  File "/usr/lib/python3.12/site-packages/bbot/scanner/scanner.py", line 281, in _prep
    await self.load_modules()
  File "/usr/lib/python3.12/site-packages/bbot/scanner/scanner.py", line 546, in load_modules
    self._fail_setup(msg)
  File "/usr/lib/python3.12/site-packages/bbot/scanner/scanner.py", line 1243, in _fail_setup
    raise ScanError(msg)
bbot.errors.ScanError: Failed to install dependencies for 3 modules: baddns_direct,baddns_zone,sslcert (--force to run module anyway)

[D3vil0p3r]

I guess that BBOT manages the search of modules in a not flexible manner:

    async def install_module(self, module):
        success = True
        preloaded = self.all_modules_preloaded[module]

        # ansible tasks
        ansible_tasks = preloaded["deps"]["ansible"]
        if ansible_tasks:
            success &= self.tasks(module, ansible_tasks)

        # apt
        deps_apt = preloaded["deps"]["apt"]
        if deps_apt:
            self.apt_install(deps_apt)

        # shell
        deps_shell = preloaded["deps"]["shell"]
        if deps_shell:
            success &= self.shell(module, deps_shell)

        # pip
        deps_pip = preloaded["deps"]["pip"]
        deps_pip_constraints = preloaded["deps"]["pip_constraints"]
        if deps_pip:
            success &= await self.pip_install(deps_pip, constraints=deps_pip_constraints)

        # shared/common
        deps_common = preloaded["deps"]["common"]
        if deps_common:
            for dep_common in deps_common:
                if self.setup_status.get(dep_common, False) == True:
                    log.debug(
                        f'Skipping installation of dependency "{dep_common}" for module "{module}" since it is already installed'
                    )
                    continue
                ansible_tasks = self.preset.module_loader._shared_deps[dep_common]
                result = self.tasks(module, ansible_tasks)
                self.setup_status[dep_common] = result
                success &= result

        return success

It means that it would work only inside a Python virtual environment or in a Debian-based system.

Arch installs python modules by pacman so BBOT won't detect modules during scanning. Adding pacman ref in the code as done with apt could make it working, but I think that finding a more general solution to detect modules during scanning could be better and more flexible.

[TheTechromancer]

Since BBOT and its modules require a wide range of python packages (some with very specific versions), it's crucial it runs inside its own virtual environment. This is why the recommended installation method is Pipx.

That being said, we're primarily an Arch shop here at Black Lantern, so getting BBOT into the blackarch repos would be nice. Is there a process for packaging BBOT inside its own python venv?

[TheTechromancer]

Arch installs python modules by pacman so BBOT won't detect modules during scanning

Despite the apt naming, BBOT supports many different Linux distros including Arch (we have dedicated tests for it). This works by using the Ansible package module.

Any pip dependencies are automatically installed inside the current virtual env.

[D3vil0p3r]

Once back home, I will give a look. Thanks for the info

[D3vil0p3r]

That being said, we're primarily an Arch shop here at Black Lantern, so getting BBOT into the blackarch repos would be nice. Is there a process for packaging BBOT inside its own python venv?

It could be a possibility but I need to understand what is the first running Python script when BBOT is executed, because I should wrap it to execute in a virtual environment.

[D3vil0p3r]

I'm trying by wrapping bbot/cli.py by guessing it is the starting Python script. Btw, during the package installation, the following post install commands are run:

  set -e
  cd /usr/share/bbot
  python -m venv venv
  source venv/bin/activate &&
    pip install --isolated --root="/usr/share/bbot" --prefix='venv' .

but the output produces:

      RuntimeError: Unable to detect version control system. Checked: Git. Not installed: Mercurial, Darcs, Subversion, Bazaar, Fossil, Pijul.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.

Probably it occurs because BBOT uses Poetry Dynamic Versioning to get BBOT version, but it works only if we are in a git cloned directory (i.e., containing .git), but it cannot be our case because during packaging we cannot move to the target install directory the .git files.

In pyproject.toml since you already use version variable, probably the poetry_dynamic_versioning code could be removed:

[build-system]
requires = ["poetry-core>=1.0.0", "poetry-dynamic-versioning"]
build-backend = "poetry_dynamic_versioning.backend"

to prevent the mentioned issue.

[TheTechromancer]

Okay, currently how we have it set up is the __version__ variable in bbot/__init__.py is overwritten when the package is published to Pypi:

This effectively removes its dependency on Poetry.

For your PKGBUILD are you pulling from github or pypi? Since we have autopublishing set up for both our dev and stable branches, it might make sense to pip install directly from pypi, using the --pre flag if you want the dev branch.

If you need to pull directly from github, the build process will be a bit more involved, but you can use our tests.yml as a cheat sheet:

[D3vil0p3r]

Currently the PKGBUILD for using virtual environment is:

# This file is part of BlackArch Linux ( https://www.blackarch.org/ ).
# See COPYING for license details.

pkgname=bbot
pkgver=5246.95d1bc5
pkgrel=1
pkgdesc='Multipurpose scanner built to automate your Recon, Bug Bounties, and ASM.'
arch=('any')
groups=('blackarch' 'blackarch-recon')
url='https://github.com/blacklanternsecurity/bbot'
license=('GPL-3.0-or-later')
depends=('python')
makedepends=('git' 'python-setuptools' 'python-pip')
source=("git+https://github.com/blacklanternsecurity/$pkgname.git")
sha512sums=('SKIP')
install="$pkgname.install"

pkgver() {
  cd $pkgname

  ( set -o pipefail
    git describe --long --tags --abbrev=7 2>/dev/null |
      sed 's/\([^-]*-g\)/r\1/;s/-/./g' ||
    printf "%s.%s" "$(git rev-list --count HEAD)" \
      "$(git rev-parse --short=7 HEAD)"
  )
}

package() {
  cd $pkgname

  install -dm 755 "$pkgdir/usr/"{bin,share/$pkgname}
  install -Dm 644 -t "$pkgdir/usr/share/doc/$pkgname/" *.md docs/*.md

  rm -rf LICENSE *.md docs/*.md .gitignore

  cp -a * "$pkgdir/usr/share/$pkgname/"

  cat > "$pkgdir/usr/bin/$pkgname" << EOF
#!/bin/sh
source /usr/share/$pkgname/venv/bin/activate
exec python /usr/share/$pkgname/bbot/cli.py "\$@"
EOF

  chmod a+x "$pkgdir/usr/bin/$pkgname"
}

and the post install script bbot.install:

post_install() {
  set -e
  cd /usr/share/bbot
  python -m venv venv
  source venv/bin/activate &&
    export POETRY_DYNAMIC_VERSIONING=0 &&
    pip install --isolated --root="/usr/share/bbot" --prefix='venv' .
}

post_upgrade() {
  post_install "$@"
}

[TheTechromancer]

Okay, the build process will need to look something like:

• make sure the repo is fetched to depth 0 (version is calculated based on the number of git commits)
• python -m build

$ python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - poetry-core>=1.0.0
  - poetry-dynamic-versioning
* Getting build dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - poetry-core>=1.0.0
  - poetry-dynamic-versioning
* Getting build dependencies for wheel...
* Building wheel...
Successfully built bbot-2.3.0.tar.gz and bbot-2.3.0-py3-none-any.whl

• pip install <bbot_tar_file>

[TheTechromancer]

Note that if you're pulling from dev or stable (the default), this will build an identical copy to what's already in Pypi. So if you don't mind relying on the Pypi repos as an upstream, you could save a lot of work by installing directly, e.g. pip install bbot.

[D3vil0p3r]

Yeah, it was the first version of PKGBUILD I created (that comes from BlackArch PKGBUILD templates) as:

# This file is part of BlackArch Linux ( https://www.blackarch.org/ ).
# See COPYING for license details.

pkgname=bbot
pkgver=5246.95d1bc5
pkgrel=1
pkgdesc='Multipurpose scanner built to automate your Recon, Bug Bounties, and ASM.'
arch=('any')
groups=('blackarch' 'blackarch-recon')
url='https://github.com/blacklanternsecurity/bbot'
license=('GPL-3.0-or-later')
depends=('python' 'python-ansible' 'python-beautifulsoup4' 'python-cachetools'
         'python-deepdiff' 'python-dnspython' 'python-httpx' 'python-idna'
         'python-jinja' 'python-lxml' 'python-mmh3' 'python-omegaconf'
         'python-psutil' 'python-puremagic' 'python-pycryptodome'
         'python-pydantic' 'python-pyopenssl' 'python-pyjwt' 'python-pyzmq'
         'python-radixtarget' 'python-regex' 'python-setproctitle'
         'python-socksio' 'python-tabulate' 'python-tldextract'
         'python-unidecode' 'python-websockets' 'python-wordninja'
         'python-yara-python-dex' 'ansible-runner' 'baddns' 'cloudcheck'
         'xmltojson')
makedepends=('git' 'python-build' 'python-pip')
source=("git+https://github.com/blacklanternsecurity/$pkgname.git")
sha512sums=('SKIP')

pkgver() {
  cd $pkgname

  ( set -o pipefail
    git describe --long --tags --abbrev=7 2>/dev/null |
      sed 's/\([^-]*-g\)/r\1/;s/-/./g' ||
    printf "%s.%s" "$(git rev-list --count HEAD)" \
      "$(git rev-parse --short=7 HEAD)"
  )
}

build() {
  cd $pkgname

  python -m build --wheel --outdir="$startdir/dist"
}

package() {
  cd $pkgname

  pip install \
    --verbose \
    --disable-pip-version-check \
    --no-warn-script-location \
    --ignore-installed \
    --no-compile \
    --no-deps \
    --root="$pkgdir" \
    --prefix=/usr \
    --no-index \
    --find-links="file://$startdir/dist" \
    $pkgname
}

but in this case the dependencies must be installed as Arch packages because of --no-deps. Look that no virtual env can be used by this approach so for this reason dependencies must be installed by Arch packages to prevent conflicts with pkgs installed by pip.

[D3vil0p3r]

I prefer this last method but the problem was that scanner cannot find baddns and openssl.

[TheTechromancer]

Now that I'm looking at it, you're totally right that openssl needs to be a project-level dependency. I'll work on fixing that.

As for hardcoding the pip dependencies outside of 'pyproject.toml`, it's risky because we have very specific versions pinned there, and there are likely to more dependencies added/removed in the future.

Really it's absolutely necessary that bbot is installed inside its own venv. After that there shouldn't be any issues/conflicts with OS-level pip packages, and it shouldn't have any problem installing module-specific dependencies like baddns.

[D3vil0p3r]

As for hardcoding the pip dependencies outside of 'pyproject.toml`, it's risky because we have very specific versions pinned there, and there are likely to more dependencies added/removed in the future.

It is the Arch nature to install Python libs as packages. Ref: https://wiki.archlinux.org/title/Python#Arch_Repositories

A large number of popular packages are available in the Official repositories and AUR. This is the preferred way to install system-wide packages, and the only method officially supported on Arch Linux.

Really it's absolutely necessary that bbot is installed inside its own venv. After that there shouldn't be any issues/conflicts with OS-level pip packages, and it shouldn't have any problem installing module-specific dependencies like baddns.

I agree but we should understand how to manage virtual env in a PKGBUILD for BBOT, by starting from my first PKGBUILD proposal above. Even if I would add some control to manage also the case when baddns is installed as a package, for example a check that skip the pip install of baddns if baddns is already installed (i.e., if /usr/bin/baddns exists).

[TheTechromancer]

Yeah I wish I had more time to help with this but I will say I've had good luck with ChatGPT/Claude when it comes to making PKGBUILDs; prompted with something like make an arch PKGBUILD that clones a python project from github and installs it inside its own venv, you might get a good starting point.

[D3vil0p3r]

BlackArch/blackarch#4382