Updated packages #2972

ammar92 · 2024-05-21T07:42:45Z

Changes

This PR updates and removes some packages.

Issue link

Closes #2991
Closes #2992

Code Checklist

All the commits in this PR are properly PGP-signed and verified.
This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
I have written unit tests for the changes or fixes I made.
For any non-trivial functionality, I have added integration and/or end-to-end tests.
I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
I have made corresponding changes to the documentation, if necessary.
I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

stephanie0x00 · 2024-05-21T14:30:52Z

When scanning mispo.es from the onboarding I get the following errors.

Cant download the raw file for the failed dnssec boefje:

boefje-1  | [2024-05-21 14:27:17 +0000] [17] [INFO] [job_handler] Delegating boefje dns-sec[task_id=680bb87a-517c-41f2-ae74-3c7ef7b84925] to Docker runner with OCI image [openkat/dns-sec]
boefje-1  | [2024-05-21 14:27:17 +0000] [17] [ERROR] [app] An error occurred handling scheduler item[id=680bb87a-517c-41f2-ae74-3c7ef7b84925]
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
boefje-1  |     conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 400, in _get_connection
boefje-1  |     conn = self.poolmanager.connection_from_host(
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host
boefje-1  |     return self.connection_from_context(request_context)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context
boefje-1  |     raise URLSchemeUnknown(scheme)
boefje-1  | urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker
boefje-1  | 
boefje-1  | During handling of the above exception, another exception occurred:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 213, in _retrieve_server_version
boefje-1  |     return self.version(api_version=False)["ApiVersion"]
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
boefje-1  |     return self._result(self._get(url), json=True)
boefje-1  |                         ^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 44, in inner
boefje-1  |     return f(self, *args, **kwargs)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 236, in _get
boefje-1  |     return self.get(url, **self._set_request_timeout(kwargs))
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
boefje-1  |     return self.request("GET", url, **kwargs)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
boefje-1  |     resp = self.send(prep, **send_kwargs)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
boefje-1  |     r = adapter.send(request, **kwargs)
boefje-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 534, in send
boefje-1  |     raise InvalidURL(e, request=request)
boefje-1  | requests.exceptions.InvalidURL: Not supported URL scheme http+docker
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/app.py", line 247, in _start_working
boefje-1  |     handler.handle(p_item.data)
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 115, in handle
boefje-1  |     docker_runner = DockerBoefjesRunner(boefje_resource, boefje_meta)
boefje-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/docker_boefjes_runner.py", line 21, in __init__
boefje-1  |     self.docker_client = docker.from_env()
boefje-1  |                          ^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 94, in from_env
boefje-1  |     return cls(
boefje-1  |            ^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
boefje-1  |     self.api = APIClient(*args, **kwargs)
boefje-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
boefje-1  |     self._version = self._retrieve_server_version()
boefje-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 220, in _retrieve_server_version
boefje-1  |     raise DockerException(
boefje-1  | docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker
boefje-1  | [2024-05-21 14:27:17 +0000] [17] [INFO] [_client] HTTP Request: PATCH http://scheduler:8000/tasks/680bb87a-517c-41f2-ae74-3c7ef7b84925 "HTTP/1.1 200 OK"

stephanie0x00 · 2024-05-21T14:44:01Z

The Subfinder boefje tasks fail. This is the output from the raw file:

A similar error is shown for Webpage capture, nmap UDP and nmap TCP. Possibly also other boefjes, I've only enabled those who do not require additional settings such as API keys.

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
    conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 400, in _get_connection
    conn = self.poolmanager.connection_from_host(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 304, in connection_from_host
    return self.connection_from_context(request_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 326, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 213, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
                        ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 236, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 534, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/boefjes/boefjes/local.py", line 54, in run
    return boefje_resource.module.run(boefje_meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/boefjes/boefjes/plugins/pdio_subfinder/main.py", line 9, in run
    client = docker.from_env()
             ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 94, in from_env
    return cls(
           ^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/docker/api/client.py", line 220, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/app/boefjes/boefjes/job_handler.py", line 143, in handle
    boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/boefjes/boefjes/local.py", line 56, in run
    raise JobRuntimeError("Boefje failed") from e
boefjes.runtime_interfaces.JobRuntimeError: Boefje failed

ammar92 · 2024-05-22T07:01:06Z

The current requests package breaks the support for http+docker scheme used by the docker package. There are currently PRs open to fix this, so I'm waiting for that.

ammar92 · 2024-05-22T19:14:51Z

The current requests package breaks the support for http+docker scheme used by the docker package. There are currently PRs open to fix this, so I'm waiting for that.

docker/docker-py#3257 seems to be merged, now waiting for a release

* main: (78 commits) Translations update from Hosted Weblate (#3048) Translations update from Hosted Weblate (#3018) Fix empty consumes of boefjes will trigger tasks in scheduler (#3017) Fixes text in secondary menu on scan profile detail page (#3035) chore: Resolves css-issues found by sonarcloud (#3034) Add raw AuthToken SQL migration (#3009) Translations update from Hosted Weblate (#3012) Rewrite xtdb-cli.py with "click" (#2957) Phase out the Repository model from the KATalogus (#2984) Fix merge conflicts in weblate (#3007) Translations update from Hosted Weblate (#2996) Reports: Fix select all OOIs (#2909) Adding IPv6 support to documentation for Docker setups (#2813) Translations update from Hosted Weblate (#2930) User documentation for reports (#2898) Fix task api status code response for malformed id in the scheduler (#2953) Add drill trace option in dnssec boefje (#2979) Updated packages (#2972) Update granian and remove workaround for fixed bug (#2980) Fix typing in boefjes/normalizers (#2933) ...

Updated packages

e2bac92

ammar92 added the dependencies Pull requests that update a dependency file label May 21, 2024

Updated requests

6990d32

ammar92 marked this pull request as ready for review May 21, 2024 07:44

ammar92 requested a review from a team as a code owner May 21, 2024 07:44

Replaced fastapi with fastapi-slim

67fe3c5

Donnype previously approved these changes May 21, 2024

View reviewed changes

stephanie0x00 added the 😸 Review/QA feedback Review/QA feedback provided label May 21, 2024

Updated requests, previous version is yanked

b32c56e

ammar92 dismissed Donnype’s stale review via b32c56e May 22, 2024 06:54

ammar92 added 6 commits May 23, 2024 14:10

Updated docker-py

780de74

Reverted Granian

ed30af1

Revert

3af9f09

Fix

a070f26

Merge branch 'main' into fix/update-packages

1f78487

Updated docker-py

245a36b

underdarknl approved these changes May 24, 2024

View reviewed changes

underdarknl merged commit 30cfcbf into main May 24, 2024
41 checks passed

underdarknl deleted the fix/update-packages branch May 24, 2024 12:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updated packages #2972

Updated packages #2972

ammar92 commented May 21, 2024 •

edited

Loading

stephanie0x00 commented May 21, 2024

stephanie0x00 commented May 21, 2024 •

edited

Loading

ammar92 commented May 22, 2024

ammar92 commented May 22, 2024 •

edited

Loading

Updated packages #2972

Updated packages #2972

Conversation

ammar92 commented May 21, 2024 • edited Loading

Changes

Issue link

Code Checklist

Communication

Checklist for code reviewers:

Checklist for QA:

stephanie0x00 commented May 21, 2024

stephanie0x00 commented May 21, 2024 • edited Loading

ammar92 commented May 22, 2024

ammar92 commented May 22, 2024 • edited Loading

ammar92 commented May 21, 2024 •

edited

Loading

stephanie0x00 commented May 21, 2024 •

edited

Loading

ammar92 commented May 22, 2024 •

edited

Loading