Phase out the Repository model from the KATalogus

[Donnype]

Changes

This PR phases out the unused Repository model, in favor (later PRs) of a generic oci_image url for the boefje images, see #2983 and #2881. Most changes are either the removal or relocation of code and files, so do not let the number of Files changed distract you.

Issue link

Closes #2983

Demo

This is a transparent change and adds no functionality. We could almost call it a refactor.

But, what is important is that migrating this change happens without issues. There are potential caveats here however when users have actually added Repository models, although changes are low that these are actually being used for the plugins since the API code that these models can connected has actually already been deleted a long time ago in this PR. Therefore, probably 80% of users will have no issues, 19% will have to look into their katalogus database to find unused repository entries they can just delete because they did not know about or forgot them, and 1% that's either running the old API or built their own should be told to contact us to help them migrate whatever they are running behind that API to the new OCI framework we introduced.

Hence, try to create some data, enable plugins, add organizations and then checkout this PR to see if there are any migration issues!

---

Code Checklist

• All the commits in this PR are properly PGP-signed and verified.
• This PR only contains functionality relevant to the issue; tickets have been created for newly discovered issues.
• I have written unit tests for the changes or fixes I made.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have performed a self-review of my code and refactored it to the best of my abilities.

Communication

• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have made corresponding changes to the documentation, if necessary.
• I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[ammar92]

Looks good to me overall. I just had one suggestion. I didn't dive too deep into this since, as you mentioned, these changes are mostly removal or relocation of code. The modified and new tests are comprehensive and should give us confidence to merge this, well done 👍

[stephanie0x00]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make reset.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.
• I checked the logs for errors and/or warnings and made issues where necessary

What works:

Seems to look good. Haven't found anything obvious:

• Onboarding works
• Can generate normal and aggregate reports
• Tasks complete, apart from those caused by bug RAW files cannot be downloaded from Tasks page #2864
• Created 2 organisations and performed scans with both organisations
• Enabled all boefjes in one of the organisations that do not require additional settings such as API keys and/or config settings.
• Tree and graphs can be viewed.
• Boefjes in de katalogus have the Consumes and Produces links, and tasks that were performed.

What doesn't work:

• Nothing found

Bug or feature?:

• I don't think the bugs below are related, but just in case.

boefje-1  | [2024-05-30 11:15:50 +0000] [19] [INFO] [job_handler] Starting boefje webpage-capture[2e63764d-0280-466d-a894-c810bf8555a
4]
boefje-1  | [2024-05-30 11:15:54 +0000] [19] [WARNING] [helpers] /tmp/output.png not found in container c1026d5572d2 ['mcr.microsoft.com/playwright:latest']
boefje-1  | [2024-05-30 11:15:54 +0000] [19] [WARNING] [helpers] /tmp/output.har.zip not found in container c1026d5572d2 ['mcr.microsoft.com/playwright:latest']
boefje-1  | [2024-05-30 11:15:54 +0000] [19] [WARNING] [helpers] /tmp/output.json not found in container c1026d5572d2 ['mcr.microsoft.com/playwright:latest']
boefje-1  | [2024-05-30 11:15:54 +0000] [19] [ERROR] [job_handler] Error running boefje webpage-capture[2e63764d-0280-466d-a894-c810bf8555a4]
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/local.py", line 57, in run
boefje-1  |     return boefje_resource.module.run(boefje_meta)
boefje-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/plugins/kat_webpage_capture/main.py", line 84, in run
boefje-1  |     image_png, har_zip, storage_json = run_playwright(webpage=webpage, browser=BROWSER)
boefje-1  |                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/plugins/kat_webpage_capture/main.py", line 64, in run_playwright
boefje-1  |     raise WebpageCaptureException(
boefje-1  | boefjes.plugins.kat_webpage_capture.main.WebpageCaptureException: Playwright container did not return expected files, command was: /usr/bin/npx playwright screenshot -b chromium --full-page --ignore-https-errors --save-har=/tmp/output.har.zip --save-storage=/tmp/output.json http://<REDACTED>.nl/.well-known/security.txt /tmp/output.png
boefje-1  | 
boefje-1  | Container log:
boefje-1  | 2024-05-30T11:15:51.627658482Z npm WARN exec The following package was not found and will be installed: playwright@1.44.1
boefje-1  | 2024-05-30T11:15:53.824119003Z Navigating to http://<REDACTED>.nl/.well-known/security.txt
boefje-1  | 2024-05-30T11:15:53.970968276Z Error: net::ERR_CONNECTION_CLOSED at http://<REDACTED>.nl/.well-known/security.txt
boefje-1  | 2024-05-30T11:15:53.971000757Z Call log:
boefje-1  | 2024-05-30T11:15:53.971003923Z   - navigating to "http://<REDACTED>.nl/.well-known/security.txt", waiting until "load"
boefje-1  | 2024-05-30T11:15:53.971006839Z 
boefje-1  | 2024-05-30T11:15:54.096316505Z npm notice 
boefje-1  | 2024-05-30T11:15:54.096452542Z npm notice New minor version of npm available! 10.5.2 -> 10.8.1
boefje-1  | 2024-05-30T11:15:54.096631992Z npm notice Changelog: <https://github.com/npm/cli/releases/tag/v10.8.1>
boefje-1  | 2024-05-30T11:15:54.096852981Z npm notice Run `npm install -g npm@10.8.1` to update!
boefje-1  | 2024-05-30T11:15:54.096972917Z npm notice 
boefje-1  | 
boefje-1  | 
boefje-1  | The above exception was the direct cause of the following exception:
boefje-1  | 
boefje-1  | Traceback (most recent call last):
boefje-1  |   File "/app/boefjes/boefjes/job_handler.py", line 143, in handle
boefje-1  |     boefje_results = self.job_runner.run(boefje_meta, boefje_meta.environment)
boefje-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
boefje-1  |   File "/app/boefjes/boefjes/local.py", line 59, in run
boefje-1  |     raise JobRuntimeError("Boefje failed") from e
boefje-1  | boefjes.runtime_interfaces.JobRuntimeError: Boefje failed
boefje-1  | [2024-05-30 11:15:54 +0000] [19] [INFO] [job_handler] Saving to Bytes for boefje webpage-capture[2e63764d-0280-466d-a894-c810bf8555a4]
boefje-1  | [2024-05-30 11:15:54 +0000] [19] [INFO] [_client] HTTP Request: POST http://bytes:8000/bytes/boefje_meta "HTTP/1.1 201 Created"

scheduler-1  | 2024-05-30 10:48:19 [info     ] Boefje scheduler started for aa item_type=BoefjeTask organisation_id=aa scheduler_id=boefje-aa
scheduler-1  | 2024-05-30 10:48:20 [info     ] Created normalizer task: 581f7c0e-7316-4d57-8cbe-bccb1f006fab for raw data: f835d880-0285-4a84-8946-c3483f0b9c2d caller=push_tasks_for_received_raw_data normalizer_id=kat_manual_ooi organisation_id=aa raw_data_id=UUID('f835d880-0285-4a84-8946-c3483f0b9c2d') scheduler_id=normalizer-aa task_id=UUID('581f7c0e-7316-4d57-8cbe-bccb1f006fab')
scheduler-1  | /usr/local/lib/python3.11/site-packages/pydantic/main.py:347: UserWarning: Pydantic serializer warnings:
scheduler-1  |   Expected `enum` but got `str` - serialized value may not be as expected
scheduler-1  |   return self.__pydantic_serializer__.to_python(
scheduler-1  | /usr/local/lib/python3.11/site-packages/pydantic/type_adapter.py:339: UserWarning: Pydantic serializer warnings:
scheduler-1  |   Expected `enum` but got `str` - serialized value may not be as expected
scheduler-1  |   return self.serializer.to_python(
scheduler-1  | 2024-05-30 10:49:18 [info     ] Created boefje task: 582233f3-a394-4e0d-9844-65c731183e4f for ooi: Hostname|internet|mispo.es boefje_id=dns-records caller=push_tasks_for_scan_profile_mutations ooi_primary_key=Hostname|internet|mispo.es organisation_id=aa scheduler_id=boefje-aa task_id=UUID('582233f3-a394-4e0d-9844-65c731183e4f')

• I observed a few failed tasks, but those are already present on main: RAW files cannot be downloaded from Tasks page #2864