Release/2.1.0 #1

AnonymousWP · 2024-04-07T19:52:10Z

Introduced OllamaManager API class with get, delete, and put methods for managing Ollama models.
Enhanced WafDetector and CMSDetector APIs with improved error handling and logging.
Added organization handling in AddTarget and bulk target addition.
Updated Docker and Makefile configurations for better build and deployment processes.
Implemented new logging configurations and improved settings management.
Added support for Lark notifications and improved API key management in the UI.
Refactored various paths and configurations to use environment variables and Path objects for better flexibility.
Enhanced scan initiation and scheduling with user context and improved error handling.
Added new utility functions and refactored existing ones for better code organization and readability.

update codeql to v3

replace .format() with f-strings

Optimize common funcs

ops(make): add option to change password via `make`

small python cleanup

ops(docker-compose): upgrade to 2.23.3

fix(ui): stored XSS

…tall fix(install): revert changes of prebuilt chain

…-loading fix(ui): fix 500 error on scan engine add

…all-to-wiki docs(readme): redirect install & update section to the wiki pages

* build(ci): build docker images for each tag, release, push * build(ci): set bot name and login vars * docker(build): add arm64 to the build options * build(ci): add manual push workflow and use repo vars * build(ci): update vscode tasks to push latest tag * fix from feedback

build(install): improve root detection and set ownership on files

…n tagged images (#193) * build(ci): restrict build to file in docker folder & improve tags * build(ci): add a remove image without tag workflow * build(ci): set automatic execution for image removal * style(language): remove french comments

* fix(scan): fix clocked and scheduled scan not working * fix(scan): store start datetime of schedule scan in UTC * fix(celery): add more DEBUG error for celery beat (datetime of tasks ...) * style(time): add UTC to the schedule time

* fix(graph): deduplicate dorks and vulnerabilities * fix(typo): remove french language * fix(typo): remove french comments * fix(graph): remove carriage return

…de-ql-scanning build(ci): improve CodeQL configuration

web/scanEngine/templates/scanEngine/settings/api.html

+<script>
+  $(document).ready(function() {
+    const keyInfos = {
+      {% for apiKey in apiKeys %}


web/static/custom/custom.js

@@ -1874,11 +1890,12 @@
 	var domain_name = $('#target_name_modal').val();
 	var description = $('#target_description_modal').val();
 	var h1_handle = $('#h1_handle_modal').val();
-	add_target(domain_name, h1_handle = h1_handle, description = description);
+	var organization = $('#target_organization_modal').val();
+	add_target(domain_name, h1_handle = h1_handle, description = description, organization = organization);


web/static/custom/custom.js

@@ -1874,11 +1890,12 @@
 	var domain_name = $('#target_name_modal').val();
 	var description = $('#target_description_modal').val();
 	var h1_handle = $('#h1_handle_modal').val();
-	add_target(domain_name, h1_handle = h1_handle, description = description);
+	var organization = $('#target_organization_modal').val();
+	add_target(domain_name, h1_handle = h1_handle, description = description, organization = organization);


web/static/custom/custom.js

@@ -1874,11 +1890,12 @@
 	var domain_name = $('#target_name_modal').val();
 	var description = $('#target_description_modal').val();
 	var h1_handle = $('#h1_handle_modal').val();
-	add_target(domain_name, h1_handle = h1_handle, description = description);
+	var organization = $('#target_organization_modal').val();
+	add_target(domain_name, h1_handle = h1_handle, description = description, organization = organization);


web/static/custom/custom.js

@@ -1874,11 +1890,12 @@
 	var domain_name = $('#target_name_modal').val();
 	var description = $('#target_description_modal').val();
 	var h1_handle = $('#h1_handle_modal').val();
-	add_target(domain_name, h1_handle = h1_handle, description = description);
+	var organization = $('#target_organization_modal').val();
+	add_target(domain_name, h1_handle = h1_handle, description = description, organization = organization);


web/static/custom/custom.js

@@ -1874,11 +1890,12 @@
 	var domain_name = $('#target_name_modal').val();
 	var description = $('#target_description_modal').val();
 	var h1_handle = $('#h1_handle_modal').val();
-	add_target(domain_name, h1_handle = h1_handle, description = description);
+	var organization = $('#target_organization_modal').val();
+	add_target(domain_name, h1_handle = h1_handle, description = description, organization = organization);


web/static/custom/custom.js

@@ -1874,11 +1890,12 @@
 	var domain_name = $('#target_name_modal').val();
 	var description = $('#target_description_modal').val();
 	var h1_handle = $('#h1_handle_modal').val();
-	add_target(domain_name, h1_handle = h1_handle, description = description);
+	var organization = $('#target_organization_modal').val();
+	add_target(domain_name, h1_handle = h1_handle, description = description, organization = organization);


web/static/custom/mitch.js

-    var subdomain_count = data[0]['children'][0]['children'].length;
+    // Find the 'Subdomains' node in the children
+    var subdomainsNode = treeData.children.find(child => child.description === 'Subdomains');
+    var subdomain_count = subdomainsNode ? subdomainsNode.children.length : 0;


web/api/views.py

+            else:
+                return Response({'status': False, 'message': 'Could not detect CMS!'})
+        except Exception as e:
+            return Response({'status': False, 'message': str(e)})


To fix the problem, we need to ensure that detailed exception information is not exposed to the end user. Instead, we should log the detailed exception on the server and return a generic error message to the user. This approach maintains the ability to debug issues using server logs while protecting sensitive information from being exposed.

Modify the exception handling block to log the exception using the logger and return a generic error message to the user.

Ensure that the logging captures the full stack trace for debugging purposes.

web/api/views.py

+            else:
+                return Response({'error': result['message']}, status=500)
+        except Exception as e:
+            return Response({'error': str(e)}, status=500)


To fix the problem, we need to ensure that detailed error information is not exposed to the end user. Instead, we should log the detailed error message on the server and return a generic error message to the user. This can be achieved by modifying the exception handling block to log the exception and return a generic error message.

Modify the exception handling block in the GfList class to log the exception using the logger and return a generic error message.

Ensure that the logger is properly configured to capture and store the detailed error information.

web/reNgine/common_func.py

+	cms_json_path =  cms_dir_path + '/cms.json'
+
+	if os.path.isfile(cms_json_path):
+		cms_file_content = json.loads(open(cms_json_path, 'r').read())


web/reNgine/definitions.py

@@ -1,6 +1,9 @@
 #!/usr/bin/python
 import logging
 import re
+import os.path


web/reNgine/settings.py

+RENGINE_RAISE_ON_ERROR = env.bool('RENGINE_RAISE_ON_ERROR', default=False)
+
+with open(Path(RENGINE_HOME) / 'reNgine' / 'version.txt', 'r', encoding="utf-8") as f:
+    RENGINE_CURRENT_VERSION = f.read().strip()


web/scanEngine/views.py

+            try:
+                handler[key].objects.first().delete()
+                response["deleted"].append(key)
+            except KeyError:


web/scanEngine/views.py

web/targetApp/views.py

@@ -168,10 +180,10 @@
                    io_string = io.StringIO(txt_content)
                    for target in io_string:
                        target_domain = target.rstrip("\n").rstrip("\r")
-                        domain = None  # Move the domain variable declaration here
+                        domain = None


web/targetApp/views.py

…xtensions dev(django): install django extensions to have more commands

…volume-empty build(docker): replace staticfiles volume to prevent empty directory

- Disabled the "Check Update" button in the Tool Arsenal section and replaced it with a placeholder indicating that the update feature is coming soon.

chore(ui): disable update button in tool arsenal

* feat(tests): add unit tests for api views * feat(unit-tests): add 12 unit tests for API app * feat(unit-tests): minor changes on test scan & nmap * feat(unit-tests): add 32 unit tests for api app * feat(unit-tests): add 24 unit tests for api app * feat(unit-test): correct some bugs and refactor * feat(unit-tests): replace hardcoded values * feat(unit-tests): change viewset aliases basename * feat(unit-test): correctly cast all int parameter value * feat(unit-tests): bug fixes and add of 6 unit tests * feat(unit-tests): add docstring * feat(unit-tests): refactor API unit tests * refactor(api): improve import structure and add tests for activity logs - Refactored import statements in views.py for better readability and maintainability. - Added handling for single integer subdomain_ids in InitiateSubTask. - Enhanced querysets in ListActivityLogsViewSet and ListScanLogsViewSet to include ordering by id. - Introduced new tests for ListActivityLogsViewSet in test_scan.py. - Updated TestDataGenerator in test_utils.py to include methods for creating scan_activity and command. - Added a docstring to safe_int_cast in common_func.py. * fix(tests): update subscan creation and assertion logic Modified the subscan creation process in test_scan.py to handle multiple subscans and updated related assertions. * feat: add timezone-aware datetime handling and new dashboard view test - Updated date handling in the index view to use timezone-aware datetimes. - Refactored test imports to use a common BaseTestCase from utils.test_base. - Added a new test suite for dashboard views, including tests for various dashboard functionalities. - Enhanced TestDataGenerator to include discovered_date for endpoints. * test: add __all__ declarations to test files for better module export control Added __all__ declarations to various test files to explicitly define the public interface of these modules. This change improves module export control and clarifies which classes and functions are intended for external use. * test: add missing __all declarations - Introduced __all__ declarations in test_scan.py and test_base.py for better module export control. - Removed an unused import in test_dashboard.py. * feat: add new test cases and utility functions for startScan views and models - Added new test cases for startScan views and models. Introduced MockTemplate utility for mocking Django templates in tests. - Updated TestDataGenerator to include create_scan_activity and create_command methods. - Replaced direct type casting with safe_int_cast in startScan views. - Removed redundant client login steps in TestDashboardViews. - Deleted obsolete test_scan.py file. * feat(scanEngine): enhance test utilities and add unit tests for scanEngine views - Imports and Models: Updated imports in test_utils.py to include additional models from scanEngine and other related modules. - Test Data Generation: Refactored and expanded the TestDataGenerator class to include methods for creating various test objects related to scanEngine. - Views Cleanup: Removed debug print statements from scanEngine/views.py and refactored string concatenations to use f-strings. - Unit Tests: Added a new test file test_scan_engine.py containing comprehensive unit tests for the scanEngine views, covering functionalities like adding, updating, and deleting engines, wordlists, and tools. * refactor: update test data generator and scan engine tests - Added methods create_interesting_lookup_model and create_search_history to TestDataGenerator. - Removed the method create_lookup from TestDataGenerator. - Updated test_scan_engine.py to use interesting_lookup_model instead of lookup. * refactor: remove redundant code in views and test_base modules - Removed redundant installation command in add_tool function within views.py. - Removed unused import statement in test_base.py. * feat: enhance target and organization management with validation and tests - Imports and Dependencies: Reorganized and added necessary imports for better modularity and functionality. - File Upload Validation: Added checks for empty file uploads and invalid IP addresses during target addition. - Error Handling: Improved error handling in delete_target and delete_organization views to handle non-existent entities. - Form Enhancements: Updated AddOrganizationForm to use ModelForm and improved domain selection logic. - Utility Functions: Moved get_ip_info and get_ips_from_cidr_range functions to common_func.py for better reusability. - Validators: Added a new validate_ip function in validators.py to validate IP addresses. - Unit Tests: Added comprehensive unit tests for target and organization views to ensure proper functionality and error handling. * feat: add logging for error conditions & fix CodeQL warnings - Added logging for various error conditions in add_target and delete_target views. - Added __all__ declaration in test_target_app.py for better module export control. - Minor cleanup in common_func.py without functional changes. * refactor(logging): use lazy formatting for logger messages - Updated logger messages in web/targetApp/views.py to use lazy formatting. * feat(tests): add validation and error handling for note operations - Added validation checks and error handling for missing or invalid IDs in note operations. - Improved error messages for better clarity in the list_note, flip_todo_status, flip_important_status, and delete_note functions. - Added required field checks in the AddReconNote API view. * fix(tests): adjust test assertions * refactor: improve error handling and code readability in API views Overview - Enhanced error handling and logging across multiple API views. - Refactored code for better readability and maintainability. - Added new test cases to cover edge scenarios and improve test coverage. - Updated Docker configuration for development environment. Details - OllamaManager: Simplified error handling and added comments for clarity. - GPTAttackSuggestion: Improved error handling and refactored code for better readability. - FetchMostCommonVulnerability: Refactored query logic and improved error handling. - AddTarget: Added validation for domain names and improved error messages. - DeleteVulnerability: Added validation for input data and improved error handling. - ListTechnology: Refactored query logic for better readability. - get_ips_from_cidr_range: Improved error handling and logging. - Test cases: Added new test cases for various scenarios, including failure cases and edge cases. - Docker: Updated docker-compose.dev.yml to enable remote debugging and added a new port. - Miscellaneous: Various minor improvements and bug fixes across different files.

* feat(todo): enhance todo functionality and error handling - Refactored the JavaScript code for creating and displaying todo items to use a template-based approach. - Improved error handling and user feedback for adding, deleting, and updating todo items. - Updated API endpoints and views to include better validation and error messages. - Enhanced the UI with new icons and required fields for better user experience. - Added a hidden template for todo items to streamline the creation of new items dynamically. * feat(todo): enhance todo functionality with async operations and input validation - Refactored JavaScript code to use const instead of var for variable declarations. - Updated event handlers to use async functions for better handling of asynchronous operations. - Improved input validation in the HTML form for adding todos, including setting minimum and maximum lengths and allowed character patterns. - Enhanced error handling and user feedback for adding, deleting, and updating todo items. - Added detailed docstrings to Python view functions for better code documentation and readability. - Introduced logging for JSON decode errors in the Python views. * fix(todo): streamline button click handling and improve UI feedback - Simplified the button click handling logic in todo.js by removing redundant code and improving readability. - Enhanced the user interface feedback for important task toggling and deletion confirmation. - Added autocomplete="on" to the task input field in index.html for better user experience. - Removed redundant class addition in the todo item creation process. * refactor(todo): modularize event listeners and improve search functionality - Refactored todo.js to modularize event listeners into separate functions. - Enhanced search functionality to save and load search terms from local storage. - Updated the UI to include a clear search button and improved button styling for task actions. - Adjusted HTML and CSS to support the new search and action button features. * refactor(todo.js): remove redundant variable assignments for PerfectScrollbar instances - Removed redundant variable assignments for PerfectScrollbar instances in the populateTodofunction. - Added a missing semicolon in the importantBtnListener function. * fix(todo): correct function names and enhance task filtering - Corrected the function name from populateTodofunction to populateTodo. - Added a new listener function addTaskPopupListener to handle task popups. - Renamed addActionsBtnListener to actionsBtnListener and added a call to searchFunction within it. - Introduced a new property is_important for tasks and updated the template to conditionally display the important badge. - Enhanced the search functionality to apply the current filter after searching. - Added a new function applyCurrentFilter to filter tasks based on their status (Todo, Done, Important). * fix(note): reset var name for note saving * fix: update data keys in todo.js - Updated key names in todo.js to ensure consistency with backend expectations. * fix: update CSS styles - Adjusted CSS styles in todolist.css for better alignment and spacing. * chore: standardize "to-do" terminology across templates and scripts - Updated various HTML templates and JavaScript files to standardize the terminology from "Todo" to "to-do". * fix: update content type in tests and error messages and correct HTML placeholder - Updated test cases to include content_type='application/json' in POST requests. - Modified error messages in AddReconNote view to remove checks for scan_history_id and subdomain_id. - Corrected the placeholder text in the HTML input field for adding a to-do. * refactor(todo): simplify badge update logic and remove unused checkbox elements - Simplified the logic for updating badge notifications in the todo list by consolidating repeated code into a single function. - Removed unused checkbox elements from the todo item template and related HTML. * feat: add close button to modal - Added a "Close" button to the modal footer in the note index template. - Implemented functionality to hide the modal when the "Close" button is clicked.

* bug(fetch_url): check if there's a list of urls * bug(custom_header): rework the run command and correctly split parameters * feat: add input sanitization and utf-8 encoding support - Introduced a clean_quotes function to sanitize input data by removing double quotes. - Updated form handling in add_engine and update_engine views to use the clean_quotes function for input sanitization. - Added UTF-8 encoding support to various file operations to ensure proper handling of text files. - Enhanced parse_custom_header function to validate header values and raise errors for invalid formats.

web/reNgine/common_func.py

+        subprocess.Popen: The Popen object for the executed command.
+    """
+    return subprocess.Popen(
+        command,


To fix the problem, we should avoid passing user input directly to the subprocess.Popen method. Instead, we can use a predefined allowlist of commands or sanitize the input to ensure it is safe. Here are the steps to fix the issue:

Sanitize the Input: Use shlex.split to safely parse the command string into a list of arguments.

Avoid Shell Execution: Set the shell parameter to False to avoid the risks associated with shell execution.

fix: change install_type value in .env-dist and add missing imports

AnonymousWP and others added 30 commits February 21, 2024 20:28

chore: update version number to 2.0.3

d692102

update codeql to v3

32955f1

Fix subdomains list empty in Target

50e139f

Fix top menu text overflow in low resolution

97a2909

Update auto comment workflow due to deprecation warnings

8633a1e

Change Redirect URL after login to prevent 500 error

54b9d53

fix-1030: Add missing slug on target summary link

8576b60

chore: update version number to 2.0.3

f757da3

replace .format() with f-strings

7ba20b7

Fix subdomains list empty in Target

7b0f9e2

Fix top menu text overflow in low resolution

cd1b214

Update auto comment workflow due to deprecation warnings

e803810

Change Redirect URL after login to prevent 500 error

e2dca49

fix-1030: Add missing slug on target summary link

43deee1

chore: update version number to 2.0.3

7cff813

remove unused uuid library

19240f9

remove non-existet params from docstrings

e6bedbf

optimize get_nmap_cmd based on TODO

5378d83

simplify xml2json func

c29a903

use f-strings

1c5de66

Merge pull request #1198 from jxdv/update-codeql

7549635

update codeql to v3

Merge pull request #1195 from jxdv/use-fstrings

8aa7d36

replace .format() with f-strings

Merge pull request #1194 from jxdv/optimize-common-funcs

5491cb3

Optimize common funcs

Merge branch 'release/2.1.0' into 955-i-forgot-the-password

d0b2bcb

Update make.bat

93af479

Merge pull request #1171 from yogeshojha/955-i-forgot-the-password

c657e07

ops(make): add option to change password via `make`

Merge pull request #1133 from jxdv/py-cleanup

ae21433

small python cleanup

Merge branch 'release/2.1.0' into update-docker-compose

3969353

Merge pull request #1119 from yogeshojha/update-docker-compose

dd362ff

ops(docker-compose): upgrade to 2.23.3

Remove recursive loop in FFUF & add comments

29a7488

AnonymousWP and others added 16 commits August 29, 2024 20:51

Merge pull request #180 from Security-Tools-Alliance/179-fix-xss-stored

4290504

fix(ui): stored XSS

fix(install): revert changes of prebuilt chain

75f042f

Merge pull request #183 from Security-Tools-Alliance/fix-prebuilt-ins…

8c07b69

…tall fix(install): revert changes of prebuilt chain

fix(ui): fix 500 error on scan engine add

f9aaef2

Merge pull request #184 from Security-Tools-Alliance/fix-default-yaml…

fe4cf72

…-loading fix(ui): fix 500 error on scan engine add

fix(readme): redirect install & update section to the wiki pages

f8ecea7

Merge pull request #185 from Security-Tools-Alliance/link-readme-inst…

48ec4ac

…all-to-wiki docs(readme): redirect install & update section to the wiki pages

fix(install): improve root detection and set ownership on files

7f5b8bc

style(install): Apply suggestions

3a48a44

Merge pull request #186 from Security-Tools-Alliance/fix-install-bugs

8843dd4

build(install): improve root detection and set ownership on files

fix(graph): de-duplicate dorks and vulnerabilities (#188)

11c43bb

* fix(graph): deduplicate dorks and vulnerabilities * fix(typo): remove french language * fix(typo): remove french comments * fix(graph): remove carriage return

build(ci): improve CodeQL configuration

17b0f34

Merge pull request #194 from Security-Tools-Alliance/build(ci)/add-co…

a9b012a

…de-ql-scanning build(ci): improve CodeQL configuration

github-advanced-security bot found potential problems Sep 4, 2024

View reviewed changes

psyray and others added 9 commits September 6, 2024 17:35

dev(django): install django extensions to have more commands

9862f76

Merge pull request #196 from Security-Tools-Alliance/install-django-e…

9e45e8e

…xtensions dev(django): install django extensions to have more commands

fix(docker): replace staticfiles volume to prevent empty directory

8183c04

Merge pull request #199 from Security-Tools-Alliance/fix-staticfiles-…

708b988

…volume-empty build(docker): replace staticfiles volume to prevent empty directory

chore(ui): disable update button in tool arsenal

8fc8211

- Disabled the "Check Update" button in the Tool Arsenal section and replaced it with a placeholder indicating that the update feature is coming soon.

Merge pull request #200 from Security-Tools-Alliance/disable-tool-update

1a716a8

chore(ui): disable update button in tool arsenal

github-advanced-security bot found potential problems Sep 18, 2024

View reviewed changes

psyray and others added 2 commits September 18, 2024 22:44

fix: change install_type value in .env-dist and add missing imports

c3b031b

Merge pull request #201 from Security-Tools-Alliance/fix-little-bugs

2b79276

fix: change install_type value in .env-dist and add missing imports

@@ -1261,22 +1261,24 @@
-            def execute_command(command, shell, cwd):
-                """
-                Execute a command using subprocess.
-                Args:
-                    command (str or list): The command to execute.
-                    shell (bool): Whether to use shell execution.
-                    cwd (str): The working directory for the command.
-                Returns:
-                    subprocess.Popen: The Popen object for the executed command.
-                """
-                return subprocess.Popen(
-                    command,
-                    stdout=subprocess.PIPE,
-                    stderr=subprocess.STDOUT,
-                    universal_newlines=True,
-                    shell=shell,
-                    cwd=cwd
-                )
+            def execute_command(command, shell, cwd):
+                """
+                Execute a command using subprocess.
+                Args:
+                    command (str or list): The command to execute.
+                    shell (bool): Whether to use shell execution.
+                    cwd (str): The working directory for the command.
+                Returns:
+                    subprocess.Popen: The Popen object for the executed command.
+                """
+                if isinstance(command, str):
+                    command = shlex.split(command)
+                return subprocess.Popen(
+                    command,
+                    stdout=subprocess.PIPE,
+                    stderr=subprocess.STDOUT,
+                    universal_newlines=True,
+                    shell=False,
+                    cwd=cwd
+                )

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Release/2.1.0 #1

Release/2.1.0 #1

AnonymousWP commented Apr 7, 2024 •

edited by psyray

Loading

Release/2.1.0 #1

Are you sure you want to change the base?

Release/2.1.0 #1

Conversation

AnonymousWP commented Apr 7, 2024 • edited by psyray Loading

AnonymousWP commented Apr 7, 2024 •

edited by psyray

Loading