fix(custom_header): not correctly parsing parameters

[psyray]

Fixes #19
When you set custom header in the scan, value does not been parsed correctly, making httpx crash and not resolving http request.
As httpx is the base of the scan, no other scan will be launched because no URL was found.

I've reworked the run_command and stream_command to be more rock solid.
I've also removed duplicated code

Should fix other problems when parameters contains spaces in their value.

Now custom headers is fully effective and working fine

• Test command with global headers
• Test command with task headers
• Test command with both headers
• Test command without headers

Summary by Sourcery

Fix the issue with custom header parsing that caused httpx crashes, refactor command execution functions for improved stability, and enhance logging for better debugging.

Bug Fixes:

• Fix parsing of custom headers to prevent crashes in httpx and ensure proper HTTP request resolution.

Enhancements:

• Improve the robustness of the run_command and stream_command functions by refactoring and removing duplicated code.
• Add logging for debugging purposes in various functions, including fetch_url and generate_header_param.

[AnonymousWP]

Due to ocervell's experience, I think it's better if he (or another developer) from the team reviews this.

[psyray]

@sourcery-ai review

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request addresses a bug in custom header parsing for scans, improves the robustness of command execution, and enhances logging and error handling. The changes primarily affect the tasks.py and common_func.py files, focusing on refactoring the run_command and stream_command functions, as well as improving the generate_header_param function.

File-Level Changes

Change	Details	Files
Refactored and improved command execution functions	Separated command preparation logic into a new function prepare_command Created a new function create_command_object to handle database operations Improved error handling and logging in command execution Refactored run_command and stream_command for better code organization and readability	web/reNgine/tasks.py web/reNgine/common_func.py
Enhanced custom header parsing and parameter generation	Added debug logging to generate_header_param function Improved the logic for selecting the appropriate header format based on the tool name	web/reNgine/common_func.py
Improved URL handling and error checking in fetch_url function	Added checks for empty URL lists Improved logging for URL sources (user-provided vs. database-gathered)	web/reNgine/tasks.py
Added new utility functions for command execution and output processing	Created process_line function to handle output line processing Added write_history function to manage command history file operations Implemented execute_command function to standardize subprocess creation	web/reNgine/common_func.py

---

Tips

• Trigger a new Sourcery review by commenting @sourcery-ai review on the pull request.
• Continue your discussion with Sourcery by replying directly to review comments.
• You can change your review settings at any time by accessing your dashboard:
  • Enable or disable the Sourcery-generated pull request summary or reviewer's guide;
  • Change the review language;
• You can always contact us if you have any questions or feedback.

[sourcery-ai]

Hey @psyray - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟡 General issues: 1 issue found
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟡 Complexity: 1 issue found
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.}

[AnonymousWP]

I think this is the only thing I can find. Not sure how much effort I should put into testing every possibility.