bug(scan): Global Variable - custom_header does not parse correctly

[psyray]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Submitted by @Zeetaz

When trying to run a scan utilzing the global variable "custom_header" it will be parsed differently depending on which tool it is assigned to.

For example, running the following YAML scanner config:

enable_http_crawl: true           # All tools
threads: 10                       # All tools

custom_header: 'User-Agent: Hello_I_Will_Die_PoC'

fetch_url: {
  'uses_tools': ['katana'],
}

It will break:

celery_1       |   warnings.warn(
celery_1       |
celery_1       | fetch_url                          | WARNING | Task fetch_url is RUNNING
celery_1       | fetch_url                          | ERROR | 'str' object has no attribute 'items'
celery_1       | Traceback (most recent call last):
celery_1       |   File "/usr/src/app/reNgine/celery_custom_task.py", line 129, in __call__
celery_1       |     self.result = self.run(*args, **kwargs)
celery_1       |   File "/usr/src/app/reNgine/tasks.py", line 1812, in fetch_url
celery_1       |     f'{key}: {value}' for key, value in custom_header.items()
celery_1       | AttributeError: 'str' object has no attribute 'items'
celery_1       | fetch_url                          | WARNING | Task fetch_url status is FAILED | Error: AttributeError("'str' object has no attribute 'items'")

If you instead were to run a vulnerability scan instead for fetch_url it will function correctly for nuclei.

It is possible to make this work as a temporary solution:

enable_http_crawl: true           # All tools
threads: 10                       # All tools

custom_header: {
  '"User-Agent': 'Hello_Working_PoC"'
}

fetch_url: {
  'uses_tools': ['katana'],
}

But this reuqires you to run two separate scans for vulnerability scans and fetching URL's... I am also not sure if it will break other tools or if it is only katana that is currently broken an expects "items" instead of strings.

Expected Behavior

The expected behavior, for most tools and for katan, would be to run the configuration as:

custom_header = 'User-Agent: testing'
etc, etc.

And it should be appended to the tool (katan in this case) as

-H "User-Agent: testing"

Steps To Reproduce

1. Run the following scanner configuration on any target:

enable_http_crawl: true           # All tools
threads: 10                       # All tools

custom_header: 'User-Agent: Hello_I_Will_Die_PoC'

fetch_url: {
 'uses_tools': ['katana'],
}

2. Note that the scan fails as it does not expect a str value.

3. Run the following - modified - configuration:

enable_http_crawl: true           # All tools
threads: 10                       # All tools

custom_header: {
  '"User-Agent': 'Hello_Working_PoC"'
}

fetch_url: {
  'uses_tools': ['katana'],
}

4. Note that this will work, however it does not follow the expected syntax. This is also an extremely bad work-around as it will not work with nuclei or any other tools as they won't parse it the same.

(it will append the header to katana as -H "User-Agent: Hello_Working_PoC")

Environment

- reNgine: 2.0.3
- OS: 
- Python: 
- Docker Engine: 
- Docker Compose: 
- Browser:

Anything else?

No response