Bug - Scanning issues after recent merge/ commits

[alph4byt3]

Issue Description

Scans stop working after subdomain discovery but before probing subdomains for endpoints. It seems to be happening during the HTTPX part of the scan.

Here's a normal vuln scan using the default engine in a newly installed instance.

The build I was using before Thursday 26th was fine and that was the day new changes were committed, today I reinstalled rengine due to a power outage hence why I'm now here.

• I have confirmed that this issue can be reproduced as described on a latest version/pull of reNgine: (yes / no)

yes

Technical details

None

[github-actions]

👋 Hi @alph4byt3,
Issues is only for reporting a bug/feature request. Please read documentation before raising an issue https://rengine.wiki
For very limited support, questions, and discussions, please join reNgine Discord channel: https://discord.gg/azv6fzhNCE
Please include all the requested and relevant information when opening a bug report. Improper reports will be closed without any response.

[yogeshojha]

Weird! Let me try immediately on my older VPS build. But I don't think any changes has happened in scanning part, which would fail this. anyways let me check

[yogeshojha]

How did you update btw? I assume

make down && git pull && make build && make up

right?

[yogeshojha]

@alph4byt3 I can confirm everything is working as expected. My build was from Aug 16 and updated to the most recent. It is working as expected.

[wapwn]

@alph4byt3 @yogeshojha
that's because httpx just updated, the returned json data is a bit different
in the http_crawler function you just need to change the subdomain query to

subdomain = Subdomain.objects.get(scan_history=task, name=json_st['input'])

original code

subdomain = Subdomain.objects.get(scan_history=task, name=json_st['url'].split("//")[-1])

[alph4byt3]

How did you update btw? I assume

make down && git pull && make build && make up

right?

Hey what's up, I didn't update it. I'm on a VM so I went back to a previous screenshot from a couple of days ago without reNgine installed and reinstalled it from scratch (git clone etc etc)

I'm quite busy at the moment so when I get free time later I'll go check on it again for any other things I can find.

I saw httpx did update something regarding json output so maybe wapwn above is correct

[yogeshojha]

Hi @wapwn Thank you very much for pointing this out! You're awesome 🌟

This has been fixed, tested on the most recent version of HTTPX. If this problem still persists, feel free to raise a new GitHub issue, or we can reopen this.

[yogeshojha]

Thanks @alph4byt3 for reporting this.