Suppress default HTTP headers

[engn33r]

The HTTP headers User-Agent, Accept, and Accept-Encoding appear to always be set in HTTP requests, with no way to remove or suppress them. I believe that the Accept header is set in /modules/http/scanner.go, but can someone point me to where the User-Agent and Accept-Encoding headers are set for each HTTP request?

It would be great if there was a --suppress-default-headers flag to avoid these headers being added into every HTTP request and I can look into implementing it. I applaud PR #284 for the HTTP header customization features it introduces, but I think that suppressing the User-Agent and Accept-Encoding headers may require modifications outside of the /modules/http/scanner.go file.

[engn33r]

I had to modify several files (/lib/http/client.go, /lib/http/request.go, /lib/http/transport.go, /modules/http/scanner.go) and I almost certainly broke some things in the process, but I implemented this in my fork if anyone is interested. I don't think a PR of this would help in the current form I implemented it, except perhaps the flag I implemented to remove the Accept header, but let me know if I'm mistaken.