-
Notifications
You must be signed in to change notification settings - Fork 309
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
http: allow raw header capture (#347) #349
Conversation
The golang textproto library does a few things when parsing the HTTP headers: * consume some whitespace characters (e.g. \r\n) * canonicalizes the header keys (e.g. "content-type" => "Content-Type") * moves the headers into a map This all makes sense when parsing HTTP, but for a scanner some may want to have the exact headers, to match on order, non-canonical keys, etc. This adds that option, if '--raw-headers' is specified during an HTTP scan. This is accomplished by implementing a tee reader on the pconn interface, that tees before the bufio reader is put in place. The tee copy can be disabled once the headers have been read, so as to not waste memory while consuming the HTTP body. While denoted as "raw headers", this will also capture the raw status line as well. (cherry picked from commit 83e55e0) Signed-off-by: Jeff Cody <jcody@censys.io>
@codyprime excellent! Thank you |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good to me.
Same as the other PR that was approved & sanity tested against a few hosts in the wild with interesting headers.
@codyprime the "--raw-headers" extension works well until you use "--max-redirects=1" and hit a device that does a redirect. In that case, zgrab2 panics, as shown below:
zgrab2 does not panic in the following combination of flags:
|
@dav3 Thanks, I can confirm. Working on a fix now, I'll push it up to this PR and cherry-pick it for the TLS 1.3 feature branch when done |
See also: #347
The golang textproto library does a few things when parsing the HTTP
headers:
This all makes sense when parsing HTTP, but for a scanner some may want
to have the exact headers, to match on order, non-canonical keys, etc.
This adds that option, if '--raw-headers' is specified during an HTTP
scan. This is accomplished by implementing a tee reader on the pconn
interface, that tees before the bufio reader is put in place. The
tee copy can be disabled once the headers have been read, so as to not
waste memory while consuming the HTTP body.
While denoted as "raw headers", this will also capture the raw status
line as well.
(cherry picked from commit 83e55e0)
Signed-off-by: Jeff Cody jcody@censys.io
How to Test
stdout output should then look something like:
Notes & Caveats
This has already been merged & tested on the TLS 1.3 feature branch