should take the first title when there are multiple title

[AaronChen0]

echo https://blog.thelevelup.com/ | httpx -title
Get: https://blog.thelevelup.com [instagram]
Expect: https://blog.thelevelup.com [Blog | LevelUp | Mobile Ordering, Restaurant Loyalty, Customer Engagement, and CRM |]

When a page has multiple <title>...</title>, the first match probably is the real title.

[ehsandeep]

Hi @AaronChen0, there are multiple variations of titles, and might be not possible to cover all the cases with pre-defined rules, keeping this issue in mind, we added support for custom extractors based on user-defined custom regex, i.e using -extract-regex

[ehsandeep]

@AaronChen0 httpx internally uses hardcoded regex only to detect/extract these titles, so regex is what you or we need to fix it, as mentioned we have identified multiple cases that can not satisfy all the cases, but let me know if you have a solution with a better regex to capture all the cases.

httpx/common/httpx/title.go

Line 15 in bbc8e48

reTitle = regexp.MustCompile(`(?im)<\s*title.*>(.*?)<\s*/\s*title>`)

[AaronChen0]

Complex regex would slow down httpx. In this title case, can we just take the first match? Do you have a good reason to take the last match (the current httpx behavior)?

[ehsandeep]

@AaronChen0 please go through the multiple issues we have closed regarding title extraction and cases that need to consider to get maximum and valid coverage - https://github.com/projectdiscovery/httpx/issues?q=is%3Aissue+title+is%3Aclosed, at the same time we also understood that it's not possible to capture all the cases, in other word if we update regex to capture new case will result into missing old ones, for the very same reason we added support to let the user define when required.

[AaronChen0]

I know nothing about go, but this shouldn't need to change the regex. I want change in after-regex. When the go code have a match list for the regex, why it just take the last one, not the first one. Do you have a reason to do that? I have viewed the other title issues, but none is similar to this one.

[ehsandeep]

Hey @AaronChen0, apologies for this, somehow I overlooked the base question/issue you pointed out here, as this is a valid concern I've created GH ticket for this #539