Crawling Facebook pages creates multiple occurrences of "/unsupportedbrowser" link where there is only one in the page

[farjasju]

The facebook.com/unsupportedbrowser appears up to 50 times in one crawled page where there is only one such link in the original page (in fact, almost every link with an occurrence higher than 2 is facebook.com/unsupportedbrowser). Suspected issue in the redirection process.

[boogheta]

I think the issue comes from the redirection resolver pipeline here: https://github.com/medialab/hyphe/blob/master/hyphe_backend/crawler/hcicrawler/pipelines.py#L87
It seems like the ResolverAgent uses no UserAgent whatsoever neither any kind of header, worth trying to add it