http error 405

[wangxisea]

I got 405 when I ran it. it says: HTTP status code is not handled or not allowed.
Would you mind to take a look? Thanks.

[gijs]

This is due to Funda blocking crawlers. Configuring a proxy middleware in Scrapy may help but I didn't try that. Good luck

[jobvisser03]

I tried configuring the settings.py by including:
DOWNLOADER_MIDDLEWARES = { 'funda.middlewares.MyCustomDownloaderMiddleware': 543, 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, }

This doesn't seem to work unfortunately, is there anything else one needs to take care of? Does anyone have experience with this?

[gijs]

I think Funda has recently put some sort of rate limiter in place. It detects robots based on several parameters. They suspected me being a robot anyway, as they prompted me with a captcha.

Scrapy can probably solve the captcha but I didn't look into that.. https://github.com/pombredanne/decaptcha

I'm curious if anyone can get it to work again

[aliaamin]

Same problem here, would appreciate if anyone can help wih some tips how to overcome the 405 error.

[gijs]

I'm building a Funda scraper based on headless Chrome. Whenever a captcha is detected, the scraper takes a screenshot and sends it to me via Telegram. I can reply with the two words which the scraper uses to solve the captcha and continue scraping.

[igorkoehne]

Is your scraper working properly? I was trying to use selenium, but I am not even sure if this would be the correct way to go, since I am just starting in this world. If you could share your code it would be the best thing that happened to me this year o/

[gijs]

@igorkoehne (I assume you're talking to me) - unfortunately I cannot share this specific codebase at the moment because it contains a bunch of API keys / needs cleaning up - and I have no time for that now.

In the meantime, Google came up with Puppeteer. Building your own captcha-evading scraper should be even easier using this highlevel API for Headless Chrome. I'm going to rewrite my own scraper to use it, too.

[gijs]

In other news, detection of unmodified versions of Headless Chrome seems easy... mostly because headless doesnt have WebGL capabilities which can be sniffed.

If Funda is already detecting Headless Chrome, sticking to Selenium's Chrome Webdriver will be a better option.

Good luck scraping them!

[igorkoehne]

Thanks for the tips, I will give it a try!

[khpeek]

As a quick reply, the 405 error appears to be the result of fingerprinting of headless browsers by Funda. I managed to circumvent it by (1) changing my user agent (using Scrapy Random User Agent), (2) using the Scrapy Splash plugin.

[AntoniosMavropoulos]

Do both (1) and (2) need to be in place?
If yes, could you please post the code that you used?
Thanks!

[tangvip]

@khpeek
do you mean that you need to use both methods OR either one of them can solve the problem?
Thanks!

[arnabsinha4u]

@tangvip with the usage of just (1) Scrapy Random User Agent, the error persists. Have not tried it with Scrapy Splash plugin

[MarcDuQuesne]

Hi folks, any update?

[arnabsinha4u]

I have done away with scrapping the website. Instead, I am using RSS feeds which are parameterized and serves the purpose. RSS feeds have the latest details and not historic but ofcourse, over time you can create your own history, should that be a need.

[Kalli]

@arnabsinha4u could you please tell me where you find those RSS feeds you mention?
Do you mean something along the lines of these: http://partnerapi.funda.nl/feeds/Aanbod.svc/rss/?type=koop&zo=/amsterdam/

Is there a feed that has the postal codes as well?

[Suidgeest]

Hi Kurt, mind posting your latest working code (referring to your comment Sep 6th, 2017) Thank you!

[fab343]

Hi all, any updates on the problem?

[fab343]

I have done away with scrapping the website. Instead, I am using RSS feeds which are parameterized and serves the purpose. RSS feeds have the latest details and not historic but ofcourse, over time you can create your own history, should that be a need.

are you talking about this rss: http://partnerapi.funda.nl/feeds/Aanbod.svc/rss/?type=koop&zo=/amsterdam/ ?