Is it just me or have 429 errors really increase these past few days? #538
Comments
|
Experiencing the same issue with the node library from @pat310 lately. |
|
Any workarounds? |
|
Same! the only workaround I was able to do is to re-run the code multiple times in different machines or using colab with different account. |
|
Same here, they are locking down folks. |
|
How is there no reliable solution for this in 2022 😢 - was getting issues on pat's node js proj and was about to try this one, but then saw same thing happening |
|
Supporting an external API for Google Trends isn't a priority for Google.
…On Sat, Oct 29, 2022 at 7:27 AM, Roy Hermann < ***@***.*** > wrote:
How is there no reliable solution for this in 2022 😢
—
Reply to this email directly, view it on GitHub (
#538 (comment)
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AAIWU4PQD42HKDQLFUPJJYDWFUX63ANCNFSM6AAAAAAROMMXOI
).
You are receiving this because you are subscribed to this thread. Message
ID: <GeneralMills/pytrends/issues/538/1295852672 @ github. com>
|
|
Not facing the issue after using proxies and sleep 60 seconds after each request. |
Do you mind sharing where you're getting proxy from and how you're implementing them? |
I’m using proxies from newipnow. I’m using 25 proxies and using random proxy on each request. |
|
Guys, I just figured that if you downgrade the pytrend library to 4.7.2 or 4.7.3 it works. Also, collecting data for different geographical locations may stop the process, use only one location at a time, with up to 5 keywords. For more than 5 keywords, you need to apply normalization, and that is by using one shared keyword as a control in all sets of 5 keywords. |
|
Alternatively, you may want to try R instead of Python: https://cran.r-project.org/web/packages/gtrendsR/gtrendsR.pdf |
This worked for me! Thanks for the fix! |
|
Would be great to figure out why downgrading solves this.
…On Tue, Nov 1 2022 at 01:21, Josh Hascall < ***@***.*** > wrote:
>
>
> Guys, I just figured that if you downgrade the pytrend library to 4.7.2 or
> 4.7.3 it works. Also, collecting data for different geographical locations
> may stop the process, use only one location at a time, with up to 5
> keywords. For more than 5 keywords, you need to apply normalization, and
> that is by using one shared keyword as a control in all sets of 5
> keywords.
>
>
This worked for me! Thanks for the fix!
—
Reply to this email directly, view it on GitHub (
#538 (comment)
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AAIWU4LC6FE6BG5DSUQSFSDWGCSHJANCNFSM6AAAAAAROMMXOI
).
You are receiving this because you commented. Message ID: <GeneralMills/pytrends/issues/538/1298044590
@ github. com>
|
Update: now it doesn't work... It only took 12 hours for them to block it |
|
It is still working for me! try to increase the sleep time between requests for each subset of keywords (I use random number between 5 and 30 seconds wait). Also, don't use only one machine with the same IP address, alternate between your machine and Google Colab. |
|
Is it working because you downgraded or because your querying strategy has changed?
…On Thu, Nov 3 2022 at 00:49, Reem Omer < ***@***.*** > wrote:
It is still working for me! try to increase the sleep time between
requests for each subset of keywords (I use random number between 5 and 30
seconds wait). Also, don't use only one machine with the same IP address,
alternate between your machine and Google Colab.
—
Reply to this email directly, view it on GitHub (
#538 (comment)
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AAIWU4MQEMHX7QEYYPRONETWGM75HANCNFSM6AAAAAAROMMXOI
).
You are receiving this because you commented. Message ID: <GeneralMills/pytrends/issues/538/1301636319
@ github. com>
|
|
Both! I figured that even if I downgraded and sent too many requests, Google may block my IP, therefore, I had to change the IP this can be done by using proxies, vpn, Google Colab has also different range of IPs. |
|
How do you know the downgrade is the cause of success? Could you help identify any changes?
…On Thu, Nov 3 2022 at 03:46, Reem Omer < ***@***.*** > wrote:
Both! I figured that even if I downgraded and sent too many requests,
Google may block my IP, therefore, I had to change the IP this can be done
by using proxies, vpn, Google Colab has also different range of IPs.
—
Reply to this email directly, view it on GitHub (
#538 (comment)
) , or unsubscribe (
https://github.com/notifications/unsubscribe-auth/AAIWU4KKFZKX2R5XJCTUQNTWGNUVJANCNFSM6AAAAAAROMMXOI
).
You are receiving this because you commented. Message ID: <GeneralMills/pytrends/issues/538/1301743075
@ github. com>
|
|
@ReemOmer I am curious, how do you use Google Colab to scrape? I don't believe they have an API... And I haven't found any guide like that.... |
|
@emlazzarin honestly I didn't open any of the main files in both versions, so I don't know what the difference is. |
|
I have an extremely novice understanding of the inner-workings of the package, but could this problem have something to do with cookies expiring on the trends.google site? I previously have been able to workaround 429 errors with this solution but now that doesn't work either. Scrolling through the request headers, I noticed a cookie expiration time that ends in the same minute as submitting the request. |
|
Looks like no user-agent is specified in the requests, meaning they are blocked more often. I fixed it here: 18f230d |
|
Fixed by #553. |
I used to get 429 error every time I request more than 6 items per hour or so.
But recently, especially today I am not able to request more than 1 per hour without getting 429. Is it just my IP acting up?
The text was updated successfully, but these errors were encountered: