Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem about Pytrends build_payload #567

Closed
ziqingwang-kate opened this issue Mar 20, 2023 · 6 comments
Closed

Problem about Pytrends build_payload #567

ziqingwang-kate opened this issue Mar 20, 2023 · 6 comments

Comments

@ziqingwang-kate
Copy link

Dear Sir or Madam,

I hope this email finds you well. This is Ziqing Wang, a research associate from Stanford GSB. We are recently using Pytrends to conduct a research project. I am writing to inquire some questions about pytrends.

From March 8th, we always got the error message "TooManyRequestsError code 429" when running the function “pytrends.build_payload”. Even people who use the pytrends for the first time with a new IP address get the same error.

I am so appreciated it if you can give me some suggestions about this. Or are there any alternative functions in pytrends that can scrape historical volume data of Google Trends with keywords?

Many thanks,
Ziqing

@jiangcheng1999
Copy link

use : pytrend = TrendReq(retries=3), another way is to swtich your proxy constantly, but then you need to find some valid proxies

@jfabian14085
Copy link

Having the same problem, seems like Google made some changes to prevent any sort of scraping.

@JacobFredman
Copy link

Having the same problem too,
I don't now if it's new, but Google requests uniq token per request - even if you do a slightly change in your request - you need a new token that suit to that request.

Terseus added a commit to Terseus/pytrends that referenced this issue Mar 26, 2023
Google Trends slightly changed its HTTP requests calls:

- the "get cookie" request changed its URL, needed now an extra
  "explore/"; this was the main reason of the premature 429 error
  responses.
- the "widgets" request is now POST instead of GET.

Before this change the requests returned almost consistently 429 error
responses, now it returns results again.

Fixes GeneralMills#566, GeneralMills#567, GeneralMills#561
@sripriya97
Copy link

@ziqingwang-kate #561 (comment) check this out, adding NID cookie worked for me.

1 similar comment
@sripriya97
Copy link

@ziqingwang-kate #561 (comment) check this out, adding NID cookie worked for me.

@emlazzarin
Copy link
Collaborator

Closed because this should be resolved in #570.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants