Can i use this tool/project for my research paper project ?

[avisekksarma]

Hello to the developers of this project. Words fail to convey the depth of my gratitude for the radiant joy that spread across my face when this tool scraped the tweets from twitter after trying out so many other tools/projects.

Now, I have following questions:

1. Will this project/repo be continued/maintained atleast upto today's level in coming days?
2. Are there any chances of twitter finding out we are using scrapers and blocking this tool or our ip address i.e. in any way can we be found out by twitter side and be terminated of our twitter account or blocked in ip address or terminate this github repo or are there any chances twitter change some frontend and break this tool?
3. Finally can i use,rely and trust this project on doing out my year long research project where i will be doing topic modeling and sentiment analysis of tweets in nepali language? I mean hopefully this tool works for long.

Once Again.Thank you

[pielco11]

Hi @avisekksarma and thank you so much for your kind words!

1. I'm worried I cannot answer to this. Better say that I do not have enough elements to compose a satisfying answer. We all are doing this in our free time and free of charge. I'd really like to keep a tool like this alive, but I don't know what will happen in the next future. Will is for granted but we cannot guarantee anything more.
2. Let me make a distinguo here. This tool is not communicating directly with Twitter, instead it scrapes a specific Nitter instance. So Twitter will only see a Nitter instance making requests and at least it will block that IP. This is why it's important that you setup your own custom Nitter instance in order to not compromise the experience of other users.
3. I hope this tool will last enough, too

Last but not least, please remember to cite all the contributors based on:

• PRs made
• Issues reported

Commits are public, so please refer to them as evidence.

Should you have any other question or concern, please feel free to reach out. You are more than welcome.

[archy-bold]

To add to this, Nitter search stopped working recently because of an internal change at Twitter. As far as I'm aware, it's not clear why that happened and the issue was reversed some days later. Twitter are doing everything they can to fight the public scraping of their data, so I wouldn't rely on this tool to work indefinitely. Since this project has an extra dependency beyond Twitter, it means issues may be outside the control of the contributors on this project and rely on the Nitter developers to fix.

There was a PR on the Nitter project (zedeus/nitter#830) that fixed those issues by forcing Nitter to send authenticated headers. This effectively makes it look like a logged in user is performing a search. Maybe this would be of interest to you? As this is less likely to break. But bear in mind that any Twitter user you use to login with may end up being banned for scraping.

I know they're planning their own REST API (zedeus/nitter#192), but there's no clear sign on when that will be available.

Sorry there's no better answer to give, Musk's takeover of Twitter has clearly screwed over research projects.