-
Notifications
You must be signed in to change notification settings - Fork 200
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use of V2 API #445
Comments
Hi, I am also trying to use rtweet with the V2 API from twitter (with a developer account), and for some reason I am having a difficult time getting historical data from my timeline. I would like to pull the last 30 days form my timeline, but I am still just getting the results I would get if I just pull without the developers account. |
There is no 30 day search or user timeline option in the Twitter API V2 yet but these are both coming soon. We are as excited as you are to see rtweet expand to support V2! |
I would be very happy to contribute v2 API functionality, especially now since it seems Also, it seems like the internals are undergoing a fairly extensive re-write at the moment, largely by @hadley. @hadley, do you plan on implementing v2 API infrastructure, or will you have to switch your attention other projects soon? If not, is there any dev-facing documentation about the new internals to read to get oriented, or is the appropriate place to start in the code itself? |
Good questions @alexpghayes. I tried reaching @mkearney via several methods (twitter, github, email) but got no responses over several months. I requested to rOpenSci to maintain the package because I had some interest on new functionality and there were lots of issues and pending PR. @sckott gave me permissions like 2 weeks ago (2021/02/15), but we haven't talked about governance strategy, if anyone want to take over the maintenance for me it is ok. Initially I wanted to solve the issues and later on start making more profound changes without breaking changes (see #471). But since @hadley contributed with a new internal process to call Twitter's API and rewritten some functions this will no longer be possible. He is now working (temporary?) on other projects (see comments on #526 and #523). About the API v2 itself I haven't read much about it yet. As far as I know the path to the endpoint is different and now you can select which fields you want to get when making GET requests. You'll probably need to start modifying the TWIT_method to accept v2 endpoints and work from there (TWIT_GET and the new user facing function). There is no documentation of the internals. Hope this helps |
I looked into the v2 api very little but I wondered if it might be better to pull the hard code version number out of |
My impression from a day of playing with @hadley I can make a PR with a revision of |
@alexpghayes yeah, that’d be great. |
@alexpghayes once the PR is merged, I'd suggested by making a list of 1.1 vs 2 endpoints to figure out where we need to add behaviour vs change existing behaviour. It'll probably be easiest to start by adding functions for capabilities that are not available in v1.1, in order to get a sense of the API and what additional parameters the functions will need. I'd suggest picking one function to implement (maybe #363) then doing a PR to make a solid foundation for future work. |
Two key resources appear to be the migration guide: https://developer.twitter.com/en/docs/twitter-api/migrate and the data dictionary: https://developer.twitter.com/en/docs/twitter-api/data-dictionary/using-fields-and-expansions. In particular, the data returned by Based on the level of enthusiasm for https://github.com/cjbarrie/academictwitteR it seems like a good place to begin would be with the full archive search. I'll start on a PR this week. |
From the migration guide, the new API endpoints are:
|
I started on this at https://github.com/alexpghayes/rtweet/tree/v2 but the current bearer token interface is challenging enough to work with that I'm putting this on hold until #469 is resolved. |
Dear All, thanks a lot for your work!! Do you have a broad estimate when the usage of V2 API will be available? I would be particularly interested in searching by conversation_ids. |
Probably this summer I'll make a sprint on rtweet (if it is not included by then, I would include v2 API). Then I would like to get some feedback and would leave some time before finally submitting to CRAN. The best estimate I have of a CRAN release with v2 support is before the end of the year. Note that rtweet development version has a function to retrieve threads: tweet_threading. If tweet_threading doesn't work and you want to have sooner support from rtweet, you can send pull requests and we can work on it. I think that @alexpghayes was also interested and started working on this. |
Now that #542 is merged |
I'm sorry if there are elementary questions, I'm only asking as I'd like to help in anyway I can. Last year, after getting a response that V2 wasn't supported, I made a few changes to the rtweet internals and I got it working. (I had just changed the URLs that were being called from within rtweet.) I've gone through the thread a couple of times and also the various other issues mentioned here, but I couldn't get a few things.
Thanks. |
@sugarcane29 it's not a one-to-one mapping from v1 to v2 because more has changed than just the url (e.g. you can now select which fields to include in the results). So this makes it an opportunity to revisit rtweet's API to make bigger changes. I think it would be a missed opportunity to just change the URLs and not reconsider the large API. |
Hi @sugarcane29, many thanks for opening the issue and wanting to help. Help maintaining this package is always welcomed!
Hope this helps to clarify your doubts. |
Just in case it wasn't abundantly clear from me being totally AWOL, |
just coming into this discussion - anyone taking this up? |
@billmclellan I'm currently doing other things. But certainly this would be very much welcomed. |
I've not created a package before, but I've done lots of functions and scripts for my own analysis. Here's what I've started for myself using httr instead of rtweet, as suggested by Twitter for their v2 api. Am I on the right track? |
@billmclellan This is a first step. The attached code is fine for a script but not for a package. The code to use twitter's API v2 should be inside the code of the package (you'll need to fork and modify/add files) and use internal functions of rtweet in order to be added (to ensure user consistency, for instance here you don't handle API rate limits or check users input). |
thanks @llrs that's what I thought - thanks for confirming |
Link to Twitter's documentation on Getting started with R and v2 of the Twitter API (though note that |
@brshallo This was fixed a little while back on our end. |
Thanks Jessica, for checking in and reporting the updates. To everybody here: I hope in a couple of weeks to retake the development of rtweet (I've moved to the devel branch, if you don't see any change). |
Update to all interested in this issue! The latest rtweet release 1.1.0 supports the streaming endpoints. Next release will support the archive an recent tweets among others because I can easily support all the endpoints requiring OAuth 2.0 App Only (With the bearer token) Unfortunately, I have some problems authenticating via Oauth 2.0 with PKCE, see this comment for a brief summary, which means no support for bookmarks yet (#344) and other endpoints requiring it. I don't want to add support for endpoints that could work via OAuth 1.0; I expect its support might end soon, although I might find a way to support it easily. See this table of endpoints and authentications to know other endpoints afected by this roadblock. Some decisions I am facing in case someone wants to add their opinion:
Thanks all for you patience. |
Hello,
|
Thanks for sharing your preferences @erima2020 . There is the While I agree on maintaining the old output for old code is always nice (which I break in the 1.0.2 release), the current parsing of the data to generate the output is slow (imho) and forces for instance, a user interested in media to retrieve everything. Given that old code will be updated I think it is a good opportunity to provide faster and better interface for the users (or completely abandon the package :/). As discussed privately the |
Now (since version 1.1 for the streaming endpoints) it is possible to use the API v2. |
Is there a possibility using this package with the Twitter V2 API.
I'm trying to do some historical search and it appears that the V2 API allows for it in the free version.
Ref: start_time and end_time in the api: https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-recent
I tried changing the query parameters in the source and rebuilding the package, but it throwing lots of errors.
Thanks.
The text was updated successfully, but these errors were encountered: