Fix user.videos crawling issue

[lhphat02]

Issue: Can't crawl videos from user.videos more than 35 videos (even if using cursor).

The code I used for testing:

async def get_user_videos(username):
    start_time = time.time()
    row_count = 0

    async with TikTokApi() as api:
        await api.create_sessions(headless=False, ms_tokens=[ms_token], num_sessions=1, sleep_after=3)
        user = api.user(username)
        user_data = await user.info()
        post_count = user_data["userInfo"]["stats"].get("videoCount")

        async for video in user.videos(count=post_count):
            url = f"https://www.tiktok.com/@{video.as_dict['author']['uniqueId']}/video/{video.id}"
            print(f"URL: {url}") 
            row_count += 1

    end_time = time.time()
    elapsed_time = end_time - start_time
    print(f"Execution time: {elapsed_time} seconds")
    print(f"Total rows: {row_count}")
    print(f"Rows per second: {row_count / elapsed_time}")

Before modifying the videos method:

2024-04-13 14:20:04,220 - TikTokApi.tiktok - ERROR - Got an unexpected status code: {'log_pb': {'impr_id': '202404130720035F6460F88AF0BF0E31DB'}, 'statusCode': 10201, 'statusMsg': '', 'status_code': 10201, 'status_msg': ''}
Execution time: 11.261611223220825 seconds
Total rows: 0
Rows per second: 0.0

After modifying the videos method:

URL: https://www.tiktok.com/@sofm_official/video/6817297421245107457
URL: https://www.tiktok.com/@sofm_official/video/6815619623837289729
...
URL: https://www.tiktok.com/@sofm_official/video/6815419939957017857
URL: https://www.tiktok.com/@sofm_official/video/6815113300146228481
URL: https://www.tiktok.com/@sofm_official/video/6814374629558258945
Execution time: 14.023724794387817 seconds
Total rows: 135
Rows per second: 9.626543730666045

Please check this

[anarchopythonista]

Applying this patch locally fixed the issue I was experiencing with the 6.3.0 release. Thank you!

[mi01]

This fix is breaking the count parameter, since the function will always return multiples of 35 (or less if the number of videos is smaller). But it might work if we add an additional break statement in this loop:

TikTok-Api/TikTokApi/api/user.py

Lines 134 to 136 in 7101654

	for video in resp.get("itemList", []):
	yield self.parent.video(data=video)
	found += 1

for video in resp.get("itemList", []):
    yield self.parent.video(data=video)
    found += 1
    if found == count:
        break

Still the cursor parameter is useless and confusing for the user of this function.

[jesse-moderwell]

Applying this patch locally fixed the issue I was experiencing with the 6.3.0 release. Thank you!

Same!

[vagvalas]

Same here, tried with hashtag.py got limit on 35-40..
tried to figured out what did you done here, i dont quietly understand completely.
I found
/TikTokApi/api/user.py

     found = 0
        while found < count:
            params = {
                "secUid": self.sec_uid,
                "count": count,
                "cursor": cursor,
            }

but in hashtag.py is already the changed version of yours, but doesnt crawl more..

/TikTokApi/api/hashtag.py

        found = 0
        while found < count:
            params = {
                "challengeID": self.id,
                "count": 35,
                "cursor": cursor,
            }

So for hashtag doesn't the 35 multiples works.. just fetches 35 and stops..
am i missing something?

[davidteather]

LGTM

[wouterdedroog]

This fix is breaking the count parameter, since the function will always return multiples of 35 (or less if the number of videos is smaller). But it might work if we add an additional break statement in this loop:

I noticed this after updating to 6.4.0 from 6.3.0. Does requesting videos in increments of 35 cause any additional requests (instead of requesting just 5 videos)? @mi01

[mi01]

This fix is breaking the count parameter, since the function will always return multiples of 35 (or less if the number of videos is smaller). But it might work if we add an additional break statement in this loop:

I noticed this after updating to 6.4.0 from 6.3.0. Does requesting videos in increments of 35 cause any additional requests (instead of requesting just 5 videos)? @mi01

This causes no additional request AFAIK.