Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throttle feed loading to avoid YouTube rate limits #11743

Merged
merged 2 commits into from
Nov 24, 2024
Merged

Conversation

Stypox
Copy link
Member

@Stypox Stypox commented Nov 24, 2024

What is it?

  • Bugfix (user facing)
  • Feature (user facing)
  • Codebase improvement (dev facing)
  • Meta improvement to the project (dev facing)

Description of the changes in your PR

This PR introduces a delay between every BATCH_SIZE=50 channel extractions while loading the feed, to avoid YouTube rate limits and hopefully solve #11661. The delay varies in range DELAY_BETWEEN_BATCHES_MILLIS=(6000..12000) (on average 8s). I also lowered PARALLEL_EXTRACTIONS from 6 to 3, as suggested by AudricV. The value of BATCH_SIZE was chosen because people reported problems starting from ~100 subscriptions, so 50 should be safe.

Before/After Screenshots/Screen Record

In this video BATCH_SIZE=10 and DELAY_BETWEEN_BATCHES_MILLIS=(2000..8000) for demonstration purposes.

screen-20241124-140028.mp4

Fixes the following issue(s)

APK testing

The APK can be found by going to the "Checks" tab below the title. On the left pane, click on "CI", scroll down to "artifacts" and click "app" to download the zip file which contains the debug APK of this PR. You can find more info and a video demonstration on this wiki page.

Due diligence

@github-actions github-actions bot added the size/small PRs with less than 50 changed lines label Nov 24, 2024
@ShareASmile ShareASmile added the youtube Service, https://www.youtube.com/ label Nov 24, 2024
@TobiGr TobiGr added the feed Issue is related to the feed label Nov 24, 2024
@AudricV AudricV mentioned this pull request Nov 24, 2024
7 tasks
Copy link
Member

@TobiGr TobiGr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code looks good, did not test the latest change

Copy link

sonarcloud bot commented Nov 24, 2024

@Stypox
Copy link
Member Author

Stypox commented Nov 24, 2024

I tested it on my phone with BATCH_SIZE=10 and the number of subscriptions did not always increase by exactly 10, indicating that non-YouTube channels do not count towards the throttling.

@Stypox Stypox merged commit acf2e88 into release-0.27.3 Nov 24, 2024
6 checks passed
@TobiGr TobiGr deleted the slower-feed branch November 24, 2024 15:35
@Stypox Stypox mentioned this pull request Nov 24, 2024
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feed Issue is related to the feed size/small PRs with less than 50 changed lines youtube Service, https://www.youtube.com/
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants