[YouTube] Update clients versions, restore access to some streams and more #1168
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The API keys are not used anymore by official clients in almost all cases (still used by the Android app until it gets a configuration) for all requests we made. Clients and device OS versions have been bumped to their latest stable version known. Methods and fields related to API keys have been renamed or deleted if they're no longer relevant.
YouTube disabled the effectiveness of the parameters which were used (the player response we get redirects to another video), but new parameters which work around Android's client integrity checks have been found.
The new action data can return multiple contact actions instead of only one, which will be concatenated by a new line return. This commit fixes tests of the CrisisResources test class of YoutubeSearchExtractorTest.
YouTube now returns a Shorts tab for InteractiveTabbedHeader gaming channels, which contains Shorts about the game of the topic channel but are not uploaded on the game topic channel. As this tab is already supported by the extractor, fetching a gaming topic channel now returns a tab instead of none. Channel name and channel URL of these Shorts needs to be set to null in a separate commit, as Shorts on this tab do not have the topic channel as their uploader.
AudricV
added
bug
|
I would like to know more about this change. What is DroidGuard's VM? And how did you find out the new player parameters? |
Stypox
requested changes
Apr 16, 2024
...actor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeMetaInfoHelper.java
Outdated
Show resolved
Hide resolved
...actor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeMetaInfoHelper.java
Outdated
Show resolved
Hide resolved
extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
Outdated
Show resolved
Hide resolved
extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
Outdated
Show resolved
Hide resolved
extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
You renamed a lot of things in this class, maybe it would be better to choose a naming that is independent of implementation details? E.g. isHardcodedClientValid instead of isHardcodedClientVersionValid. No need to make changes right now unless you have a clear idea, though.
...in/java/org/schabi/newpipe/extractor/services/youtube/extractors/YoutubeStreamExtractor.java
Show resolved
Hide resolved
...src/test/java/org/schabi/newpipe/extractor/services/youtube/YoutubeChannelExtractorTest.java
Outdated
Show resolved
Hide resolved
See this presentation and reVanced threads about fixing the integrity checks which caused playback issues and our current failure to get streams from the ANDROID client in the extractor: https://www.romainthomas.fr/publication/22-sstic-blackhat-droidguard-safetynet/
|
Stypox
approved these changes
Apr 20, 2024
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates YouTube hardcoded clients' versions to newer ones and removes usage of InnerTube API keys, as official clients do not use them anymore in all endpoints we use.
It also restores the availability some streams (itag 22 on musical videos, itag 139 (+DRC audio streams - used for stable audio feature on mobile apps), when iOS client fetch is not forced, itags 599 and 600) by using new player parameters found to workaround the integrity checks on the official Android app, which requires to pass DroidGuard's VM and to associate a visitor ID to InnerTube requests.
Finally, it adds the missing
prettyPrintquery parameter to mixes continuations and implements the new crisis meta info action data, while fixing YouTube tests and updating mocks on the fly.Due to the removal of the usage of InnerTube API keys, some methods in
YoutubeParsingHelperwhich can be used by clients have been changed:areHardcodedClientVersionAndKeyValidwas renamed toisHardcodedClientVersionValidgetKeywas removed, since there is no InnerTube API key anymoreresetClientVersionAndKeywas renamed toresetClientVersionisHardcodedYoutubeMusicKeyValidwas renamed toisHardcodedYoutubeMusicClientVersionValidgetYoutubeMusicKeywas renamed togetYoutubeMusicClientVersionand now just returns the version instead of (key, name, version), since there is no key anymore and the name is fixed ("67") and therefore hardcodedCloses #1112, as it is not relevant anymore, and fixes multiple issues which have been not reported as issues in the extractor/NewPipe's issue tracker.