Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AENetworks] Update extractor for the current site(s) #26795

Closed
wants to merge 1 commit into from

Conversation

kevinoconnor7
Copy link
Contributor

@kevinoconnor7 kevinoconnor7 commented Oct 3, 2020

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

  • Bug fix
  • Improvement
  • New extractor
  • New feature

Description of your pull request and other information

This gets the AENetworks extractor back into a working state. The highlights are:

  • Support for the play subdomain that the content is now hosted on
  • Extract ThePlatform URLs from the GraphQL endpoint they now use
  • Properly extract video information from the updates made to the non-play subdomain sites.
  • Properly extract the series name, season number, and episode numbers
  • Added more explicit error messages for when extraction fails

Fixes #23363

@yemaozitw
Copy link

Works for shows but not specials like:

https://play.aetv.com/specials/voices-magnified-locked-up-in-america/full-special
https://play.aetv.com/specials/biography-i-want-my-mtv-2/full-special
https://play.aetv.com/specials/the-nine-lives-of-ozzy-osbourne/full-special

These get the following errors:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--hls-prefer-native', '--cookies', 'cookies.txt', '--fragment-retries', '35', '--retries', '35', '--all-subs', '--sub-format', 'dfxp', 'https://play.aetv.com/specials/voices-magnified-locked-up-in-america/full-special']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.09.20
[debug] Python version 3.7.3 (CPython) - Linux-4.19.0-10-amd64-x86_64-with-debian-10.6
[debug] exe versions: ffmpeg 4.1.6-1, ffprobe 4.1.6-1, rtmpdump 2.4
[debug] Proxy map: {}
[aenetworks] voices-magnified-locked-up-in-america: Downloading webpage
WARNING: unable to extract videoId; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[aenetworks] Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 400: Bad Request (caused by <HTTPError 400: 'Bad Request'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "/home/ymz/bin/youtube-dl/youtube_dl/extractor/common.py", line 632, in _request_webpage
return self._downloader.urlopen(url_or_request)
File "/home/ymz/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2238, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
File "/usr/lib/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python3.7/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)

@kevinoconnor7
Copy link
Contributor Author

Hmm yeah specials are weird. They don't have the video ID in the normal spot and the only place I can find it on the page is within a giant JSON blob. Unfortunately it's mixed in with others. I'll need to dig into the JSON a bit and compare a few samples to see if I can find a reliable way to find the correct one.

@yemaozitw
Copy link

If specials are too difficult then might as well go forward with this PR since it fixes most of the problem.

@rredford6
Copy link

@kevinoconnor7 Does the GraphQL server provide different playlists? There's an issue with ads getting injected: #26047

@kevinoconnor7
Copy link
Contributor Author

@rredford6 Not that I'm aware, though I'm honestly not super familiar with the entire The Platform flow. When I wrote this PR I mostly tried to extract the data that had previously been used (i.e. getting the public The Platform URL).

Copy link

@merval merval left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Nice work. Was able to successfully download an episode

This gets the AENetworks extractor back into a working state. The
highlights are:

  * Support for the `play` subdomain that the content is now hosted on
  * Extract ThePlatform URLs from the GraphQL endpoint they now use
  * Properly extract video information from the updates made to the
  non-play subdomain sites.
  * Properly extract the series name, season number, and episode numbers
  * Added more explicit error messages for when extraction fails

Fixes ytdl-org#23363
@remitamine remitamine closed this in fa604d9 Dec 6, 2020
ThirumalaiK pushed a commit to ThirumalaiK/youtube-dl that referenced this pull request Jan 28, 2021
- Fix Fastly format extraction
- Add support for play and watch subdomains
- Extract series metadata

closes ytdl-org#23363
closes ytdl-org#23390
closes ytdl-org#26795
closes ytdl-org#26985
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AETV.com Unable to extract video url
4 participants