diff --git a/README.md b/README.md
index 170c85c48..ecadd622f 100644
--- a/README.md
+++ b/README.md
@@ -421,7 +421,7 @@ I will add some memorable short links to the binaries so you can download them e
out, youtube-dlc will ask interactively.
-2, --twofactor TWOFACTOR Two-factor authentication code
-n, --netrc Use .netrc authentication data
- --video-password PASSWORD Video password (vimeo, smotri, youku)
+ --video-password PASSWORD Video password (vimeo, youku)
## Adobe Pass Options:
--ap-mso MSO Adobe Pass multiple-system operator (TV
@@ -734,6 +734,7 @@ Also filtering work for comparisons `=` (equals), `^=` (starts with), `$=` (ends
- `container`: Name of the container format
- `protocol`: The protocol that will be used for the actual download, lower-case (`http`, `https`, `rtsp`, `rtmp`, `rtmpe`, `mms`, `f4m`, `ism`, `http_dash_segments`, `m3u8`, or `m3u8_native`)
- `format_id`: A short description of the format
+ - `language`: Language code
Any string comparison may be prefixed with negation `!` in order to produce an opposite comparison, e.g. `!*=` (does not contain).
diff --git a/docs/supportedsites.md b/docs/supportedsites.md
index 0b183b272..d0a545f7e 100644
--- a/docs/supportedsites.md
+++ b/docs/supportedsites.md
@@ -34,6 +34,8 @@
- **adobetv:video**
- **AdultSwim**
- **aenetworks**: A+E Networks: A&E, Lifetime, History.com, FYI Network and History Vault
+ - **aenetworks:collection**
+ - **aenetworks:show**
- **afreecatv**: afreecatv.com
- **AirMozilla**
- **AliExpressLive**
@@ -52,9 +54,11 @@
- **Aparat**
- **AppleConnect**
- **AppleDaily**: 臺灣蘋果日報
+ - **ApplePodcasts**
- **appletrailers**
- **appletrailers:section**
- **archive.org**: archive.org videos
+ - **ArcPublishing**
- **ARD**
- **ARD:mediathek**
- **ARDBetaMediathek**
@@ -94,6 +98,10 @@
- **BellMedia**
- **Bet**
- **bfi:player**
+ - **bfmtv**
+ - **bfmtv:article**
+ - **bfmtv:live**
+ - **BibelTV**
- **Bigflix**
- **Bild**: Bild.de
- **BiliBili**
@@ -101,6 +109,7 @@
- **BilibiliAudioAlbum**
- **BiliBiliPlayer**
- **BioBioChileTV**
+ - **Biography**
- **BIQLE**
- **BitChute**
- **BitChuteChannel**
@@ -110,6 +119,7 @@
- **blinkx**
- **Bloomberg**
- **BokeCC**
+ - **BongaCams**
- **BostonGlobe**
- **Box**
- **Bpb**: Bundeszentrale für politische Bildung
@@ -144,6 +154,7 @@
- **CBS**
- **CBSInteractive**
- **CBSLocal**
+ - **CBSLocalArticle**
- **cbsnews**: CBS News
- **cbsnews:embed**
- **cbsnews:livevideo**: CBS News Live Videos
@@ -193,9 +204,9 @@
- **CrooksAndLiars**
- **crunchyroll**
- **crunchyroll:playlist**
- - **CSNNE**
- **CSpan**: C-SPAN
- **CtsNews**: 華視新聞
+ - **CTV**
- **CTVNews**
- **cu.ntv.co.jp**: Nippon Television Network
- **Culturebox**
@@ -271,7 +282,6 @@
- **ESPNArticle**
- **EsriVideo**
- **Europa**
- - **EveryonesMixtape**
- **EWETV**
- **ExpoTV**
- **Expressen**
@@ -313,11 +323,11 @@
- **FrontendMasters**
- **FrontendMastersCourse**
- **FrontendMastersLesson**
+ - **FujiTVFODPlus7**
- **Funimation**
- **Funk**
- **Fusion**
- **Fux**
- - **FXNetworks**
- **Gaia**
- **GameInformer**
- **GameSpot**
@@ -336,6 +346,8 @@
- **Go**
- **GodTube**
- **Golem**
+ - **google:podcasts**
+ - **google:podcasts:feed**
- **GoogleDrive**
- **Goshgay**
- **GPUTechConf**
@@ -350,6 +362,7 @@
- **hgtv.com:show**
- **HiDive**
- **HistoricFilms**
+ - **history:player**
- **history:topic**: History.com Topic
- **hitbox**
- **hitbox:live**
@@ -370,6 +383,8 @@
- **HungamaSong**
- **Hypem**
- **ign.com**
+ - **IHeartRadio**
+ - **iheartradio:podcast**
- **imdb**: Internet Movie Database trailers
- **imdb:list**: Internet Movie Database lists
- **Imgur**
@@ -403,14 +418,14 @@
- **JWPlatform**
- **Kakao**
- **Kaltura**
- - **KanalPlay**: Kanal 5/9/11 Play
- **Kankan**
- **Karaoketv**
- **KarriereVideos**
- **Katsomo**
- **KeezMovies**
- **Ketnet**
- - **KhanAcademy**
+ - **khanacademy**
+ - **khanacademy:unit**
- **KickStarter**
- **KinjaEmbed**
- **KinoPoisk**
@@ -427,7 +442,8 @@
- **la7.it**
- **laola1tv**
- **laola1tv:embed**
- - **lbry.tv**
+ - **lbry**
+ - **lbry:channel**
- **LCI**
- **Lcp**
- **LcpPlay**
@@ -493,6 +509,7 @@
- **META**
- **metacafe**
- **Metacritic**
+ - **mewatch**
- **Mgoon**
- **MGTV**: 芒果TV
- **MiaoPai**
@@ -503,8 +520,6 @@
- **mixcloud**
- **mixcloud:playlist**
- **mixcloud:user**
- - **Mixer:live**
- - **Mixer:vod**
- **MLB**
- **Mnet**
- **MNetTV**
@@ -547,6 +562,11 @@
- **Naver**
- **Naver:live**
- **NBA**
+ - **nba:watch**
+ - **nba:watch:collection**
+ - **NBAChannel**
+ - **NBAEmbed**
+ - **NBAWatchEmbed**
- **NBC**
- **NBCNews**
- **nbcolympics**
@@ -576,8 +596,10 @@
- **NextTV**: 壹電視
- **Nexx**
- **NexxEmbed**
- - **nfl.com**
+ - **nfl.com** (Currently broken)
+ - **nfl.com:article** (Currently broken)
- **NhkVod**
+ - **NhkVodProgram**
- **nhl.com**
- **nick.com**
- **nick.de**
@@ -592,7 +614,6 @@
- **njoy:embed**
- **NJPWWorld**: 新日本プロレスワールド
- **NobelPrize**
- - **Noco**
- **NonkTube**
- **Noovo**
- **Normalboots**
@@ -610,6 +631,7 @@
- **Npr**
- **NRK**
- **NRKPlaylist**
+ - **NRKRadioPodkast**
- **NRKSkole**: NRK Skole
- **NRKTV**: NRK TV and NRK Radio
- **NRKTVDirekte**: NRK TV Direkte and NRK Radio Direkte
@@ -688,7 +710,6 @@
- **Playwire**
- **pluralsight**
- **pluralsight:course**
- - **plus.google**: Google Plus
- **podomatic**
- **Pokemon**
- **PokemonWatch**
@@ -719,6 +740,7 @@
- **qqmusic:singer**: QQ音乐 - 歌手
- **qqmusic:toplist**: QQ音乐 - 排行榜
- **QuantumTV**
+ - **Qub**
- **Quickline**
- **QuicklineLive**
- **R7**
@@ -811,18 +833,17 @@
- **Shared**: shared.sx
- **ShowRoomLive**
- **Sina**
+ - **sky.it**
+ - **sky:news**
+ - **sky:sports**
+ - **sky:sports:news**
+ - **skyacademy.it**
- **SkylineWebcams**
- - **SkyNews**
- **skynewsarabia:article**
- **skynewsarabia:video**
- - **SkySports**
- **Slideshare**
- **SlidesLive**
- **Slutload**
- - **smotri**: Smotri.com
- - **smotri:broadcast**: Smotri.com broadcasts
- - **smotri:community**: Smotri.com community videos
- - **smotri:user**: Smotri.com user videos
- **Snotr**
- **Sohu**
- **SonyLIV**
@@ -883,7 +904,6 @@
- **Tagesschau**
- **tagesschau:player**
- **Tass**
- - **TastyTrade**
- **TBS**
- **TDSLifeway**
- **Teachable**
@@ -906,6 +926,7 @@
- **TeleQuebecEmission**
- **TeleQuebecLive**
- **TeleQuebecSquat**
+ - **TeleQuebecVideo**
- **TeleTask**
- **Telewebion**
- **TennisTV**
@@ -923,6 +944,7 @@
- **ThisAV**
- **ThisOldHouse**
- **TikTok**
+ - **TikTokUser** (Currently broken)
- **tinypic**: tinypic.com videos
- **TMZ**
- **TMZArticle**
@@ -955,12 +977,15 @@
- **TV2DKBornholmPlay**
- **TV4**: tv4.se and tv4play.se
- **TV5MondePlus**: TV5MONDE+
+ - **tv5unis**
+ - **tv5unis:video**
- **tv8.it**
- **TVA**
- **TVANouvelles**
- **TVANouvellesArticle**
- **TVC**
- **TVCArticle**
+ - **TVer**
- **tvigle**: Интернет-телевидение Tvigle.ru
- **tvland.com**
- **TVN24**
@@ -1089,6 +1114,7 @@
- **vube**: Vube.com
- **VuClip**
- **VVVVID**
+ - **VVVVIDShow**
- **VyboryMos**
- **Vzaar**
- **Wakanim**
@@ -1111,6 +1137,7 @@
- **WeiboMobile**
- **WeiqiTV**: WQTV
- **Wistia**
+ - **WistiaPlaylist**
- **wnl**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
- **WorldStarHipHop**
- **WSJ**: Wall Street Journal
@@ -1118,7 +1145,7 @@
- **WWE**
- **XBef**
- **XboxClips**
- - **XFileShare**: XFileShare based sites: ClipWatching, GoUnlimited, GoVid, HolaVid, Streamty, TheVideoBee, Uqload, VidBom, vidlo, VidLocker, VidShare, VUp, XVideoSharing
+ - **XFileShare**: XFileShare based sites: Aparat, ClipWatching, GoUnlimited, GoVid, HolaVid, Streamty, TheVideoBee, Uqload, VidBom, vidlo, VidLocker, VidShare, VUp, XVideoSharing
- **XHamster**
- **XHamsterEmbed**
- **XHamsterUser**
@@ -1142,6 +1169,8 @@
- **yahoo:japannews**: Yahoo! Japan News
- **YandexDisk**
- **yandexmusic:album**: Яндекс.Музыка - Альбом
+ - **yandexmusic:artist:albums**: Яндекс.Музыка - Артист - Альбомы
+ - **yandexmusic:artist:tracks**: Яндекс.Музыка - Артист - Треки
- **yandexmusic:playlist**: Яндекс.Музыка - Плейлист
- **yandexmusic:track**: Яндекс.Музыка - Трек
- **YandexVideo**
@@ -1169,9 +1198,9 @@
- **youtube:subscriptions**: YouTube.com subscriptions feed, ":ytsubs" for short (requires authentication)
- **youtube:tab**: YouTube.com tab
- **youtube:watchlater**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)
+ - **YoutubeYtBe**: youtu.be
- **YoutubeYtUser**: YouTube.com user videos, URL or "ytuser" keyword
- **Zapiks**
- - **Zaq1**
- **Zattoo**
- **ZattooLive**
- **ZDF-3sat**
diff --git a/test/test_InfoExtractor.py b/test/test_InfoExtractor.py
index bdd01e41a..22e3d26a7 100644
--- a/test/test_InfoExtractor.py
+++ b/test/test_InfoExtractor.py
@@ -98,6 +98,55 @@ def test_html_search_meta(self):
self.assertRaises(RegexNotFoundError, ie._html_search_meta, 'z', html, None, fatal=True)
self.assertRaises(RegexNotFoundError, ie._html_search_meta, ('z', 'x'), html, None, fatal=True)
+ def test_search_json_ld_realworld(self):
+ # https://github.com/ytdl-org/youtube-dl/issues/23306
+ expect_dict(
+ self,
+ self.ie._search_json_ld(r'''''', None),
+ {
+ 'title': '1 On 1 With Kleio',
+ 'description': 'Kleio Valentien',
+ 'url': 'https://gvideo.eporner.com/xN49A1cT3eB/xN49A1cT3eB.mp4',
+ 'timestamp': 1449347075,
+ 'duration': 743.0,
+ 'view_count': 1120958,
+ 'width': 1920,
+ 'height': 1080,
+ })
+
def test_download_json(self):
uri = encode_data_uri(b'{"foo": "blah"}', 'application/json')
self.assertEqual(self.ie._download_json(uri, None), {'foo': 'blah'})
@@ -108,6 +157,18 @@ def test_download_json(self):
self.assertEqual(self.ie._download_json(uri, None, fatal=False), None)
def test_parse_html5_media_entries(self):
+ # inline video tag
+ expect_dict(
+ self,
+ self.ie._parse_html5_media_entries(
+ 'https://127.0.0.1/video.html',
+ r'', None)[0],
+ {
+ 'formats': [{
+ 'url': 'https://127.0.0.1/vid.mp4',
+ }],
+ })
+
# from https://www.r18.com/
# with kpbs in label
expect_dict(
diff --git a/test/test_all_urls.py b/test/test_all_urls.py
index 8dcdc4e58..130038c0d 100644
--- a/test/test_all_urls.py
+++ b/test/test_all_urls.py
@@ -39,7 +39,7 @@ def test_youtube_playlist_matching(self):
assertTab('https://www.youtube.com/embedded')
assertTab('https://www.youtube.com/feed') # Own channel's home page
assertTab('https://www.youtube.com/playlist?list=UUBABnxM4Ar9ten8Mdjj1j0Q')
- assertPlaylist('https://www.youtube.com/course?list=ECUl4u3cNGP61MdtwGTqZA0MreSaDybji8')
+ assertTab('https://www.youtube.com/course?list=ECUl4u3cNGP61MdtwGTqZA0MreSaDybji8')
assertTab('https://www.youtube.com/playlist?list=PLwP_SiAcdui0KVebT0mU9Apz359a4ubsC')
assertTab('https://www.youtube.com/watch?v=AV6J6_AeFEQ&playnext=1&list=PL4023E734DA416012') # 668
self.assertFalse('youtube:playlist' in self.matching_ies('PLtS2H6bU1M'))
@@ -60,8 +60,8 @@ def test_youtube_channel_matching(self):
assertChannel('https://www.youtube.com/channel/HCtnHdj3df7iM?feature=gb_ch_rec')
assertChannel('https://www.youtube.com/channel/HCtnHdj3df7iM/videos')
- # def test_youtube_user_matching(self):
- # self.assertMatch('http://www.youtube.com/NASAgovVideo/videos', ['youtube:tab'])
+ def test_youtube_user_matching(self):
+ self.assertMatch('http://www.youtube.com/NASAgovVideo/videos', ['youtube:tab'])
def test_youtube_feeds(self):
self.assertMatch('https://www.youtube.com/feed/library', ['youtube:tab'])
diff --git a/test/test_subtitles.py b/test/test_subtitles.py
index 86e20cb4b..0014d57b6 100644
--- a/test/test_subtitles.py
+++ b/test/test_subtitles.py
@@ -264,16 +264,24 @@ def test_allsubtitles(self):
class TestRaiPlaySubtitles(BaseTestSubtitles):
- url = 'http://www.raiplay.it/video/2014/04/Report-del-07042014-cb27157f-9dd0-4aee-b788-b1f67643a391.html'
IE = RaiPlayIE
- def test_allsubtitles(self):
+ def test_subtitles_key(self):
+ self.url = 'http://www.raiplay.it/video/2014/04/Report-del-07042014-cb27157f-9dd0-4aee-b788-b1f67643a391.html'
self.DL.params['writesubtitles'] = True
self.DL.params['allsubtitles'] = True
subtitles = self.getSubtitles()
self.assertEqual(set(subtitles.keys()), set(['it']))
self.assertEqual(md5(subtitles['it']), 'b1d90a98755126b61e667567a1f6680a')
+ def test_subtitles_array_key(self):
+ self.url = 'https://www.raiplay.it/video/2020/12/Report---04-01-2021-2e90f1de-8eee-4de4-ac0e-78d21db5b600.html'
+ self.DL.params['writesubtitles'] = True
+ self.DL.params['allsubtitles'] = True
+ subtitles = self.getSubtitles()
+ self.assertEqual(set(subtitles.keys()), set(['it']))
+ self.assertEqual(md5(subtitles['it']), '4b3264186fbb103508abe5311cfcb9cd')
+
class TestVikiSubtitles(BaseTestSubtitles):
url = 'http://www.viki.com/videos/1060846v-punch-episode-18'
diff --git a/test/test_utils.py b/test/test_utils.py
index 16ad40831..f733b5d2b 100644
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -21,6 +21,7 @@
encode_base_n,
caesar,
clean_html,
+ clean_podcast_url,
date_from_str,
DateRange,
detect_exe_version,
@@ -554,6 +555,11 @@ def test_url_or_none(self):
self.assertEqual(url_or_none('http$://foo.de'), None)
self.assertEqual(url_or_none('http://foo.de'), 'http://foo.de')
self.assertEqual(url_or_none('//foo.de'), '//foo.de')
+ self.assertEqual(url_or_none('s3://foo.de'), None)
+ self.assertEqual(url_or_none('rtmpte://foo.de'), 'rtmpte://foo.de')
+ self.assertEqual(url_or_none('mms://foo.de'), 'mms://foo.de')
+ self.assertEqual(url_or_none('rtspu://foo.de'), 'rtspu://foo.de')
+ self.assertEqual(url_or_none('ftps://foo.de'), 'ftps://foo.de')
def test_parse_age_limit(self):
self.assertEqual(parse_age_limit(None), None)
@@ -1465,6 +1471,10 @@ def test_get_elements_by_attribute(self):
self.assertEqual(get_elements_by_attribute('class', 'foo', html), [])
self.assertEqual(get_elements_by_attribute('class', 'no-such-foo', html), [])
+ def test_clean_podcast_url(self):
+ self.assertEqual(clean_podcast_url('https://www.podtrac.com/pts/redirect.mp3/chtbl.com/track/5899E/traffic.megaphone.fm/HSW7835899191.mp3'), 'https://traffic.megaphone.fm/HSW7835899191.mp3')
+ self.assertEqual(clean_podcast_url('https://play.podtrac.com/npr-344098539/edge1.pod.npr.org/anon.npr-podcasts/podcast/npr/waitwait/2020/10/20201003_waitwait_wwdtmpodcast201003-015621a5-f035-4eca-a9a1-7c118d90bc3c.mp3'), 'https://edge1.pod.npr.org/anon.npr-podcasts/podcast/npr/waitwait/2020/10/20201003_waitwait_wwdtmpodcast201003-015621a5-f035-4eca-a9a1-7c118d90bc3c.mp3')
+
if __name__ == '__main__':
unittest.main()
diff --git a/youtube_dlc/YoutubeDL.py b/youtube_dlc/YoutubeDL.py
index ef6fe0a78..b1217b5c8 100644
--- a/youtube_dlc/YoutubeDL.py
+++ b/youtube_dlc/YoutubeDL.py
@@ -342,6 +342,8 @@ class YoutubeDL(object):
_pps = []
_download_retcode = None
_num_downloads = None
+ _playlist_level = 0
+ _playlist_urls = set()
_screen_file = None
def __init__(self, params=None, auto_init=True):
@@ -957,119 +959,23 @@ def process_ie_result(self, ie_result, download=True, extra_info={}):
return self.process_ie_result(
new_result, download=download, extra_info=extra_info)
elif result_type in ('playlist', 'multi_video'):
- # We process each entry in the playlist
- playlist = ie_result.get('title') or ie_result.get('id')
- self.to_screen('[download] Downloading playlist: %s' % playlist)
-
- playlist_results = []
-
- playliststart = self.params.get('playliststart', 1) - 1
- playlistend = self.params.get('playlistend')
- # For backwards compatibility, interpret -1 as whole list
- if playlistend == -1:
- playlistend = None
-
- playlistitems_str = self.params.get('playlist_items')
- playlistitems = None
- if playlistitems_str is not None:
- def iter_playlistitems(format):
- for string_segment in format.split(','):
- if '-' in string_segment:
- start, end = string_segment.split('-')
- for item in range(int(start), int(end) + 1):
- yield int(item)
- else:
- yield int(string_segment)
- playlistitems = orderedSet(iter_playlistitems(playlistitems_str))
-
- ie_entries = ie_result['entries']
-
- def make_playlistitems_entries(list_ie_entries):
- num_entries = len(list_ie_entries)
- return [
- list_ie_entries[i - 1] for i in playlistitems
- if -num_entries <= i - 1 < num_entries]
-
- def report_download(num_entries):
+ # Protect from infinite recursion due to recursively nested playlists
+ # (see https://github.com/ytdl-org/youtube-dl/issues/27833)
+ webpage_url = ie_result['webpage_url']
+ if webpage_url in self._playlist_urls:
self.to_screen(
- '[%s] playlist %s: Downloading %d videos' %
- (ie_result['extractor'], playlist, num_entries))
+ '[download] Skipping already downloaded playlist: %s'
+ % ie_result.get('title') or ie_result.get('id'))
+ return
- if isinstance(ie_entries, list):
- n_all_entries = len(ie_entries)
- if playlistitems:
- entries = make_playlistitems_entries(ie_entries)
- else:
- entries = ie_entries[playliststart:playlistend]
- n_entries = len(entries)
- self.to_screen(
- '[%s] playlist %s: Collected %d video ids (downloading %d of them)' %
- (ie_result['extractor'], playlist, n_all_entries, n_entries))
- elif isinstance(ie_entries, PagedList):
- if playlistitems:
- entries = []
- for item in playlistitems:
- entries.extend(ie_entries.getslice(
- item - 1, item
- ))
- else:
- entries = ie_entries.getslice(
- playliststart, playlistend)
- n_entries = len(entries)
- report_download(n_entries)
- else: # iterable
- if playlistitems:
- entries = make_playlistitems_entries(list(itertools.islice(
- ie_entries, 0, max(playlistitems))))
- else:
- entries = list(itertools.islice(
- ie_entries, playliststart, playlistend))
- n_entries = len(entries)
- report_download(n_entries)
-
- if self.params.get('playlistreverse', False):
- entries = entries[::-1]
-
- if self.params.get('playlistrandom', False):
- random.shuffle(entries)
-
- x_forwarded_for = ie_result.get('__x_forwarded_for_ip')
-
- for i, entry in enumerate(entries, 1):
- self.to_screen('[download] Downloading video %s of %s' % (i, n_entries))
- # This __x_forwarded_for_ip thing is a bit ugly but requires
- # minimal changes
- if x_forwarded_for:
- entry['__x_forwarded_for_ip'] = x_forwarded_for
- extra = {
- 'n_entries': n_entries,
- 'playlist': playlist,
- 'playlist_id': ie_result.get('id'),
- 'playlist_title': ie_result.get('title'),
- 'playlist_uploader': ie_result.get('uploader'),
- 'playlist_uploader_id': ie_result.get('uploader_id'),
- 'playlist_index': playlistitems[i - 1] if playlistitems else i + playliststart,
- 'extractor': ie_result['extractor'],
- 'webpage_url': ie_result['webpage_url'],
- 'webpage_url_basename': url_basename(ie_result['webpage_url']),
- 'extractor_key': ie_result['extractor_key'],
- }
-
- reason = self._match_entry(entry, incomplete=True)
- if reason is not None:
- if reason.endswith('has already been recorded in the archive') and self.params.get('break_on_existing'):
- print('[download] tried downloading a file that\'s already in the archive, stopping since --break-on-existing is set.')
- break
- else:
- self.to_screen('[download] ' + reason)
- continue
-
- entry_result = self.__process_iterable_entry(entry, download, extra)
- # TODO: skip failed (empty) entries?
- playlist_results.append(entry_result)
- ie_result['entries'] = playlist_results
- self.to_screen('[download] Finished downloading playlist: %s' % playlist)
- return ie_result
+ self._playlist_level += 1
+ self._playlist_urls.add(webpage_url)
+ try:
+ return self.__process_playlist(ie_result, download)
+ finally:
+ self._playlist_level -= 1
+ if not self._playlist_level:
+ self._playlist_urls.clear()
elif result_type == 'compat_list':
self.report_warning(
'Extractor %s returned a compat_list result. '
@@ -1094,6 +1000,122 @@ def _fixup(r):
else:
raise Exception('Invalid result type: %s' % result_type)
+ def __process_playlist(self, ie_result, download):
+ # We process each entry in the playlist
+ playlist = ie_result.get('title') or ie_result.get('id')
+
+ self.to_screen('[download] Downloading playlist: %s' % playlist)
+
+ playlist_results = []
+
+ playliststart = self.params.get('playliststart', 1) - 1
+ playlistend = self.params.get('playlistend')
+ # For backwards compatibility, interpret -1 as whole list
+ if playlistend == -1:
+ playlistend = None
+
+ playlistitems_str = self.params.get('playlist_items')
+ playlistitems = None
+ if playlistitems_str is not None:
+ def iter_playlistitems(format):
+ for string_segment in format.split(','):
+ if '-' in string_segment:
+ start, end = string_segment.split('-')
+ for item in range(int(start), int(end) + 1):
+ yield int(item)
+ else:
+ yield int(string_segment)
+ playlistitems = orderedSet(iter_playlistitems(playlistitems_str))
+
+ ie_entries = ie_result['entries']
+
+ def make_playlistitems_entries(list_ie_entries):
+ num_entries = len(list_ie_entries)
+ return [
+ list_ie_entries[i - 1] for i in playlistitems
+ if -num_entries <= i - 1 < num_entries]
+
+ def report_download(num_entries):
+ self.to_screen(
+ '[%s] playlist %s: Downloading %d videos' %
+ (ie_result['extractor'], playlist, num_entries))
+
+ if isinstance(ie_entries, list):
+ n_all_entries = len(ie_entries)
+ if playlistitems:
+ entries = make_playlistitems_entries(ie_entries)
+ else:
+ entries = ie_entries[playliststart:playlistend]
+ n_entries = len(entries)
+ self.to_screen(
+ '[%s] playlist %s: Collected %d video ids (downloading %d of them)' %
+ (ie_result['extractor'], playlist, n_all_entries, n_entries))
+ elif isinstance(ie_entries, PagedList):
+ if playlistitems:
+ entries = []
+ for item in playlistitems:
+ entries.extend(ie_entries.getslice(
+ item - 1, item
+ ))
+ else:
+ entries = ie_entries.getslice(
+ playliststart, playlistend)
+ n_entries = len(entries)
+ report_download(n_entries)
+ else: # iterable
+ if playlistitems:
+ entries = make_playlistitems_entries(list(itertools.islice(
+ ie_entries, 0, max(playlistitems))))
+ else:
+ entries = list(itertools.islice(
+ ie_entries, playliststart, playlistend))
+ n_entries = len(entries)
+ report_download(n_entries)
+
+ if self.params.get('playlistreverse', False):
+ entries = entries[::-1]
+
+ if self.params.get('playlistrandom', False):
+ random.shuffle(entries)
+
+ x_forwarded_for = ie_result.get('__x_forwarded_for_ip')
+
+ for i, entry in enumerate(entries, 1):
+ self.to_screen('[download] Downloading video %s of %s' % (i, n_entries))
+ # This __x_forwarded_for_ip thing is a bit ugly but requires
+ # minimal changes
+ if x_forwarded_for:
+ entry['__x_forwarded_for_ip'] = x_forwarded_for
+ extra = {
+ 'n_entries': n_entries,
+ 'playlist': playlist,
+ 'playlist_id': ie_result.get('id'),
+ 'playlist_title': ie_result.get('title'),
+ 'playlist_uploader': ie_result.get('uploader'),
+ 'playlist_uploader_id': ie_result.get('uploader_id'),
+ 'playlist_index': playlistitems[i - 1] if playlistitems else i + playliststart,
+ 'extractor': ie_result['extractor'],
+ 'webpage_url': ie_result['webpage_url'],
+ 'webpage_url_basename': url_basename(ie_result['webpage_url']),
+ 'extractor_key': ie_result['extractor_key'],
+ }
+
+ reason = self._match_entry(entry, incomplete=True)
+ if reason is not None:
+ if reason.endswith('has already been recorded in the archive') and self.params.get('break_on_existing'):
+ print('[download] tried downloading a file that\'s already in the archive, stopping since --break-on-existing is set.')
+ break
+ else:
+ self.to_screen('[download] ' + reason)
+ continue
+
+ entry_result = self.__process_iterable_entry(entry, download, extra)
+ # TODO: skip failed (empty) entries?
+ playlist_results.append(entry_result)
+ ie_result['entries'] = playlist_results
+ self.to_screen('[download] Finished downloading playlist: %s' % playlist)
+ return ie_result
+
@__handle_extraction_exceptions
def __process_iterable_entry(self, entry, download, extra_info):
return self.process_ie_result(
@@ -1138,7 +1160,7 @@ def _build_format_filter(self, filter_spec):
'*=': lambda attr, value: value in attr,
}
str_operator_rex = re.compile(r'''(?x)
- \s*(?Pext|acodec|vcodec|container|protocol|format_id)
+ \s*(?Pext|acodec|vcodec|container|protocol|format_id|language)
\s*(?P!\s*)?(?P%s)(?P\s*\?)?
\s*(?P[a-zA-Z0-9._-]+)
\s*$
@@ -1679,7 +1701,7 @@ def is_wellformed(f):
if req_format is None:
req_format = self._default_format_spec(info_dict, download=download)
if self.params.get('verbose'):
- self.to_stdout('[debug] Default format spec: %s' % req_format)
+ self._write_string('[debug] Default format spec: %s\n' % req_format)
format_selector = self.build_format_selector(req_format)
@@ -1889,7 +1911,7 @@ def dl(name, info, subtitle=False):
for ph in self._progress_hooks:
fd.add_progress_hook(ph)
if self.params.get('verbose'):
- self.to_stdout('[debug] Invoking downloader on %r' % info.get('url'))
+ self.to_screen('[debug] Invoking downloader on %r' % info.get('url'))
return fd.download(name, info, subtitle)
subtitles_are_requested = any([self.params.get('writesubtitles', False),
@@ -2505,7 +2527,7 @@ def _write_thumbnails(self, info_dict, filename):
thumb_ext = determine_ext(t['url'], 'jpg')
suffix = '_%s' % t['id'] if len(thumbnails) > 1 else ''
thumb_display_id = '%s ' % t['id'] if len(thumbnails) > 1 else ''
- t['filename'] = thumb_filename = os.path.splitext(filename)[0] + suffix + '.' + thumb_ext
+ t['filename'] = thumb_filename = replace_extension(filename + suffix, thumb_ext, info_dict.get('ext'))
if self.params.get('nooverwrites', False) and os.path.exists(encodeFilename(thumb_filename)):
self.to_screen('[%s] %s: Thumbnail %sis already present' %
diff --git a/youtube_dlc/downloader/hls.py b/youtube_dlc/downloader/hls.py
index 0f2c06f40..7aaebc940 100644
--- a/youtube_dlc/downloader/hls.py
+++ b/youtube_dlc/downloader/hls.py
@@ -42,11 +42,13 @@ def can_download(manifest, info_dict):
# no segments will definitely be appended to the end of the playlist.
# r'#EXT-X-PLAYLIST-TYPE:EVENT', # media segments may be appended to the end of
# # event media playlists [4]
+ r'#EXT-X-MAP:', # media initialization [5]
# 1. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.4
# 2. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.2
# 3. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.2
# 4. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.5
+ # 5. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.5
)
check_results = [not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES]
is_aes128_enc = '#EXT-X-KEY:METHOD=AES-128' in manifest
@@ -170,8 +172,12 @@ def is_ad_fragment_end(s):
iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
- frag_content = AES.new(
- decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
+ # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
+ # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
+ # not what it decrypts to.
+ if not test:
+ frag_content = AES.new(
+ decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
self._append_fragment(ctx, frag_content)
# We only download the first fragment during the test
if test:
diff --git a/youtube_dlc/extractor/acast.py b/youtube_dlc/extractor/acast.py
index b17c792d2..b9355a2c8 100644
--- a/youtube_dlc/extractor/acast.py
+++ b/youtube_dlc/extractor/acast.py
@@ -2,21 +2,48 @@
from __future__ import unicode_literals
import re
-import functools
from .common import InfoExtractor
-from ..compat import compat_str
from ..utils import (
clean_html,
- float_or_none,
+ clean_podcast_url,
int_or_none,
- try_get,
- unified_timestamp,
- OnDemandPagedList,
+ parse_iso8601,
)
-class ACastIE(InfoExtractor):
+class ACastBaseIE(InfoExtractor):
+ def _extract_episode(self, episode, show_info):
+ title = episode['title']
+ info = {
+ 'id': episode['id'],
+ 'display_id': episode.get('episodeUrl'),
+ 'url': clean_podcast_url(episode['url']),
+ 'title': title,
+ 'description': clean_html(episode.get('description') or episode.get('summary')),
+ 'thumbnail': episode.get('image'),
+ 'timestamp': parse_iso8601(episode.get('publishDate')),
+ 'duration': int_or_none(episode.get('duration')),
+ 'filesize': int_or_none(episode.get('contentLength')),
+ 'season_number': int_or_none(episode.get('season')),
+ 'episode': title,
+ 'episode_number': int_or_none(episode.get('episode')),
+ }
+ info.update(show_info)
+ return info
+
+ def _extract_show_info(self, show):
+ return {
+ 'creator': show.get('author'),
+ 'series': show.get('title'),
+ }
+
+ def _call_api(self, path, video_id, query=None):
+ return self._download_json(
+ 'https://feeder.acast.com/api/v1/shows/' + path, video_id, query=query)
+
+
+class ACastIE(ACastBaseIE):
IE_NAME = 'acast'
_VALID_URL = r'''(?x)
https?://
@@ -28,15 +55,15 @@ class ACastIE(InfoExtractor):
'''
_TESTS = [{
'url': 'https://www.acast.com/sparpodcast/2.raggarmordet-rosterurdetforflutna',
- 'md5': '16d936099ec5ca2d5869e3a813ee8dc4',
+ 'md5': 'f5598f3ad1e4776fed12ec1407153e4b',
'info_dict': {
'id': '2a92b283-1a75-4ad8-8396-499c641de0d9',
'ext': 'mp3',
'title': '2. Raggarmordet - Röster ur det förflutna',
- 'description': 'md5:4f81f6d8cf2e12ee21a321d8bca32db4',
+ 'description': 'md5:a992ae67f4d98f1c0141598f7bebbf67',
'timestamp': 1477346700,
'upload_date': '20161024',
- 'duration': 2766.602563,
+ 'duration': 2766,
'creator': 'Anton Berg & Martin Johnson',
'series': 'Spår',
'episode': '2. Raggarmordet - Röster ur det förflutna',
@@ -45,7 +72,7 @@ class ACastIE(InfoExtractor):
'url': 'http://embed.acast.com/adambuxton/ep.12-adam-joeschristmaspodcast2015',
'only_matching': True,
}, {
- 'url': 'https://play.acast.com/s/rattegangspodden/s04e09-styckmordet-i-helenelund-del-22',
+ 'url': 'https://play.acast.com/s/rattegangspodden/s04e09styckmordetihelenelund-del2-2',
'only_matching': True,
}, {
'url': 'https://play.acast.com/s/sparpodcast/2a92b283-1a75-4ad8-8396-499c641de0d9',
@@ -54,40 +81,14 @@ class ACastIE(InfoExtractor):
def _real_extract(self, url):
channel, display_id = re.match(self._VALID_URL, url).groups()
- s = self._download_json(
- 'https://feeder.acast.com/api/v1/shows/%s/episodes/%s' % (channel, display_id),
- display_id)
- media_url = s['url']
- if re.search(r'[0-9a-f]{8}-(?:[0-9a-f]{4}-){3}[0-9a-f]{12}', display_id):
- episode_url = s.get('episodeUrl')
- if episode_url:
- display_id = episode_url
- else:
- channel, display_id = re.match(self._VALID_URL, s['link']).groups()
- cast_data = self._download_json(
- 'https://play-api.acast.com/splash/%s/%s' % (channel, display_id),
- display_id)['result']
- e = cast_data['episode']
- title = e.get('name') or s['title']
- return {
- 'id': compat_str(e['id']),
- 'display_id': display_id,
- 'url': media_url,
- 'title': title,
- 'description': e.get('summary') or clean_html(e.get('description') or s.get('description')),
- 'thumbnail': e.get('image'),
- 'timestamp': unified_timestamp(e.get('publishingDate') or s.get('publishDate')),
- 'duration': float_or_none(e.get('duration') or s.get('duration')),
- 'filesize': int_or_none(e.get('contentLength')),
- 'creator': try_get(cast_data, lambda x: x['show']['author'], compat_str),
- 'series': try_get(cast_data, lambda x: x['show']['name'], compat_str),
- 'season_number': int_or_none(e.get('seasonNumber')),
- 'episode': title,
- 'episode_number': int_or_none(e.get('episodeNumber')),
- }
+ episode = self._call_api(
+ '%s/episodes/%s' % (channel, display_id),
+ display_id, {'showInfo': 'true'})
+ return self._extract_episode(
+ episode, self._extract_show_info(episode.get('show') or {}))
-class ACastChannelIE(InfoExtractor):
+class ACastChannelIE(ACastBaseIE):
IE_NAME = 'acast:channel'
_VALID_URL = r'''(?x)
https?://
@@ -102,34 +103,24 @@ class ACastChannelIE(InfoExtractor):
'info_dict': {
'id': '4efc5294-5385-4847-98bd-519799ce5786',
'title': 'Today in Focus',
- 'description': 'md5:9ba5564de5ce897faeb12963f4537a64',
+ 'description': 'md5:c09ce28c91002ce4ffce71d6504abaae',
},
- 'playlist_mincount': 35,
+ 'playlist_mincount': 200,
}, {
'url': 'http://play.acast.com/s/ft-banking-weekly',
'only_matching': True,
}]
- _API_BASE_URL = 'https://play.acast.com/api/'
- _PAGE_SIZE = 10
@classmethod
def suitable(cls, url):
return False if ACastIE.suitable(url) else super(ACastChannelIE, cls).suitable(url)
- def _fetch_page(self, channel_slug, page):
- casts = self._download_json(
- self._API_BASE_URL + 'channels/%s/acasts?page=%s' % (channel_slug, page),
- channel_slug, note='Download page %d of channel data' % page)
- for cast in casts:
- yield self.url_result(
- 'https://play.acast.com/s/%s/%s' % (channel_slug, cast['url']),
- 'ACast', cast['id'])
-
def _real_extract(self, url):
- channel_slug = self._match_id(url)
- channel_data = self._download_json(
- self._API_BASE_URL + 'channels/%s' % channel_slug, channel_slug)
- entries = OnDemandPagedList(functools.partial(
- self._fetch_page, channel_slug), self._PAGE_SIZE)
- return self.playlist_result(entries, compat_str(
- channel_data['id']), channel_data['name'], channel_data.get('description'))
+ show_slug = self._match_id(url)
+ show = self._call_api(show_slug, show_slug)
+ show_info = self._extract_show_info(show)
+ entries = []
+ for episode in (show.get('episodes') or []):
+ entries.append(self._extract_episode(episode, show_info))
+ return self.playlist_result(
+ entries, show.get('id'), show.get('title'), show.get('description'))
diff --git a/youtube_dlc/extractor/adn.py b/youtube_dlc/extractor/adn.py
index c95ad2173..d611ee237 100644
--- a/youtube_dlc/extractor/adn.py
+++ b/youtube_dlc/extractor/adn.py
@@ -10,6 +10,7 @@
from .common import InfoExtractor
from ..aes import aes_cbc_decrypt
from ..compat import (
+ compat_HTTPError,
compat_b64decode,
compat_ord,
)
@@ -18,11 +19,13 @@
bytes_to_long,
ExtractorError,
float_or_none,
+ int_or_none,
intlist_to_bytes,
long_to_bytes,
pkcs1pad,
strip_or_none,
- urljoin,
+ try_get,
+ unified_strdate,
)
@@ -31,16 +34,27 @@ class ADNIE(InfoExtractor):
_VALID_URL = r'https?://(?:www\.)?animedigitalnetwork\.fr/video/[^/]+/(?P\d+)'
_TEST = {
'url': 'http://animedigitalnetwork.fr/video/blue-exorcist-kyoto-saga/7778-episode-1-debut-des-hostilites',
- 'md5': 'e497370d847fd79d9d4c74be55575c7a',
+ 'md5': '0319c99885ff5547565cacb4f3f9348d',
'info_dict': {
'id': '7778',
'ext': 'mp4',
- 'title': 'Blue Exorcist - Kyôto Saga - Épisode 1',
+ 'title': 'Blue Exorcist - Kyôto Saga - Episode 1',
'description': 'md5:2f7b5aa76edbc1a7a92cedcda8a528d5',
+ 'series': 'Blue Exorcist - Kyôto Saga',
+ 'duration': 1467,
+ 'release_date': '20170106',
+ 'comment_count': int,
+ 'average_rating': float,
+ 'season_number': 2,
+ 'episode': 'Début des hostilités',
+ 'episode_number': 1,
}
}
+
_BASE_URL = 'http://animedigitalnetwork.fr'
- _RSA_KEY = (0xc35ae1e4356b65a73b551493da94b8cb443491c0aa092a357a5aee57ffc14dda85326f42d716e539a34542a0d3f363adf16c5ec222d713d5997194030ee2e4f0d1fb328c01a81cf6868c090d50de8e169c6b13d1675b9eeed1cbc51e1fffca9b38af07f37abd790924cd3bee59d0257cfda4fe5f3f0534877e21ce5821447d1b, 65537)
+ _API_BASE_URL = 'https://gw.api.animedigitalnetwork.fr/'
+ _PLAYER_BASE_URL = _API_BASE_URL + 'player/'
+ _RSA_KEY = (0x9B42B08905199A5CCE2026274399CA560ECB209EE9878A708B1C0812E1BB8CB5D1FB7441861147C1A1F2F3A0476DD63A9CAC20D3E983613346850AA6CB38F16DC7D720FD7D86FC6E5B3D5BBC72E14CD0BF9E869F2CEA2CCAD648F1DCE38F1FF916CEFB2D339B64AA0264372344BC775E265E8A852F88144AB0BD9AA06C1A4ABB, 65537)
_POS_ALIGN_MAP = {
'start': 1,
'end': 3,
@@ -54,26 +68,24 @@ class ADNIE(InfoExtractor):
def _ass_subtitles_timecode(seconds):
return '%01d:%02d:%02d.%02d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 100)
- def _get_subtitles(self, sub_path, video_id):
- if not sub_path:
+ def _get_subtitles(self, sub_url, video_id):
+ if not sub_url:
return None
enc_subtitles = self._download_webpage(
- urljoin(self._BASE_URL, sub_path),
- video_id, 'Downloading subtitles location', fatal=False) or '{}'
+ sub_url, video_id, 'Downloading subtitles location', fatal=False) or '{}'
subtitle_location = (self._parse_json(enc_subtitles, video_id, fatal=False) or {}).get('location')
if subtitle_location:
enc_subtitles = self._download_webpage(
- urljoin(self._BASE_URL, subtitle_location),
- video_id, 'Downloading subtitles data', fatal=False,
- headers={'Origin': 'https://animedigitalnetwork.fr'})
+ subtitle_location, video_id, 'Downloading subtitles data',
+ fatal=False, headers={'Origin': 'https://animedigitalnetwork.fr'})
if not enc_subtitles:
return None
# http://animedigitalnetwork.fr/components/com_vodvideo/videojs/adn-vjs.min.js
dec_subtitles = intlist_to_bytes(aes_cbc_decrypt(
bytes_to_intlist(compat_b64decode(enc_subtitles[24:])),
- bytes_to_intlist(binascii.unhexlify(self._K + '4b8ef13ec1872730')),
+ bytes_to_intlist(binascii.unhexlify(self._K + 'ab9f52f5baae7c72')),
bytes_to_intlist(compat_b64decode(enc_subtitles[:24]))
))
subtitles_json = self._parse_json(
@@ -119,59 +131,76 @@ def _get_subtitles(self, sub_path, video_id):
def _real_extract(self, url):
video_id = self._match_id(url)
- webpage = self._download_webpage(url, video_id)
- player_config = self._parse_json(self._search_regex(
- r'playerConfig\s*=\s*({.+});', webpage,
- 'player config', default='{}'), video_id, fatal=False)
- if not player_config:
- config_url = urljoin(self._BASE_URL, self._search_regex(
- r'(?:id="player"|class="[^"]*adn-player-container[^"]*")[^>]+data-url="([^"]+)"',
- webpage, 'config url'))
- player_config = self._download_json(
- config_url, video_id,
- 'Downloading player config JSON metadata')['player']
-
- video_info = {}
- video_info_str = self._search_regex(
- r'videoInfo\s*=\s*({.+});', webpage,
- 'video info', fatal=False)
- if video_info_str:
- video_info = self._parse_json(
- video_info_str, video_id, fatal=False) or {}
-
- options = player_config.get('options') or {}
- metas = options.get('metas') or {}
- links = player_config.get('links') or {}
- sub_path = player_config.get('subtitles')
- error = None
- if not links:
- links_url = player_config.get('linksurl') or options['videoUrl']
- token = options['token']
- self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
- message = bytes_to_intlist(json.dumps({
- 'k': self._K,
- 'e': 60,
- 't': token,
- }))
+ video_base_url = self._PLAYER_BASE_URL + 'video/%s/' % video_id
+ player = self._download_json(
+ video_base_url + 'configuration', video_id,
+ 'Downloading player config JSON metadata')['player']
+ options = player['options']
+
+ user = options['user']
+ if not user.get('hasAccess'):
+ raise ExtractorError(
+ 'This video is only available for paying users', expected=True)
+ # self.raise_login_required() # FIXME: Login is not implemented
+
+ token = self._download_json(
+ user.get('refreshTokenUrl') or (self._PLAYER_BASE_URL + 'refresh/token'),
+ video_id, 'Downloading access token', headers={
+ 'x-player-refresh-token': user['refreshToken']
+ }, data=b'')['token']
+
+ links_url = try_get(options, lambda x: x['video']['url']) or (video_base_url + 'link')
+ self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
+ message = bytes_to_intlist(json.dumps({
+ 'k': self._K,
+ 't': token,
+ }))
+
+ # Sometimes authentication fails for no good reason, retry with
+ # a different random padding
+ links_data = None
+ for _ in range(3):
padded_message = intlist_to_bytes(pkcs1pad(message, 128))
n, e = self._RSA_KEY
encrypted_message = long_to_bytes(pow(bytes_to_long(padded_message), e, n))
authorization = base64.b64encode(encrypted_message).decode()
- links_data = self._download_json(
- urljoin(self._BASE_URL, links_url), video_id,
- 'Downloading links JSON metadata', headers={
- 'Authorization': 'Bearer ' + authorization,
- })
- links = links_data.get('links') or {}
- metas = metas or links_data.get('meta') or {}
- sub_path = sub_path or links_data.get('subtitles') or \
- 'index.php?option=com_vodapi&task=subtitles.getJSON&format=json&id=' + video_id
- sub_path += '&token=' + token
- error = links_data.get('error')
- title = metas.get('title') or video_info['title']
+
+ try:
+ links_data = self._download_json(
+ links_url, video_id, 'Downloading links JSON metadata', headers={
+ 'X-Player-Token': authorization
+ }, query={
+ 'freeWithAds': 'true',
+ 'adaptive': 'false',
+ 'withMetadata': 'true',
+ 'source': 'Web'
+ })
+ break
+ except ExtractorError as e:
+ if not isinstance(e.cause, compat_HTTPError):
+ raise e
+
+ if e.cause.code == 401:
+ # This usually goes away with a different random pkcs1pad, so retry
+ continue
+
+ error = self._parse_json(e.cause.read(), video_id)
+ message = error.get('message')
+ if e.cause.code == 403 and error.get('code') == 'player-bad-geolocation-country':
+ self.raise_geo_restricted(msg=message)
+ else:
+ raise ExtractorError(message)
+ else:
+ raise ExtractorError('Giving up retrying')
+
+ links = links_data.get('links') or {}
+ metas = links_data.get('metadata') or {}
+ sub_url = (links.get('subtitles') or {}).get('all')
+ video_info = links_data.get('video') or {}
+ title = metas['title']
formats = []
- for format_id, qualities in links.items():
+ for format_id, qualities in (links.get('streaming') or {}).items():
if not isinstance(qualities, dict):
continue
for quality, load_balancer_url in qualities.items():
@@ -189,19 +218,26 @@ def _real_extract(self, url):
for f in m3u8_formats:
f['language'] = 'fr'
formats.extend(m3u8_formats)
- if not error:
- error = options.get('error')
- if not formats and error:
- raise ExtractorError('%s said: %s' % (self.IE_NAME, error), expected=True)
self._sort_formats(formats)
+ video = (self._download_json(
+ self._API_BASE_URL + 'video/%s' % video_id, video_id,
+ 'Downloading additional video metadata', fatal=False) or {}).get('video') or {}
+ show = video.get('show') or {}
+
return {
'id': video_id,
'title': title,
- 'description': strip_or_none(metas.get('summary') or video_info.get('resume')),
- 'thumbnail': video_info.get('image'),
+ 'description': strip_or_none(metas.get('summary') or video.get('summary')),
+ 'thumbnail': video_info.get('image') or player.get('image'),
'formats': formats,
- 'subtitles': self.extract_subtitles(sub_path, video_id),
- 'episode': metas.get('subtitle') or video_info.get('videoTitle'),
- 'series': video_info.get('playlistTitle'),
+ 'subtitles': self.extract_subtitles(sub_url, video_id),
+ 'episode': metas.get('subtitle') or video.get('name'),
+ 'episode_number': int_or_none(video.get('shortNumber')),
+ 'series': show.get('title'),
+ 'season_number': int_or_none(video.get('season')),
+ 'duration': int_or_none(video_info.get('duration') or video.get('duration')),
+ 'release_date': unified_strdate(video.get('releaseDate')),
+ 'average_rating': float_or_none(video.get('rating') or metas.get('rating')),
+ 'comment_count': int_or_none(video.get('commentsCount')),
}
diff --git a/youtube_dlc/extractor/aenetworks.py b/youtube_dlc/extractor/aenetworks.py
index 611b948f5..8e4963131 100644
--- a/youtube_dlc/extractor/aenetworks.py
+++ b/youtube_dlc/extractor/aenetworks.py
@@ -5,20 +5,32 @@
from .theplatform import ThePlatformIE
from ..utils import (
- extract_attributes,
ExtractorError,
+ GeoRestrictedError,
int_or_none,
- smuggle_url,
update_url_query,
-)
-from ..compat import (
- compat_urlparse,
+ urlencode_postdata,
)
class AENetworksBaseIE(ThePlatformIE):
+ _BASE_URL_REGEX = r'''(?x)https?://
+ (?:(?:www|play|watch)\.)?
+ (?P
+ (?:history(?:vault)?|aetv|mylifetime|lifetimemovieclub)\.com|
+ fyi\.tv
+ )/'''
_THEPLATFORM_KEY = 'crazyjava'
_THEPLATFORM_SECRET = 's3cr3t'
+ _DOMAIN_MAP = {
+ 'history.com': ('HISTORY', 'history'),
+ 'aetv.com': ('AETV', 'aetv'),
+ 'mylifetime.com': ('LIFETIME', 'lifetime'),
+ 'lifetimemovieclub.com': ('LIFETIMEMOVIECLUB', 'lmc'),
+ 'fyi.tv': ('FYI', 'fyi'),
+ 'historyvault.com': (None, 'historyvault'),
+ 'biography.com': (None, 'biography'),
+ }
def _extract_aen_smil(self, smil_url, video_id, auth=None):
query = {'mbr': 'true'}
@@ -31,7 +43,7 @@ def _extract_aen_smil(self, smil_url, video_id, auth=None):
'assetTypes': 'high_video_s3'
}, {
'assetTypes': 'high_video_s3',
- 'switch': 'hls_ingest_fastly'
+ 'switch': 'hls_high_fastly',
}]
formats = []
subtitles = {}
@@ -44,6 +56,8 @@ def _extract_aen_smil(self, smil_url, video_id, auth=None):
tp_formats, tp_subtitles = self._extract_theplatform_smil(
m_url, video_id, 'Downloading %s SMIL data' % (q.get('switch') or q['assetTypes']))
except ExtractorError as e:
+ if isinstance(e, GeoRestrictedError):
+ raise
last_e = e
continue
formats.extend(tp_formats)
@@ -57,24 +71,45 @@ def _extract_aen_smil(self, smil_url, video_id, auth=None):
'subtitles': subtitles,
}
+ def _extract_aetn_info(self, domain, filter_key, filter_value, url):
+ requestor_id, brand = self._DOMAIN_MAP[domain]
+ result = self._download_json(
+ 'https://feeds.video.aetnd.com/api/v2/%s/videos' % brand,
+ filter_value, query={'filter[%s]' % filter_key: filter_value})['results'][0]
+ title = result['title']
+ video_id = result['id']
+ media_url = result['publicUrl']
+ theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
+ r'https?://link\.theplatform\.com/s/([^?]+)', media_url, 'theplatform_path'), video_id)
+ info = self._parse_theplatform_metadata(theplatform_metadata)
+ auth = None
+ if theplatform_metadata.get('AETN$isBehindWall'):
+ resource = self._get_mvpd_resource(
+ requestor_id, theplatform_metadata['title'],
+ theplatform_metadata.get('AETN$PPL_pplProgramId') or theplatform_metadata.get('AETN$PPL_pplProgramId_OLD'),
+ theplatform_metadata['ratings'][0]['rating'])
+ auth = self._extract_mvpd_auth(
+ url, video_id, requestor_id, resource)
+ info.update(self._extract_aen_smil(media_url, video_id, auth))
+ info.update({
+ 'title': title,
+ 'series': result.get('seriesName'),
+ 'season_number': int_or_none(result.get('tvSeasonNumber')),
+ 'episode_number': int_or_none(result.get('tvSeasonEpisodeNumber')),
+ })
+ return info
+
class AENetworksIE(AENetworksBaseIE):
IE_NAME = 'aenetworks'
IE_DESC = 'A+E Networks: A&E, Lifetime, History.com, FYI Network and History Vault'
- _VALID_URL = r'''(?x)
- https?://
- (?:www\.)?
- (?P
- (?:history(?:vault)?|aetv|mylifetime|lifetimemovieclub)\.com|
- fyi\.tv
- )/
- (?:
- shows/(?P[^/]+(?:/[^/]+){0,2})|
- movies/(?P[^/]+)(?:/full-movie)?|
- specials/(?P[^/]+)/(?:full-special|preview-)|
- collections/[^/]+/(?P[^/]+)
- )
- '''
+ _VALID_URL = AENetworksBaseIE._BASE_URL_REGEX + r'''(?P
+ shows/[^/]+/season-\d+/episode-\d+|
+ (?:
+ (?:movie|special)s/[^/]+|
+ (?:shows/[^/]+/)?videos
+ )/[^/?#&]+
+ )'''
_TESTS = [{
'url': 'http://www.history.com/shows/mountain-men/season-1/episode-1',
'info_dict': {
@@ -91,22 +126,23 @@ class AENetworksIE(AENetworksBaseIE):
'skip_download': True,
},
'add_ie': ['ThePlatform'],
+ 'skip': 'This video is only available for users of participating TV providers.',
}, {
- 'url': 'http://www.history.com/shows/ancient-aliens/season-1',
+ 'url': 'http://www.aetv.com/shows/duck-dynasty/season-9/episode-1',
'info_dict': {
- 'id': '71889446852',
+ 'id': '600587331957',
+ 'ext': 'mp4',
+ 'title': 'Inlawful Entry',
+ 'description': 'md5:57c12115a2b384d883fe64ca50529e08',
+ 'timestamp': 1452634428,
+ 'upload_date': '20160112',
+ 'uploader': 'AENE-NEW',
},
- 'playlist_mincount': 5,
- }, {
- 'url': 'http://www.mylifetime.com/shows/atlanta-plastic',
- 'info_dict': {
- 'id': 'SERIES4317',
- 'title': 'Atlanta Plastic',
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
},
- 'playlist_mincount': 2,
- }, {
- 'url': 'http://www.aetv.com/shows/duck-dynasty/season-9/episode-1',
- 'only_matching': True
+ 'add_ie': ['ThePlatform'],
}, {
'url': 'http://www.fyi.tv/shows/tiny-house-nation/season-1/episode-8',
'only_matching': True
@@ -117,78 +153,125 @@ class AENetworksIE(AENetworksBaseIE):
'url': 'http://www.mylifetime.com/movies/center-stage-on-pointe/full-movie',
'only_matching': True
}, {
- 'url': 'https://www.lifetimemovieclub.com/movies/a-killer-among-us',
+ 'url': 'https://watch.lifetimemovieclub.com/movies/10-year-reunion/full-movie',
'only_matching': True
}, {
'url': 'http://www.history.com/specials/sniper-into-the-kill-zone/full-special',
'only_matching': True
}, {
- 'url': 'https://www.historyvault.com/collections/america-the-story-of-us/westward',
+ 'url': 'https://www.aetv.com/specials/hunting-jonbenets-killer-the-untold-story/preview-hunting-jonbenets-killer-the-untold-story',
+ 'only_matching': True
+ }, {
+ 'url': 'http://www.history.com/videos/history-of-valentines-day',
'only_matching': True
}, {
- 'url': 'https://www.aetv.com/specials/hunting-jonbenets-killer-the-untold-story/preview-hunting-jonbenets-killer-the-untold-story',
+ 'url': 'https://play.aetv.com/shows/duck-dynasty/videos/best-of-duck-dynasty-getting-quack-in-shape',
'only_matching': True
}]
- _DOMAIN_TO_REQUESTOR_ID = {
- 'history.com': 'HISTORY',
- 'aetv.com': 'AETV',
- 'mylifetime.com': 'LIFETIME',
- 'lifetimemovieclub.com': 'LIFETIMEMOVIECLUB',
- 'fyi.tv': 'FYI',
- }
def _real_extract(self, url):
- domain, show_path, movie_display_id, special_display_id, collection_display_id = re.match(self._VALID_URL, url).groups()
- display_id = show_path or movie_display_id or special_display_id or collection_display_id
- webpage = self._download_webpage(url, display_id, headers=self.geo_verification_headers())
- if show_path:
- url_parts = show_path.split('/')
- url_parts_len = len(url_parts)
- if url_parts_len == 1:
- entries = []
- for season_url_path in re.findall(r'(?s)]+data-href="(/shows/%s/season-\d+)"' % url_parts[0], webpage):
- entries.append(self.url_result(
- compat_urlparse.urljoin(url, season_url_path), 'AENetworks'))
- if entries:
- return self.playlist_result(
- entries, self._html_search_meta('aetn:SeriesId', webpage),
- self._html_search_meta('aetn:SeriesTitle', webpage))
- else:
- # single season
- url_parts_len = 2
- if url_parts_len == 2:
- entries = []
- for episode_item in re.findall(r'(?s)<[^>]+class="[^"]*(?:episode|program)-item[^"]*"[^>]*>', webpage):
- episode_attributes = extract_attributes(episode_item)
- episode_url = compat_urlparse.urljoin(
- url, episode_attributes['data-canonical'])
- entries.append(self.url_result(
- episode_url, 'AENetworks',
- episode_attributes.get('data-videoid') or episode_attributes.get('data-video-id')))
- return self.playlist_result(
- entries, self._html_search_meta('aetn:SeasonId', webpage))
-
- video_id = self._html_search_meta('aetn:VideoID', webpage)
- media_url = self._search_regex(
- [r"media_url\s*=\s*'(?P[^']+)'",
- r'data-media-url=(?P(?:https?:)?//[^\s>]+)',
- r'data-media-url=(["\'])(?P(?:(?!\1).)+?)\1'],
- webpage, 'video url', group='url')
- theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
- r'https?://link\.theplatform\.com/s/([^?]+)', media_url, 'theplatform_path'), video_id)
- info = self._parse_theplatform_metadata(theplatform_metadata)
- auth = None
- if theplatform_metadata.get('AETN$isBehindWall'):
- requestor_id = self._DOMAIN_TO_REQUESTOR_ID[domain]
- resource = self._get_mvpd_resource(
- requestor_id, theplatform_metadata['title'],
- theplatform_metadata.get('AETN$PPL_pplProgramId') or theplatform_metadata.get('AETN$PPL_pplProgramId_OLD'),
- theplatform_metadata['ratings'][0]['rating'])
- auth = self._extract_mvpd_auth(
- url, video_id, requestor_id, resource)
- info.update(self._search_json_ld(webpage, video_id, fatal=False))
- info.update(self._extract_aen_smil(media_url, video_id, auth))
- return info
+ domain, canonical = re.match(self._VALID_URL, url).groups()
+ return self._extract_aetn_info(domain, 'canonical', '/' + canonical, url)
+
+
+class AENetworksListBaseIE(AENetworksBaseIE):
+ def _call_api(self, resource, slug, brand, fields):
+ return self._download_json(
+ 'https://yoga.appsvcs.aetnd.com/graphql',
+ slug, query={'brand': brand}, data=urlencode_postdata({
+ 'query': '''{
+ %s(slug: "%s") {
+ %s
+ }
+}''' % (resource, slug, fields),
+ }))['data'][resource]
+
+ def _real_extract(self, url):
+ domain, slug = re.match(self._VALID_URL, url).groups()
+ _, brand = self._DOMAIN_MAP[domain]
+ playlist = self._call_api(self._RESOURCE, slug, brand, self._FIELDS)
+ base_url = 'http://watch.%s' % domain
+
+ entries = []
+ for item in (playlist.get(self._ITEMS_KEY) or []):
+ doc = self._get_doc(item)
+ canonical = doc.get('canonical')
+ if not canonical:
+ continue
+ entries.append(self.url_result(
+ base_url + canonical, AENetworksIE.ie_key(), doc.get('id')))
+
+ description = None
+ if self._PLAYLIST_DESCRIPTION_KEY:
+ description = playlist.get(self._PLAYLIST_DESCRIPTION_KEY)
+
+ return self.playlist_result(
+ entries, playlist.get('id'),
+ playlist.get(self._PLAYLIST_TITLE_KEY), description)
+
+
+class AENetworksCollectionIE(AENetworksListBaseIE):
+ IE_NAME = 'aenetworks:collection'
+ _VALID_URL = AENetworksBaseIE._BASE_URL_REGEX + r'(?:[^/]+/)*(?:list|collections)/(?P[^/?#&]+)/?(?:[?#&]|$)'
+ _TESTS = [{
+ 'url': 'https://watch.historyvault.com/list/america-the-story-of-us',
+ 'info_dict': {
+ 'id': '282',
+ 'title': 'America The Story of Us',
+ },
+ 'playlist_mincount': 12,
+ }, {
+ 'url': 'https://watch.historyvault.com/shows/america-the-story-of-us-2/season-1/list/america-the-story-of-us',
+ 'only_matching': True
+ }, {
+ 'url': 'https://www.historyvault.com/collections/mysteryquest',
+ 'only_matching': True
+ }]
+ _RESOURCE = 'list'
+ _ITEMS_KEY = 'items'
+ _PLAYLIST_TITLE_KEY = 'display_title'
+ _PLAYLIST_DESCRIPTION_KEY = None
+ _FIELDS = '''id
+ display_title
+ items {
+ ... on ListVideoItem {
+ doc {
+ canonical
+ id
+ }
+ }
+ }'''
+
+ def _get_doc(self, item):
+ return item.get('doc') or {}
+
+
+class AENetworksShowIE(AENetworksListBaseIE):
+ IE_NAME = 'aenetworks:show'
+ _VALID_URL = AENetworksBaseIE._BASE_URL_REGEX + r'shows/(?P[^/?#&]+)/?(?:[?#&]|$)'
+ _TESTS = [{
+ 'url': 'http://www.history.com/shows/ancient-aliens',
+ 'info_dict': {
+ 'id': 'SH012427480000',
+ 'title': 'Ancient Aliens',
+ 'description': 'md5:3f6d74daf2672ff3ae29ed732e37ea7f',
+ },
+ 'playlist_mincount': 168,
+ }]
+ _RESOURCE = 'series'
+ _ITEMS_KEY = 'episodes'
+ _PLAYLIST_TITLE_KEY = 'title'
+ _PLAYLIST_DESCRIPTION_KEY = 'description'
+ _FIELDS = '''description
+ id
+ title
+ episodes {
+ canonical
+ id
+ }'''
+
+ def _get_doc(self, item):
+ return item
class HistoryTopicIE(AENetworksBaseIE):
@@ -204,6 +287,7 @@ class HistoryTopicIE(AENetworksBaseIE):
'description': 'md5:7b57ea4829b391995b405fa60bd7b5f7',
'timestamp': 1375819729,
'upload_date': '20130806',
+ 'uploader': 'AENE-NEW',
},
'params': {
# m3u8 download
@@ -212,36 +296,47 @@ class HistoryTopicIE(AENetworksBaseIE):
'add_ie': ['ThePlatform'],
}]
- def theplatform_url_result(self, theplatform_url, video_id, query):
- return {
- '_type': 'url_transparent',
- 'id': video_id,
- 'url': smuggle_url(
- update_url_query(theplatform_url, query),
- {
- 'sig': {
- 'key': self._THEPLATFORM_KEY,
- 'secret': self._THEPLATFORM_SECRET,
- },
- 'force_smil_url': True
- }),
- 'ie_key': 'ThePlatform',
- }
+ def _real_extract(self, url):
+ display_id = self._match_id(url)
+ return self.url_result(
+ 'http://www.history.com/videos/' + display_id,
+ AENetworksIE.ie_key())
+
+
+class HistoryPlayerIE(AENetworksBaseIE):
+ IE_NAME = 'history:player'
+ _VALID_URL = r'https?://(?:www\.)?(?P(?:history|biography)\.com)/player/(?P\d+)'
+ _TESTS = []
+
+ def _real_extract(self, url):
+ domain, video_id = re.match(self._VALID_URL, url).groups()
+ return self._extract_aetn_info(domain, 'id', video_id, url)
+
+
+class BiographyIE(AENetworksBaseIE):
+ _VALID_URL = r'https?://(?:www\.)?biography\.com/video/(?P[^/?#&]+)'
+ _TESTS = [{
+ 'url': 'https://www.biography.com/video/vincent-van-gogh-full-episode-2075049808',
+ 'info_dict': {
+ 'id': '30322987',
+ 'ext': 'mp4',
+ 'title': 'Vincent Van Gogh - Full Episode',
+ 'description': 'A full biography about the most influential 20th century painter, Vincent Van Gogh.',
+ 'timestamp': 1311970571,
+ 'upload_date': '20110729',
+ 'uploader': 'AENE-NEW',
+ },
+ 'params': {
+ # m3u8 download
+ 'skip_download': True,
+ },
+ 'add_ie': ['ThePlatform'],
+ }]
def _real_extract(self, url):
display_id = self._match_id(url)
webpage = self._download_webpage(url, display_id)
- video_id = self._search_regex(
- r']+src="[^"]+\btpid=(\d+)', webpage, 'tpid')
- result = self._download_json(
- 'https://feeds.video.aetnd.com/api/v2/history/videos',
- video_id, query={'filter[id]': video_id})['results'][0]
- title = result['title']
- info = self._extract_aen_smil(result['publicUrl'], video_id)
- info.update({
- 'title': title,
- 'description': result.get('description'),
- 'duration': int_or_none(result.get('duration')),
- 'timestamp': int_or_none(result.get('added'), 1000),
- })
- return info
+ player_url = self._search_regex(
+ r']+src="(%s)' % HistoryPlayerIE._VALID_URL,
+ webpage, 'player URL')
+ return self.url_result(player_url, HistoryPlayerIE.ie_key())
diff --git a/youtube_dlc/extractor/amcnetworks.py b/youtube_dlc/extractor/amcnetworks.py
index 6fb3d6c53..b8027bbca 100644
--- a/youtube_dlc/extractor/amcnetworks.py
+++ b/youtube_dlc/extractor/amcnetworks.py
@@ -1,6 +1,8 @@
# coding: utf-8
from __future__ import unicode_literals
+import re
+
from .theplatform import ThePlatformIE
from ..utils import (
int_or_none,
@@ -11,25 +13,22 @@
class AMCNetworksIE(ThePlatformIE):
- _VALID_URL = r'https?://(?:www\.)?(?:amc|bbcamerica|ifc|(?:we|sundance)tv)\.com/(?:movies|shows(?:/[^/]+)+)/(?P[^/?#]+)'
+ _VALID_URL = r'https?://(?:www\.)?(?Pamc|bbcamerica|ifc|(?:we|sundance)tv)\.com/(?P(?:movies|shows(?:/[^/]+)+)/[^/?#&]+)'
_TESTS = [{
- 'url': 'http://www.ifc.com/shows/maron/season-04/episode-01/step-1',
- 'md5': '',
+ 'url': 'https://www.bbcamerica.com/shows/the-graham-norton-show/videos/tina-feys-adorable-airline-themed-family-dinner--51631',
'info_dict': {
- 'id': 's3MX01Nl4vPH',
+ 'id': '4Lq1dzOnZGt0',
'ext': 'mp4',
- 'title': 'Maron - Season 4 - Step 1',
- 'description': 'In denial about his current situation, Marc is reluctantly convinced by his friends to enter rehab. Starring Marc Maron and Constance Zimmer.',
- 'age_limit': 17,
- 'upload_date': '20160505',
- 'timestamp': 1462468831,
+ 'title': "The Graham Norton Show - Season 28 - Tina Fey's Adorable Airline-Themed Family Dinner",
+ 'description': "It turns out child stewardesses are very generous with the wine! All-new episodes of 'The Graham Norton Show' premiere Fridays at 11/10c on BBC America.",
+ 'upload_date': '20201120',
+ 'timestamp': 1605904350,
'uploader': 'AMCN',
},
'params': {
# m3u8 download
'skip_download': True,
},
- 'skip': 'Requires TV provider accounts',
}, {
'url': 'http://www.bbcamerica.com/shows/the-hunt/full-episodes/season-1/episode-01-the-hardest-challenge',
'only_matching': True,
@@ -55,32 +54,34 @@ class AMCNetworksIE(ThePlatformIE):
'url': 'https://www.sundancetv.com/shows/riviera/full-episodes/season-1/episode-01-episode-1',
'only_matching': True,
}]
+ _REQUESTOR_ID_MAP = {
+ 'amc': 'AMC',
+ 'bbcamerica': 'BBCA',
+ 'ifc': 'IFC',
+ 'sundancetv': 'SUNDANCE',
+ 'wetv': 'WETV',
+ }
def _real_extract(self, url):
- display_id = self._match_id(url)
- webpage = self._download_webpage(url, display_id)
+ site, display_id = re.match(self._VALID_URL, url).groups()
+ requestor_id = self._REQUESTOR_ID_MAP[site]
+ properties = self._download_json(
+ 'https://content-delivery-gw.svc.ds.amcn.com/api/v2/content/amcn/%s/url/%s' % (requestor_id.lower(), display_id),
+ display_id)['data']['properties']
query = {
'mbr': 'true',
'manifest': 'm3u',
}
- media_url = self._search_regex(
- r'window\.platformLinkURL\s*=\s*[\'"]([^\'"]+)',
- webpage, 'media url')
- theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
- r'link\.theplatform\.com/s/([^?]+)',
- media_url, 'theplatform_path'), display_id)
+ tp_path = 'M_UwQC/media/' + properties['videoPid']
+ media_url = 'https://link.theplatform.com/s/' + tp_path
+ theplatform_metadata = self._download_theplatform_metadata(tp_path, display_id)
info = self._parse_theplatform_metadata(theplatform_metadata)
video_id = theplatform_metadata['pid']
title = theplatform_metadata['title']
rating = try_get(
theplatform_metadata, lambda x: x['ratings'][0]['rating'])
- auth_required = self._search_regex(
- r'window\.authRequired\s*=\s*(true|false);',
- webpage, 'auth required')
- if auth_required == 'true':
- requestor_id = self._search_regex(
- r'window\.requestor_id\s*=\s*[\'"]([^\'"]+)',
- webpage, 'requestor id')
+ video_category = properties.get('videoCategory')
+ if video_category and video_category.endswith('-Auth'):
resource = self._get_mvpd_resource(
requestor_id, title, video_id, rating)
query['auth'] = self._extract_mvpd_auth(
diff --git a/youtube_dlc/extractor/americastestkitchen.py b/youtube_dlc/extractor/americastestkitchen.py
index 9c9d77ae1..e20f00fc3 100644
--- a/youtube_dlc/extractor/americastestkitchen.py
+++ b/youtube_dlc/extractor/americastestkitchen.py
@@ -1,33 +1,33 @@
# coding: utf-8
from __future__ import unicode_literals
+import re
+
from .common import InfoExtractor
from ..utils import (
clean_html,
- int_or_none,
- js_to_json,
try_get,
unified_strdate,
)
class AmericasTestKitchenIE(InfoExtractor):
- _VALID_URL = r'https?://(?:www\.)?americastestkitchen\.com/(?:episode|videos)/(?P\d+)'
+ _VALID_URL = r'https?://(?:www\.)?(?:americastestkitchen|cooks(?:country|illustrated))\.com/(?Pepisode|videos)/(?P\d+)'
_TESTS = [{
'url': 'https://www.americastestkitchen.com/episode/582-weeknight-japanese-suppers',
'md5': 'b861c3e365ac38ad319cfd509c30577f',
'info_dict': {
'id': '5b400b9ee338f922cb06450c',
- 'title': 'Weeknight Japanese Suppers',
+ 'title': 'Japanese Suppers',
'ext': 'mp4',
- 'description': 'md5:3d0c1a44bb3b27607ce82652db25b4a8',
+ 'description': 'md5:64e606bfee910627efc4b5f050de92b3',
'thumbnail': r're:^https?://',
'timestamp': 1523664000,
'upload_date': '20180414',
- 'release_date': '20180414',
+ 'release_date': '20180410',
'series': "America's Test Kitchen",
'season_number': 18,
- 'episode': 'Weeknight Japanese Suppers',
+ 'episode': 'Japanese Suppers',
'episode_number': 15,
},
'params': {
@@ -36,47 +36,31 @@ class AmericasTestKitchenIE(InfoExtractor):
}, {
'url': 'https://www.americastestkitchen.com/videos/3420-pan-seared-salmon',
'only_matching': True,
+ }, {
+ 'url': 'https://www.cookscountry.com/episode/564-when-only-chocolate-will-do',
+ 'only_matching': True,
+ }, {
+ 'url': 'https://www.cooksillustrated.com/videos/4478-beef-wellington',
+ 'only_matching': True,
}]
def _real_extract(self, url):
- video_id = self._match_id(url)
-
- webpage = self._download_webpage(url, video_id)
+ resource_type, video_id = re.match(self._VALID_URL, url).groups()
+ is_episode = resource_type == 'episode'
+ if is_episode:
+ resource_type = 'episodes'
- video_data = self._parse_json(
- self._search_regex(
- r'window\.__INITIAL_STATE__\s*=\s*({.+?})\s*;\s*',
- webpage, 'initial context'),
- video_id, js_to_json)
-
- ep_data = try_get(
- video_data,
- (lambda x: x['episodeDetail']['content']['data'],
- lambda x: x['videoDetail']['content']['data']), dict)
- ep_meta = ep_data.get('full_video', {})
-
- zype_id = ep_data.get('zype_id') or ep_meta['zype_id']
-
- title = ep_data.get('title') or ep_meta.get('title')
- description = clean_html(ep_meta.get('episode_description') or ep_data.get(
- 'description') or ep_meta.get('description'))
- thumbnail = try_get(ep_meta, lambda x: x['photo']['image_url'])
- release_date = unified_strdate(ep_data.get('aired_at'))
-
- season_number = int_or_none(ep_meta.get('season_number'))
- episode = ep_meta.get('title')
- episode_number = int_or_none(ep_meta.get('episode_number'))
+ resource = self._download_json(
+ 'https://www.americastestkitchen.com/api/v6/%s/%s' % (resource_type, video_id), video_id)
+ video = resource['video'] if is_episode else resource
+ episode = resource if is_episode else resource.get('episode') or {}
return {
'_type': 'url_transparent',
- 'url': 'https://player.zype.com/embed/%s.js?api_key=jZ9GUhRmxcPvX7M3SlfejB6Hle9jyHTdk2jVxG7wOHPLODgncEKVdPYBhuz9iWXQ' % zype_id,
+ 'url': 'https://player.zype.com/embed/%s.js?api_key=jZ9GUhRmxcPvX7M3SlfejB6Hle9jyHTdk2jVxG7wOHPLODgncEKVdPYBhuz9iWXQ' % video['zypeId'],
'ie_key': 'Zype',
- 'title': title,
- 'description': description,
- 'thumbnail': thumbnail,
- 'release_date': release_date,
- 'series': "America's Test Kitchen",
- 'season_number': season_number,
- 'episode': episode,
- 'episode_number': episode_number,
+ 'description': clean_html(video.get('description')),
+ 'release_date': unified_strdate(video.get('publishDate')),
+ 'series': try_get(episode, lambda x: x['show']['title']),
+ 'episode': episode.get('title'),
}
diff --git a/youtube_dlc/extractor/animeondemand.py b/youtube_dlc/extractor/animeondemand.py
index 00ce684d1..54e097d2f 100644
--- a/youtube_dlc/extractor/animeondemand.py
+++ b/youtube_dlc/extractor/animeondemand.py
@@ -116,8 +116,6 @@ def _real_extract(self, url):
r'(?s)]+itemprop="description"[^>]*>(.+?)',
webpage, 'anime description', default=None)
- entries = []
-
def extract_info(html, video_id, num=None):
title, description = [None] * 2
formats = []
@@ -233,7 +231,7 @@ def extract_entries(html, video_id, common_info, num=None):
self._sort_formats(info['formats'])
f = common_info.copy()
f.update(info)
- entries.append(f)
+ yield f
# Extract teaser/trailer only when full episode is not available
if not info['formats']:
@@ -247,7 +245,7 @@ def extract_entries(html, video_id, common_info, num=None):
'title': m.group('title'),
'url': urljoin(url, m.group('href')),
})
- entries.append(f)
+ yield f
def extract_episodes(html):
for num, episode_html in enumerate(re.findall(
@@ -275,7 +273,8 @@ def extract_episodes(html):
'episode_number': episode_number,
}
- extract_entries(episode_html, video_id, common_info)
+ for e in extract_entries(episode_html, video_id, common_info):
+ yield e
def extract_film(html, video_id):
common_info = {
@@ -283,11 +282,18 @@ def extract_film(html, video_id):
'title': anime_title,
'description': anime_description,
}
- extract_entries(html, video_id, common_info)
+ for e in extract_entries(html, video_id, common_info):
+ yield e
- extract_episodes(webpage)
+ def entries():
+ has_episodes = False
+ for e in extract_episodes(webpage):
+ has_episodes = True
+ yield e
- if not entries:
- extract_film(webpage, anime_id)
+ if not has_episodes:
+ for e in extract_film(webpage, anime_id):
+ yield e
- return self.playlist_result(entries, anime_id, anime_title, anime_description)
+ return self.playlist_result(
+ entries(), anime_id, anime_title, anime_description)
diff --git a/youtube_dlc/extractor/anvato.py b/youtube_dlc/extractor/anvato.py
index 84e841035..b7398563b 100644
--- a/youtube_dlc/extractor/anvato.py
+++ b/youtube_dlc/extractor/anvato.py
@@ -116,7 +116,76 @@ class AnvatoIE(InfoExtractor):
'anvato_scripps_app_ios_prod_409c41960c60b308db43c3cc1da79cab9f1c3d93': 'WPxj5GraLTkYCyj3M7RozLqIycjrXOEcDGFMIJPn',
'EZqvRyKBJLrgpClDPDF8I7Xpdp40Vx73': '4OxGd2dEakylntVKjKF0UK9PDPYB6A9W',
'M2v78QkpleXm9hPp9jUXI63x5vA6BogR': 'ka6K32k7ZALmpINkjJUGUo0OE42Md1BQ',
- 'nbcu_nbcd_desktop_web_prod_93d8ead38ce2024f8f544b78306fbd15895ae5e6_secure': 'NNemUkySjxLyPTKvZRiGntBIjEyK8uqicjMakIaQ'
+ 'nbcu_nbcd_desktop_web_prod_93d8ead38ce2024f8f544b78306fbd15895ae5e6_secure': 'NNemUkySjxLyPTKvZRiGntBIjEyK8uqicjMakIaQ',
+ 'X8POa4zPPaKVZHqmWjuEzfP31b1QM9VN': 'Dn5vOY9ooDw7VSl9qztjZI5o0g08mA0z',
+ 'M2v78QkBMpNJlSPp9diX5F2PBmBy6Bog': 'ka6K32kyo7nDZfNkjQCGWf1lpApXMd1B',
+ 'bvJ0dQpav07l0hG5JgfVLF2dv1vARwpP': 'BzoQW24GrJZoJfmNodiJKSPeB9B8NOxj',
+ 'lxQMLg2XZKuEZaWgsqubBxV9INZ6bryY': 'Vm2Mx6noKds9jB71h6urazwlTG3m9x8l',
+ '04EnjvXeoSmkbJ9ckPs7oY0mcxv7PlyN': 'aXERQP9LMfQVlEDsgGs6eEA1SWznAQ8P',
+ 'mQbO2ge6BFRWVPYCYpU06YvNt80XLvAX': 'E2BV1NGmasN5v7eujECVPJgwflnLPm2A',
+ 'g43oeBzJrCml7o6fa5fRL1ErCdeD8z4K': 'RX34mZ6zVH4Nr6whbxIGLv9WSbxEKo8V',
+ 'VQrDJoP7mtdBzkxhXbSPwGB1coeElk4x': 'j2VejQx0VFKQepAF7dI0mJLKtOVJE18z',
+ 'WxA5NzLRjCrmq0NUgaU5pdMDuZO7RJ4w': 'lyY5ADLKaIOLEgAsGQCveEMAcqnx3rY9',
+ 'M4lpMXB71ie0PjMCjdFzVXq0SeRVqz49': 'n2zVkOqaLIv3GbLfBjcwW51LcveWOZ2e',
+ 'dyDZGEqN8u8nkJZcJns0oxYmtP7KbGAn': 'VXOEqQW9BtEVLajfZQSLEqxgS5B7qn2D',
+ 'E7QNjrVY5u5mGvgu67IoDgV1CjEND8QR': 'rz8AaDmdKIkLmPNhB5ILPJnjS5PnlL8d',
+ 'a4zrqjoKlfzg0dwHEWtP31VqcLBpjm4g': 'LY9J16gwETdGWa3hjBu5o0RzuoQDjqXQ',
+ 'dQP5BZroMsMVLO1hbmT5r2Enu86GjxA6': '7XR3oOdbPF6x3PRFLDCq9RkgsRjAo48V',
+ 'M4lKNBO1NFe0PjMCj1tzVXq0SeRVqzA9': 'n2zoRqGLRUv3GbLfBmTwW51LcveWOZYe',
+ 'nAZ7MZdpGCGg1pqFEbsoJOz2C60mv143': 'dYJgdqA9aT4yojETqGi7yNgoFADxqmXP',
+ '3y1MERYgOuE9NzbFgwhV6Wv2F0YKvbyz': '081xpZDQgC4VadLTavhWQxrku56DAgXV',
+ 'bmQvmEXr5HWklBMCZOcpE2Z3HBYwqGyl': 'zxXPbVNyMiMAZldhr9FkOmA0fl4aKr2v',
+ 'wA7oDNYldfr6050Hwxi52lPZiVlB86Ap': 'ZYK16aA7ni0d3l3c34uwpxD7CbReMm8Q',
+ 'g43MbKMWmFml7o7sJoSRkXxZiXRvJ3QK': 'RX3oBJonvs4Nr6rUWBCGn3matRGqJPXV',
+ 'mA9VdlqpLS0raGaSDvtoqNrBTzb8XY4q': '0XN4OjBD3fnW7r7IbmtJB4AyfOmlrE2r',
+ 'mAajOwgkGt17oGoFmEuklMP9H0GnW54d': 'lXbBLPGyzikNGeGujAuAJGjZiwLRxyXR',
+ 'vy8vjJ9kbUwrRqRu59Cj5dWZfzYErlAb': 'K8l7gpwaGcBpnAnCLNCmPZRdin3eaQX0',
+ 'xQMWBpR8oHEZaWaSMGUb0avOHjLVYn4Y': 'm2MrN4vEaf9jB7BFy5Srb40jTrN67AYl',
+ 'xyKEmVO3miRr6D6UVkt7oB8jtD6aJEAv': 'g2ddDebqDfqdgKgswyUKwGjbTWwzq923',
+ '7Qk0wa2D9FjKapacoJF27aLvUDKkLGA0': 'b2kgBEkephJaMkMTL7s1PLe4Ua6WyP2P',
+ '3QLg6nqmNTJ5VvVTo7f508LPidz1xwyY': 'g2L1GgpraipmAOAUqmIbBnPxHOmw4MYa',
+ '3y1B7zZjXTE9NZNSzZSVNPZaTNLjo6Qz': '081b5G6wzH4VagaURmcWbN5mT4JGEe2V',
+ 'lAqnwvkw6SG6D8DSqmUg6DRLUp0w3G4x': 'O2pbP0xPDFNJjpjIEvcdryOJtpkVM4X5',
+ 'awA7xd1N0Hr6050Hw2c52lPZiVlB864p': 'GZYKpn4aoT0d3l3c3PiwpxD7CbReMmXQ',
+ 'jQVqPLl9YHL1WGWtR1HDgWBGT63qRNyV': '6X03ne6vrU4oWyWUN7tQVoajikxJR3Ye',
+ 'GQRMR8mL7uZK797t7xH3eNzPIP5dOny1': 'm2vqPWGd4U31zWzSyasDRAoMT1PKRp8o',
+ 'zydq9RdmRhXLkNkfNoTJlMzaF0lWekQB': '3X7LnvE7vH5nkEkSqLiey793Un7dLB8e',
+ 'VQrDzwkB2IdBzjzu9MHPbEYkSB50gR4x': 'j2VebLzoKUKQeEesmVh0gM1eIp9jKz8z',
+ 'mAa2wMamBs17oGoFmktklMP9H0GnW54d': 'lXbgP74xZTkNGeGujVUAJGjZiwLRxy8R',
+ '7yjB6ZLG6sW8R6RF2xcan1KGfJ5dNoyd': 'wXQkPorvPHZ45N5t4Jf6qwg5Tp4xvw29',
+ 'a4zPpNeWGuzg0m0iX3tPeanGSkRKWXQg': 'LY9oa3QAyHdGW9Wu3Ri5JGeEik7l1N8Q',
+ 'k2rneA2M38k25cXDwwSknTJlxPxQLZ6M': '61lyA2aEVDzklfdwmmh31saPxQx2VRjp',
+ 'bK9Zk4OvPnvxduLgxvi8VUeojnjA02eV': 'o5jANYjbeMb4nfBaQvcLAt1jzLzYx6ze',
+ '5VD6EydM3R9orHmNMGInGCJwbxbQvGRw': 'w3zjmX7g4vnxzCxElvUEOiewkokXprkZ',
+ '70X35QbVYVYNPUmP9YfbzI06YqYQk2R1': 'vG4Aj2BMjMjoztB7zeFOnCVPJpJ8lMOa',
+ '26qYwQVG9p1Bks2GgBckjfDJOXOAMgG1': 'r4ev9X0mv5zqJc0yk5IBDcQOwZw8mnwQ',
+ 'rvVKpA56MBXWlSxMw3cobT5pdkd4Dm7q': '1J7ZkY53pZ645c93owcLZuveE7E8B3rL',
+ 'qN1zdy1zlYL23IWZGWtDvfV6WeWQWkJo': 'qN1zdy1zlYL23IWZGWtDvfV6WeWQWkJo',
+ 'jdKqRGF16dKsBviMDae7IGDl7oTjEbVV': 'Q09l7vhlNxPFErIOK6BVCe7KnwUW5DVV',
+ '3QLkogW1OUJ5VvPsrDH56DY2u7lgZWyY': 'g2LRE1V9espmAOPhE4ubj4ZdUA57yDXa',
+ 'wyJvWbXGBSdbkEzhv0CW8meou82aqRy8': 'M2wolPvyBIpQGkbT4juedD4ruzQGdK2y',
+ '7QkdZrzEkFjKap6IYDU2PB0oCNZORmA0': 'b2kN1l96qhJaMkPs9dt1lpjBfwqZoA8P',
+ 'pvA05113MHG1w3JTYxc6DVlRCjErVz4O': 'gQXeAbblBUnDJ7vujbHvbRd1cxlz3AXO',
+ 'mA9blJDZwT0raG1cvkuoeVjLC7ZWd54q': '0XN9jRPwMHnW7rvumgfJZOD9CJgVkWYr',
+ '5QwRN5qKJTvGKlDTmnf7xwNZcjRmvEy9': 'R2GP6LWBJU1QlnytwGt0B9pytWwAdDYy',
+ 'eyn5rPPbkfw2KYxH32fG1q58CbLJzM40': 'p2gyqooZnS56JWeiDgfmOy1VugOQEBXn',
+ '3BABn3b5RfPJGDwilbHe7l82uBoR05Am': '7OYZG7KMVhbPdKJS3xcWEN3AuDlLNmXj',
+ 'xA5zNGXD3HrmqMlF6OS5pdMDuZO7RJ4w': 'yY5DAm6r1IOLE3BCVMFveEMAcqnx3r29',
+ 'g43PgW3JZfml7o6fDEURL1ErCdeD8zyK': 'RX3aQn1zrS4Nr6whDgCGLv9WSbxEKo2V',
+ 'lAqp8WbGgiG6D8LTKJcg3O72CDdre1Qx': 'O2pnm6473HNJjpKuVosd3vVeh975yrX5',
+ 'wyJbYEDxKSdbkJ6S6RhW8meou82aqRy8': 'M2wPm7EgRSpQGlAh70CedD4ruzQGdKYy',
+ 'M4lgW28nLCe0PVdtaXszVXq0SeRVqzA9': 'n2zmJvg4jHv3G0ETNgiwW51LcveWOZ8e',
+ '5Qw3OVvp9FvGKlDTmOC7xwNZcjRmvEQ9': 'R2GzDdml9F1Qlnytw9s0B9pytWwAdD8y',
+ 'vy8a98X7zCwrRqbHrLUjYzwDiK2b70Qb': 'K8lVwzyjZiBpnAaSGeUmnAgxuGOBxmY0',
+ 'g4eGjJLLoiqRD3Pf9oT5O03LuNbLRDQp': '6XqD59zzpfN4EwQuaGt67qNpSyRBlnYy',
+ 'g43OPp9boIml7o6fDOIRL1ErCdeD8z4K': 'RX33alNB4s4Nr6whDPUGLv9WSbxEKoXV',
+ 'xA2ng9OkBcGKzDbTkKsJlx7dUK8R3dA5': 'z2aPnJvzBfObkwGC3vFaPxeBhxoMqZ8K',
+ 'xyKEgBajZuRr6DEC0Kt7XpD1cnNW9gAv': 'g2ddlEBvRsqdgKaI4jUK9PrgfMexGZ23',
+ 'BAogww51jIMa2JnH1BcYpXM5F658RNAL': 'rYWDmm0KptlkGv4FGJFMdZmjs9RDE6XR',
+ 'BAokpg62VtMa2JnH1mHYpXM5F658RNAL': 'rYWryDnlNslkGv4FG4HMdZmjs9RDE62R',
+ 'a4z1Px5e2hzg0m0iMMCPeanGSkRKWXAg': 'LY9eorNQGUdGW9WuKKf5JGeEik7l1NYQ',
+ 'kAx69R58kF9nY5YcdecJdl2pFXP53WyX': 'gXyRxELpbfPvLeLSaRil0mp6UEzbZJ8L',
+ 'BAoY13nwViMa2J2uo2cY6BlETgmdwryL': 'rYWwKzJmNFlkGvGtNoUM9bzwIJVzB1YR',
}
_MCP_TO_ACCESS_KEY_TABLE = {
@@ -189,19 +258,17 @@ def _get_video_json(self, access_key, video_id):
video_data_url += '&X-Anvato-Adst-Auth=' + base64.b64encode(auth_secret).decode('ascii')
anvrid = md5_text(time.time() * 1000 * random.random())[:30]
- payload = {
- 'api': {
- 'anvrid': anvrid,
- 'anvstk': md5_text('%s|%s|%d|%s' % (
- access_key, anvrid, server_time,
- self._ANVACK_TABLE.get(access_key, self._API_KEY))),
- 'anvts': server_time,
- },
+ api = {
+ 'anvrid': anvrid,
+ 'anvts': server_time,
}
+ api['anvstk'] = md5_text('%s|%s|%d|%s' % (
+ access_key, anvrid, server_time,
+ self._ANVACK_TABLE.get(access_key, self._API_KEY)))
return self._download_json(
video_data_url, video_id, transform_source=strip_jsonp,
- data=json.dumps(payload).encode('utf-8'))
+ data=json.dumps({'api': api}).encode('utf-8'))
def _get_anvato_videos(self, access_key, video_id):
video_data = self._get_video_json(access_key, video_id)
@@ -259,7 +326,7 @@ def _get_anvato_videos(self, access_key, video_id):
'description': video_data.get('def_description'),
'tags': video_data.get('def_tags', '').split(','),
'categories': video_data.get('categories'),
- 'thumbnail': video_data.get('thumbnail'),
+ 'thumbnail': video_data.get('src_image_url') or video_data.get('thumbnail'),
'timestamp': int_or_none(video_data.get(
'ts_published') or video_data.get('ts_added')),
'uploader': video_data.get('mcp_id'),
diff --git a/youtube_dlc/extractor/anvato_token_generator/__init__.py b/youtube_dlc/extractor/anvato_token_generator/__init__.py
new file mode 100644
index 000000000..6e223db9f
--- /dev/null
+++ b/youtube_dlc/extractor/anvato_token_generator/__init__.py
@@ -0,0 +1,7 @@
+from __future__ import unicode_literals
+
+from .nfl import NFLTokenGenerator
+
+__all__ = [
+ 'NFLTokenGenerator',
+]
diff --git a/youtube_dlc/extractor/anvato_token_generator/common.py b/youtube_dlc/extractor/anvato_token_generator/common.py
new file mode 100644
index 000000000..b959a903b
--- /dev/null
+++ b/youtube_dlc/extractor/anvato_token_generator/common.py
@@ -0,0 +1,6 @@
+from __future__ import unicode_literals
+
+
+class TokenGenerator:
+ def generate(self, anvack, mcp_id):
+ raise NotImplementedError('This method must be implemented by subclasses')
diff --git a/youtube_dlc/extractor/anvato_token_generator/nfl.py b/youtube_dlc/extractor/anvato_token_generator/nfl.py
new file mode 100644
index 000000000..97a2b245f
--- /dev/null
+++ b/youtube_dlc/extractor/anvato_token_generator/nfl.py
@@ -0,0 +1,30 @@
+from __future__ import unicode_literals
+
+import json
+
+from .common import TokenGenerator
+
+
+class NFLTokenGenerator(TokenGenerator):
+ _AUTHORIZATION = None
+
+ def generate(ie, anvack, mcp_id):
+ if not NFLTokenGenerator._AUTHORIZATION:
+ reroute = ie._download_json(
+ 'https://api.nfl.com/v1/reroute', mcp_id,
+ data=b'grant_type=client_credentials',
+ headers={'X-Domain-Id': 100})
+ NFLTokenGenerator._AUTHORIZATION = '%s %s' % (reroute.get('token_type') or 'Bearer', reroute['access_token'])
+ return ie._download_json(
+ 'https://api.nfl.com/v3/shield/', mcp_id, data=json.dumps({
+ 'query': '''{
+ viewer {
+ mediaToken(anvack: "%s", id: %s) {
+ token
+ }
+ }
+}''' % (anvack, mcp_id),
+ }).encode(), headers={
+ 'Authorization': NFLTokenGenerator._AUTHORIZATION,
+ 'Content-Type': 'application/json',
+ })['data']['viewer']['mediaToken']['token']
diff --git a/youtube_dlc/extractor/aparat.py b/youtube_dlc/extractor/aparat.py
index 883dcee7a..a9527e785 100644
--- a/youtube_dlc/extractor/aparat.py
+++ b/youtube_dlc/extractor/aparat.py
@@ -3,6 +3,7 @@
from .common import InfoExtractor
from ..utils import (
+ get_element_by_id,
int_or_none,
merge_dicts,
mimetype2ext,
@@ -39,23 +40,15 @@ def _real_extract(self, url):
webpage = self._download_webpage(url, video_id, fatal=False)
if not webpage:
- # Note: There is an easier-to-parse configuration at
- # http://www.aparat.com/video/video/config/videohash/%video_id
- # but the URL in there does not work
webpage = self._download_webpage(
'http://www.aparat.com/video/video/embed/vt/frame/showvideo/yes/videohash/' + video_id,
video_id)
- options = self._parse_json(
- self._search_regex(
- r'options\s*=\s*JSON\.parse\(\s*(["\'])(?P(?:(?!\1).)+)\1\s*\)',
- webpage, 'options', group='value'),
- video_id)
-
- player = options['plugins']['sabaPlayerPlugin']
+ options = self._parse_json(self._search_regex(
+ r'options\s*=\s*({.+?})\s*;', webpage, 'options'), video_id)
formats = []
- for sources in player['multiSRC']:
+ for sources in (options.get('multiSRC') or []):
for item in sources:
if not isinstance(item, dict):
continue
@@ -85,11 +78,12 @@ def _real_extract(self, url):
info = self._search_json_ld(webpage, video_id, default={})
if not info.get('title'):
- info['title'] = player['title']
+ info['title'] = get_element_by_id('videoTitle', webpage) or \
+ self._html_search_meta(['og:title', 'twitter:title', 'DC.Title', 'title'], webpage, fatal=True)
return merge_dicts(info, {
'id': video_id,
'thumbnail': url_or_none(options.get('poster')),
- 'duration': int_or_none(player.get('duration')),
+ 'duration': int_or_none(options.get('duration')),
'formats': formats,
})
diff --git a/youtube_dlc/extractor/applepodcasts.py b/youtube_dlc/extractor/applepodcasts.py
new file mode 100644
index 000000000..95758fece
--- /dev/null
+++ b/youtube_dlc/extractor/applepodcasts.py
@@ -0,0 +1,61 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+from ..utils import (
+ clean_podcast_url,
+ int_or_none,
+ parse_iso8601,
+ try_get,
+)
+
+
+class ApplePodcastsIE(InfoExtractor):
+ _VALID_URL = r'https?://podcasts\.apple\.com/(?:[^/]+/)?podcast(?:/[^/]+){1,2}.*?\bi=(?P\d+)'
+ _TESTS = [{
+ 'url': 'https://podcasts.apple.com/us/podcast/207-whitney-webb-returns/id1135137367?i=1000482637777',
+ 'md5': 'df02e6acb11c10e844946a39e7222b08',
+ 'info_dict': {
+ 'id': '1000482637777',
+ 'ext': 'mp3',
+ 'title': '207 - Whitney Webb Returns',
+ 'description': 'md5:13a73bade02d2e43737751e3987e1399',
+ 'upload_date': '20200705',
+ 'timestamp': 1593921600,
+ 'duration': 6425,
+ 'series': 'The Tim Dillon Show',
+ }
+ }, {
+ 'url': 'https://podcasts.apple.com/podcast/207-whitney-webb-returns/id1135137367?i=1000482637777',
+ 'only_matching': True,
+ }, {
+ 'url': 'https://podcasts.apple.com/podcast/207-whitney-webb-returns?i=1000482637777',
+ 'only_matching': True,
+ }, {
+ 'url': 'https://podcasts.apple.com/podcast/id1135137367?i=1000482637777',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ episode_id = self._match_id(url)
+ webpage = self._download_webpage(url, episode_id)
+ ember_data = self._parse_json(self._search_regex(
+ r'id="shoebox-ember-data-store"[^>]*>\s*({.+?})\s*<',
+ webpage, 'ember data'), episode_id)
+ episode = ember_data['data']['attributes']
+ description = episode.get('description') or {}
+
+ series = None
+ for inc in (ember_data.get('included') or []):
+ if inc.get('type') == 'media/podcast':
+ series = try_get(inc, lambda x: x['attributes']['name'])
+
+ return {
+ 'id': episode_id,
+ 'title': episode['name'],
+ 'url': clean_podcast_url(episode['assetUrl']),
+ 'description': description.get('standard') or description.get('short'),
+ 'timestamp': parse_iso8601(episode.get('releaseDateTime')),
+ 'duration': int_or_none(episode.get('durationInMilliseconds'), 1000),
+ 'series': series,
+ }
diff --git a/youtube_dlc/extractor/arcpublishing.py b/youtube_dlc/extractor/arcpublishing.py
new file mode 100644
index 000000000..ca6a6c4d8
--- /dev/null
+++ b/youtube_dlc/extractor/arcpublishing.py
@@ -0,0 +1,174 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+ extract_attributes,
+ int_or_none,
+ parse_iso8601,
+ try_get,
+)
+
+
+class ArcPublishingIE(InfoExtractor):
+ _UUID_REGEX = r'[\da-f]{8}-(?:[\da-f]{4}-){3}[\da-f]{12}'
+ _VALID_URL = r'arcpublishing:(?P[a-z]+):(?P%s)' % _UUID_REGEX
+ _TESTS = [{
+ # https://www.adn.com/politics/2020/11/02/video-senate-candidates-campaign-in-anchorage-on-eve-of-election-day/
+ 'url': 'arcpublishing:adn:8c99cb6e-b29c-4bc9-9173-7bf9979225ab',
+ 'only_matching': True,
+ }, {
+ # https://www.bostonglobe.com/video/2020/12/30/metro/footage-released-showing-officer-talking-about-striking-protesters-with-car/
+ 'url': 'arcpublishing:bostonglobe:232b7ae6-7d73-432d-bc0a-85dbf0119ab1',
+ 'only_matching': True,
+ }, {
+ # https://www.actionnewsjax.com/video/live-stream/
+ 'url': 'arcpublishing:cmg:cfb1cf1b-3ab5-4d1b-86c5-a5515d311f2a',
+ 'only_matching': True,
+ }, {
+ # https://elcomercio.pe/videos/deportes/deporte-total-futbol-peruano-seleccion-peruana-la-valorizacion-de-los-peruanos-en-el-exterior-tras-un-2020-atipico-nnav-vr-video-noticia/
+ 'url': 'arcpublishing:elcomercio:27a7e1f8-2ec7-4177-874f-a4feed2885b3',
+ 'only_matching': True,
+ }, {
+ # https://www.clickondetroit.com/video/community/2020/05/15/events-surrounding-woodward-dream-cruise-being-canceled/
+ 'url': 'arcpublishing:gmg:c8793fb2-8d44-4242-881e-2db31da2d9fe',
+ 'only_matching': True,
+ }, {
+ # https://www.wabi.tv/video/2020/12/30/trenton-company-making-equipment-pfizer-covid-vaccine/
+ 'url': 'arcpublishing:gray:0b0ba30e-032a-4598-8810-901d70e6033e',
+ 'only_matching': True,
+ }, {
+ # https://www.lateja.cr/el-mundo/video-china-aprueba-con-condiciones-su-primera/dfcbfa57-527f-45ff-a69b-35fe71054143/video/
+ 'url': 'arcpublishing:gruponacion:dfcbfa57-527f-45ff-a69b-35fe71054143',
+ 'only_matching': True,
+ }, {
+ # https://www.fifthdomain.com/video/2018/03/09/is-america-vulnerable-to-a-cyber-attack/
+ 'url': 'arcpublishing:mco:aa0ca6fe-1127-46d4-b32c-be0d6fdb8055',
+ 'only_matching': True,
+ }, {
+ # https://www.vl.no/kultur/2020/12/09/en-melding-fra-en-lytter-endret-julelista-til-lewi-bergrud/
+ 'url': 'arcpublishing:mentormedier:47a12084-650b-4011-bfd0-3699b6947b2d',
+ 'only_matching': True,
+ }, {
+ # https://www.14news.com/2020/12/30/whiskey-theft-caught-camera-henderson-liquor-store/
+ 'url': 'arcpublishing:raycom:b89f61f8-79fa-4c09-8255-e64237119bf7',
+ 'only_matching': True,
+ }, {
+ # https://www.theglobeandmail.com/world/video-ethiopian-woman-who-became-symbol-of-integration-in-italy-killed-on/
+ 'url': 'arcpublishing:tgam:411b34c1-8701-4036-9831-26964711664b',
+ 'only_matching': True,
+ }, {
+ # https://www.pilotonline.com/460f2931-8130-4719-8ea1-ffcb2d7cb685-132.html
+ 'url': 'arcpublishing:tronc:460f2931-8130-4719-8ea1-ffcb2d7cb685',
+ 'only_matching': True,
+ }]
+ _POWA_DEFAULTS = [
+ (['cmg', 'prisa'], '%s-config-prod.api.cdn.arcpublishing.com/video'),
+ ([
+ 'adn', 'advancelocal', 'answers', 'bonnier', 'bostonglobe', 'demo',
+ 'gmg', 'gruponacion', 'infobae', 'mco', 'nzme', 'pmn', 'raycom',
+ 'spectator', 'tbt', 'tgam', 'tronc', 'wapo', 'wweek',
+ ], 'video-api-cdn.%s.arcpublishing.com/api'),
+ ]
+
+ @staticmethod
+ def _extract_urls(webpage):
+ entries = []
+ # https://arcpublishing.atlassian.net/wiki/spaces/POWA/overview
+ for powa_el in re.findall(r'(]+class="[^"]*\bpowa\b[^"]*"[^>]+data-uuid="%s"[^>]*>)' % ArcPublishingIE._UUID_REGEX, webpage):
+ powa = extract_attributes(powa_el) or {}
+ org = powa.get('data-org')
+ uuid = powa.get('data-uuid')
+ if org and uuid:
+ entries.append('arcpublishing:%s:%s' % (org, uuid))
+ return entries
+
+ def _real_extract(self, url):
+ org, uuid = re.match(self._VALID_URL, url).groups()
+ for orgs, tmpl in self._POWA_DEFAULTS:
+ if org in orgs:
+ base_api_tmpl = tmpl
+ break
+ else:
+ base_api_tmpl = '%s-prod-cdn.video-api.arcpublishing.com/api'
+ if org == 'wapo':
+ org = 'washpost'
+ video = self._download_json(
+ 'https://%s/v1/ansvideos/findByUuid' % (base_api_tmpl % org),
+ uuid, query={'uuid': uuid})[0]
+ title = video['headlines']['basic']
+ is_live = video.get('status') == 'live'
+
+ urls = []
+ formats = []
+ for s in video.get('streams', []):
+ s_url = s.get('url')
+ if not s_url or s_url in urls:
+ continue
+ urls.append(s_url)
+ stream_type = s.get('stream_type')
+ if stream_type == 'smil':
+ smil_formats = self._extract_smil_formats(
+ s_url, uuid, fatal=False)
+ for f in smil_formats:
+ if f['url'].endswith('/cfx/st'):
+ f['app'] = 'cfx/st'
+ if not f['play_path'].startswith('mp4:'):
+ f['play_path'] = 'mp4:' + f['play_path']
+ if isinstance(f['tbr'], float):
+ f['vbr'] = f['tbr'] * 1000
+ del f['tbr']
+ f['format_id'] = 'rtmp-%d' % f['vbr']
+ formats.extend(smil_formats)
+ elif stream_type in ('ts', 'hls'):
+ m3u8_formats = self._extract_m3u8_formats(
+ s_url, uuid, 'mp4', 'm3u8' if is_live else 'm3u8_native',
+ m3u8_id='hls', fatal=False)
+ if all([f.get('acodec') == 'none' for f in m3u8_formats]):
+ continue
+ for f in m3u8_formats:
+ if f.get('acodec') == 'none':
+ f['preference'] = -40
+ elif f.get('vcodec') == 'none':
+ f['preference'] = -50
+ height = f.get('height')
+ if not height:
+ continue
+ vbr = self._search_regex(
+ r'[_x]%d[_-](\d+)' % height, f['url'], 'vbr', default=None)
+ if vbr:
+ f['vbr'] = int(vbr)
+ formats.extend(m3u8_formats)
+ else:
+ vbr = int_or_none(s.get('bitrate'))
+ formats.append({
+ 'format_id': '%s-%d' % (stream_type, vbr) if vbr else stream_type,
+ 'vbr': vbr,
+ 'width': int_or_none(s.get('width')),
+ 'height': int_or_none(s.get('height')),
+ 'filesize': int_or_none(s.get('filesize')),
+ 'url': s_url,
+ 'preference': -1,
+ })
+ self._sort_formats(
+ formats, ('preference', 'width', 'height', 'vbr', 'filesize', 'tbr', 'ext', 'format_id'))
+
+ subtitles = {}
+ for subtitle in (try_get(video, lambda x: x['subtitles']['urls'], list) or []):
+ subtitle_url = subtitle.get('url')
+ if subtitle_url:
+ subtitles.setdefault('en', []).append({'url': subtitle_url})
+
+ return {
+ 'id': uuid,
+ 'title': self._live_title(title) if is_live else title,
+ 'thumbnail': try_get(video, lambda x: x['promo_image']['url']),
+ 'description': try_get(video, lambda x: x['subheadlines']['basic']),
+ 'formats': formats,
+ 'duration': int_or_none(video.get('duration'), 100),
+ 'timestamp': parse_iso8601(video.get('created_date')),
+ 'subtitles': subtitles,
+ 'is_live': is_live,
+ }
diff --git a/youtube_dlc/extractor/arkena.py b/youtube_dlc/extractor/arkena.py
index 854f58767..fd46b1c77 100644
--- a/youtube_dlc/extractor/arkena.py
+++ b/youtube_dlc/extractor/arkena.py
@@ -6,13 +6,11 @@
from .common import InfoExtractor
from ..compat import compat_urlparse
from ..utils import (
- determine_ext,
ExtractorError,
float_or_none,
int_or_none,
- mimetype2ext,
parse_iso8601,
- strip_jsonp,
+ try_get,
)
@@ -20,22 +18,27 @@ class ArkenaIE(InfoExtractor):
_VALID_URL = r'''(?x)
https?://
(?:
- video\.arkena\.com/play2/embed/player\?|
+ video\.(?:arkena|qbrick)\.com/play2/embed/player\?|
play\.arkena\.com/(?:config|embed)/avp/v\d/player/media/(?P[^/]+)/[^/]+/(?P\d+)
)
'''
_TESTS = [{
- 'url': 'https://play.arkena.com/embed/avp/v2/player/media/b41dda37-d8e7-4d3f-b1b5-9a9db578bdfe/1/129411',
- 'md5': 'b96f2f71b359a8ecd05ce4e1daa72365',
+ 'url': 'https://video.qbrick.com/play2/embed/player?accountId=1034090&mediaId=d8ab4607-00090107-aab86310',
+ 'md5': '97f117754e5f3c020f5f26da4a44ebaf',
'info_dict': {
- 'id': 'b41dda37-d8e7-4d3f-b1b5-9a9db578bdfe',
+ 'id': 'd8ab4607-00090107-aab86310',
'ext': 'mp4',
- 'title': 'Big Buck Bunny',
- 'description': 'Royalty free test video',
- 'timestamp': 1432816365,
- 'upload_date': '20150528',
- 'is_live': False,
+ 'title': 'EM_HT20_117_roslund_v2.mp4',
+ 'timestamp': 1608285912,
+ 'upload_date': '20201218',
+ 'duration': 1429.162667,
+ 'subtitles': {
+ 'sv': 'count:3',
+ },
},
+ }, {
+ 'url': 'https://play.arkena.com/embed/avp/v2/player/media/b41dda37-d8e7-4d3f-b1b5-9a9db578bdfe/1/129411',
+ 'only_matching': True,
}, {
'url': 'https://play.arkena.com/config/avp/v2/player/media/b41dda37-d8e7-4d3f-b1b5-9a9db578bdfe/1/129411/?callbackMethod=jQuery1111023664739129262213_1469227693893',
'only_matching': True,
@@ -72,62 +75,89 @@ def _real_extract(self, url):
if not video_id or not account_id:
raise ExtractorError('Invalid URL', expected=True)
- playlist = self._download_json(
- 'https://play.arkena.com/config/avp/v2/player/media/%s/0/%s/?callbackMethod=_'
- % (video_id, account_id),
- video_id, transform_source=strip_jsonp)['Playlist'][0]
-
- media_info = playlist['MediaInfo']
- title = media_info['Title']
- media_files = playlist['MediaFiles']
+ media = self._download_json(
+ 'https://video.qbrick.com/api/v1/public/accounts/%s/medias/%s' % (account_id, video_id),
+ video_id, query={
+ # https://video.qbrick.com/docs/api/examples/library-api.html
+ 'fields': 'asset/resources/*/renditions/*(height,id,language,links/*(href,mimeType),type,size,videos/*(audios/*(codec,sampleRate),bitrate,codec,duration,height,width),width),created,metadata/*(title,description),tags',
+ })
+ metadata = media.get('metadata') or {}
+ title = metadata['title']
- is_live = False
+ duration = None
formats = []
- for kind_case, kind_formats in media_files.items():
- kind = kind_case.lower()
- for f in kind_formats:
- f_url = f.get('Url')
- if not f_url:
- continue
- is_live = f.get('Live') == 'true'
- exts = (mimetype2ext(f.get('Type')), determine_ext(f_url, None))
- if kind == 'm3u8' or 'm3u8' in exts:
- formats.extend(self._extract_m3u8_formats(
- f_url, video_id, 'mp4', 'm3u8_native',
- m3u8_id=kind, fatal=False, live=is_live))
- elif kind == 'flash' or 'f4m' in exts:
- formats.extend(self._extract_f4m_formats(
- f_url, video_id, f4m_id=kind, fatal=False))
- elif kind == 'dash' or 'mpd' in exts:
- formats.extend(self._extract_mpd_formats(
- f_url, video_id, mpd_id=kind, fatal=False))
- elif kind == 'silverlight':
- # TODO: process when ism is supported (see
- # https://github.com/ytdl-org/youtube-dl/issues/8118)
- continue
- else:
- tbr = float_or_none(f.get('Bitrate'), 1000)
- formats.append({
- 'url': f_url,
- 'format_id': '%s-%d' % (kind, tbr) if tbr else kind,
- 'tbr': tbr,
- })
+ thumbnails = []
+ subtitles = {}
+ for resource in media['asset']['resources']:
+ for rendition in (resource.get('renditions') or []):
+ rendition_type = rendition.get('type')
+ for i, link in enumerate(rendition.get('links') or []):
+ href = link.get('href')
+ if not href:
+ continue
+ if rendition_type == 'image':
+ thumbnails.append({
+ 'filesize': int_or_none(rendition.get('size')),
+ 'height': int_or_none(rendition.get('height')),
+ 'id': rendition.get('id'),
+ 'url': href,
+ 'width': int_or_none(rendition.get('width')),
+ })
+ elif rendition_type == 'subtitle':
+ subtitles.setdefault(rendition.get('language') or 'en', []).append({
+ 'url': href,
+ })
+ elif rendition_type == 'video':
+ f = {
+ 'filesize': int_or_none(rendition.get('size')),
+ 'format_id': rendition.get('id'),
+ 'url': href,
+ }
+ video = try_get(rendition, lambda x: x['videos'][i], dict)
+ if video:
+ if not duration:
+ duration = float_or_none(video.get('duration'))
+ f.update({
+ 'height': int_or_none(video.get('height')),
+ 'tbr': int_or_none(video.get('bitrate'), 1000),
+ 'vcodec': video.get('codec'),
+ 'width': int_or_none(video.get('width')),
+ })
+ audio = try_get(video, lambda x: x['audios'][0], dict)
+ if audio:
+ f.update({
+ 'acodec': audio.get('codec'),
+ 'asr': int_or_none(audio.get('sampleRate')),
+ })
+ formats.append(f)
+ elif rendition_type == 'index':
+ mime_type = link.get('mimeType')
+ if mime_type == 'application/smil+xml':
+ formats.extend(self._extract_smil_formats(
+ href, video_id, fatal=False))
+ elif mime_type == 'application/x-mpegURL':
+ formats.extend(self._extract_m3u8_formats(
+ href, video_id, 'mp4', 'm3u8_native',
+ m3u8_id='hls', fatal=False))
+ elif mime_type == 'application/hds+xml':
+ formats.extend(self._extract_f4m_formats(
+ href, video_id, f4m_id='hds', fatal=False))
+ elif mime_type == 'application/dash+xml':
+ formats.extend(self._extract_f4m_formats(
+ href, video_id, f4m_id='hds', fatal=False))
+ elif mime_type == 'application/vnd.ms-sstr+xml':
+ formats.extend(self._extract_ism_formats(
+ href, video_id, ism_id='mss', fatal=False))
self._sort_formats(formats)
- description = media_info.get('Description')
- video_id = media_info.get('VideoId') or video_id
- timestamp = parse_iso8601(media_info.get('PublishDate'))
- thumbnails = [{
- 'url': thumbnail['Url'],
- 'width': int_or_none(thumbnail.get('Size')),
- } for thumbnail in (media_info.get('Poster') or []) if thumbnail.get('Url')]
-
return {
'id': video_id,
'title': title,
- 'description': description,
- 'timestamp': timestamp,
- 'is_live': is_live,
+ 'description': metadata.get('description'),
+ 'timestamp': parse_iso8601(media.get('created')),
'thumbnails': thumbnails,
+ 'subtitles': subtitles,
+ 'duration': duration,
+ 'tags': media.get('tags'),
'formats': formats,
}
diff --git a/youtube_dlc/extractor/asiancrush.py b/youtube_dlc/extractor/asiancrush.py
index 0348e680c..66ce7c686 100644
--- a/youtube_dlc/extractor/asiancrush.py
+++ b/youtube_dlc/extractor/asiancrush.py
@@ -1,27 +1,91 @@
# coding: utf-8
from __future__ import unicode_literals
+import functools
import re
from .common import InfoExtractor
from .kaltura import KalturaIE
-from ..utils import extract_attributes
+from ..utils import (
+ extract_attributes,
+ int_or_none,
+ OnDemandPagedList,
+ parse_age_limit,
+ strip_or_none,
+ try_get,
+)
+
+
+class AsianCrushBaseIE(InfoExtractor):
+ _VALID_URL_BASE = r'https?://(?:www\.)?(?P(?:(?:asiancrush|yuyutv|midnightpulp)\.com|(?:cocoro|retrocrush)\.tv))'
+ _KALTURA_KEYS = [
+ 'video_url', 'progressive_url', 'download_url', 'thumbnail_url',
+ 'widescreen_thumbnail_url', 'screencap_widescreen',
+ ]
+ _API_SUFFIX = {'retrocrush.tv': '-ott'}
+
+ def _call_api(self, host, endpoint, video_id, query, resource):
+ return self._download_json(
+ 'https://api%s.%s/%s' % (self._API_SUFFIX.get(host, ''), host, endpoint), video_id,
+ 'Downloading %s JSON metadata' % resource, query=query,
+ headers=self.geo_verification_headers())['objects']
+
+ def _download_object_data(self, host, object_id, resource):
+ return self._call_api(
+ host, 'search', object_id, {'id': object_id}, resource)[0]
+
+ def _get_object_description(self, obj):
+ return strip_or_none(obj.get('long_description') or obj.get('short_description'))
+
+ def _parse_video_data(self, video):
+ title = video['name']
+
+ entry_id, partner_id = [None] * 2
+ for k in self._KALTURA_KEYS:
+ k_url = video.get(k)
+ if k_url:
+ mobj = re.search(r'/p/(\d+)/.+?/entryId/([^/]+)/', k_url)
+ if mobj:
+ partner_id, entry_id = mobj.groups()
+ break
+
+ meta_categories = try_get(video, lambda x: x['meta']['categories'], list) or []
+ categories = list(filter(None, [c.get('name') for c in meta_categories]))
+
+ show_info = video.get('show_info') or {}
+
+ return {
+ '_type': 'url_transparent',
+ 'url': 'kaltura:%s:%s' % (partner_id, entry_id),
+ 'ie_key': KalturaIE.ie_key(),
+ 'id': entry_id,
+ 'title': title,
+ 'description': self._get_object_description(video),
+ 'age_limit': parse_age_limit(video.get('mpaa_rating') or video.get('tv_rating')),
+ 'categories': categories,
+ 'series': show_info.get('show_name'),
+ 'season_number': int_or_none(show_info.get('season_num')),
+ 'season_id': show_info.get('season_id'),
+ 'episode_number': int_or_none(show_info.get('episode_num')),
+ }
-class AsianCrushIE(InfoExtractor):
- _VALID_URL_BASE = r'https?://(?:www\.)?(?P(?:(?:asiancrush|yuyutv|midnightpulp)\.com|cocoro\.tv))'
- _VALID_URL = r'%s/video/(?:[^/]+/)?0+(?P\d+)v\b' % _VALID_URL_BASE
+class AsianCrushIE(AsianCrushBaseIE):
+ _VALID_URL = r'%s/video/(?:[^/]+/)?0+(?P\d+)v\b' % AsianCrushBaseIE._VALID_URL_BASE
_TESTS = [{
- 'url': 'https://www.asiancrush.com/video/012869v/women-who-flirt/',
+ 'url': 'https://www.asiancrush.com/video/004289v/women-who-flirt',
'md5': 'c3b740e48d0ba002a42c0b72857beae6',
'info_dict': {
'id': '1_y4tmjm5r',
'ext': 'mp4',
'title': 'Women Who Flirt',
- 'description': 'md5:7e986615808bcfb11756eb503a751487',
+ 'description': 'md5:b65c7e0ae03a85585476a62a186f924c',
'timestamp': 1496936429,
'upload_date': '20170608',
'uploader_id': 'craig@crifkin.com',
+ 'age_limit': 13,
+ 'categories': 'count:5',
+ 'duration': 5812,
},
}, {
'url': 'https://www.asiancrush.com/video/she-was-pretty/011886v-pretty-episode-3/',
@@ -41,67 +105,35 @@ class AsianCrushIE(InfoExtractor):
}, {
'url': 'https://www.cocoro.tv/video/the-wonderful-wizard-of-oz/008878v-the-wonderful-wizard-of-oz-ep01/',
'only_matching': True,
+ }, {
+ 'url': 'https://www.retrocrush.tv/video/true-tears/012328v-i...gave-away-my-tears',
+ 'only_matching': True,
}]
def _real_extract(self, url):
- mobj = re.match(self._VALID_URL, url)
- host = mobj.group('host')
- video_id = mobj.group('id')
-
- webpage = self._download_webpage(url, video_id)
+ host, video_id = re.match(self._VALID_URL, url).groups()
- entry_id, partner_id, title = [None] * 3
-
- vars = self._parse_json(
- self._search_regex(
+ if host == 'cocoro.tv':
+ webpage = self._download_webpage(url, video_id)
+ embed_vars = self._parse_json(self._search_regex(
r'iEmbedVars\s*=\s*({.+?})', webpage, 'embed vars',
- default='{}'), video_id, fatal=False)
- if vars:
- entry_id = vars.get('entry_id')
- partner_id = vars.get('partner_id')
- title = vars.get('vid_label')
-
- if not entry_id:
- entry_id = self._search_regex(
- r'\bentry_id["\']\s*:\s*["\'](\d+)', webpage, 'entry id')
+ default='{}'), video_id, fatal=False) or {}
+ video_id = embed_vars.get('entry_id') or video_id
- player = self._download_webpage(
- 'https://api.%s/embeddedVideoPlayer' % host, video_id,
- query={'id': entry_id})
+ video = self._download_object_data(host, video_id, 'video')
+ return self._parse_video_data(video)
- kaltura_id = self._search_regex(
- r'entry_id["\']\s*:\s*(["\'])(?P(?:(?!\1).)+)\1', player,
- 'kaltura id', group='id')
- if not partner_id:
- partner_id = self._search_regex(
- r'/p(?:artner_id)?/(\d+)', player, 'partner id',
- default='513551')
-
- description = self._html_search_regex(
- r'(?s)]+\bclass=["\']description["\'][^>]*>(.+?)',
- webpage, 'description', fatal=False)
-
- return {
- '_type': 'url_transparent',
- 'url': 'kaltura:%s:%s' % (partner_id, kaltura_id),
- 'ie_key': KalturaIE.ie_key(),
- 'id': video_id,
- 'title': title,
- 'description': description,
- }
-
-
-class AsianCrushPlaylistIE(InfoExtractor):
- _VALID_URL = r'%s/series/0+(?P\d+)s\b' % AsianCrushIE._VALID_URL_BASE
+class AsianCrushPlaylistIE(AsianCrushBaseIE):
+ _VALID_URL = r'%s/series/0+(?P\d+)s\b' % AsianCrushBaseIE._VALID_URL_BASE
_TESTS = [{
- 'url': 'https://www.asiancrush.com/series/012481s/scholar-walks-night/',
+ 'url': 'https://www.asiancrush.com/series/006447s/fruity-samurai',
'info_dict': {
- 'id': '12481',
- 'title': 'Scholar Who Walks the Night',
- 'description': 'md5:7addd7c5132a09fd4741152d96cce886',
+ 'id': '6447',
+ 'title': 'Fruity Samurai',
+ 'description': 'md5:7535174487e4a202d3872a7fc8f2f154',
},
- 'playlist_count': 20,
+ 'playlist_count': 13,
}, {
'url': 'https://www.yuyutv.com/series/013920s/peep-show/',
'only_matching': True,
@@ -111,35 +143,58 @@ class AsianCrushPlaylistIE(InfoExtractor):
}, {
'url': 'https://www.cocoro.tv/series/008549s/the-wonderful-wizard-of-oz/',
'only_matching': True,
+ }, {
+ 'url': 'https://www.retrocrush.tv/series/012355s/true-tears',
+ 'only_matching': True,
}]
+ _PAGE_SIZE = 1000000000
+
+ def _fetch_page(self, domain, parent_id, page):
+ videos = self._call_api(
+ domain, 'getreferencedobjects', parent_id, {
+ 'max': self._PAGE_SIZE,
+ 'object_type': 'video',
+ 'parent_id': parent_id,
+ 'start': page * self._PAGE_SIZE,
+ }, 'page %d' % (page + 1))
+ for video in videos:
+ yield self._parse_video_data(video)
def _real_extract(self, url):
- playlist_id = self._match_id(url)
-
- webpage = self._download_webpage(url, playlist_id)
-
- entries = []
-
- for mobj in re.finditer(
- r']+href=(["\'])(?P%s.*?)\1[^>]*>' % AsianCrushIE._VALID_URL,
- webpage):
- attrs = extract_attributes(mobj.group(0))
- if attrs.get('class') == 'clearfix':
- entries.append(self.url_result(
- mobj.group('url'), ie=AsianCrushIE.ie_key()))
-
- title = self._html_search_regex(
- r'(?s)]\bid=["\']movieTitle[^>]+>(.+?)', webpage,
- 'title', default=None) or self._og_search_title(
- webpage, default=None) or self._html_search_meta(
- 'twitter:title', webpage, 'title',
- default=None) or self._search_regex(
- r'([^<]+)', webpage, 'title', fatal=False)
- if title:
- title = re.sub(r'\s*\|\s*.+?$', '', title)
-
- description = self._og_search_description(
- webpage, default=None) or self._html_search_meta(
- 'twitter:description', webpage, 'description', fatal=False)
+ host, playlist_id = re.match(self._VALID_URL, url).groups()
+
+ if host == 'cocoro.tv':
+ webpage = self._download_webpage(url, playlist_id)
+
+ entries = []
+
+ for mobj in re.finditer(
+ r']+href=(["\'])(?P%s.*?)\1[^>]*>' % AsianCrushIE._VALID_URL,
+ webpage):
+ attrs = extract_attributes(mobj.group(0))
+ if attrs.get('class') == 'clearfix':
+ entries.append(self.url_result(
+ mobj.group('url'), ie=AsianCrushIE.ie_key()))
+
+ title = self._html_search_regex(
+ r'(?s)]\bid=["\']movieTitle[^>]+>(.+?)', webpage,
+ 'title', default=None) or self._og_search_title(
+ webpage, default=None) or self._html_search_meta(
+ 'twitter:title', webpage, 'title',
+ default=None) or self._search_regex(
+ r'([^<]+)', webpage, 'title', fatal=False)
+ if title:
+ title = re.sub(r'\s*\|\s*.+?$', '', title)
+
+ description = self._og_search_description(
+ webpage, default=None) or self._html_search_meta(
+ 'twitter:description', webpage, 'description', fatal=False)
+ else:
+ show = self._download_object_data(host, playlist_id, 'show')
+ title = show.get('name')
+ description = self._get_object_description(show)
+ entries = OnDemandPagedList(
+ functools.partial(self._fetch_page, host, playlist_id),
+ self._PAGE_SIZE)
return self.playlist_result(entries, playlist_id, title, description)
diff --git a/youtube_dlc/extractor/bbc.py b/youtube_dlc/extractor/bbc.py
index 54cbcdc8e..b4daee54e 100644
--- a/youtube_dlc/extractor/bbc.py
+++ b/youtube_dlc/extractor/bbc.py
@@ -49,22 +49,17 @@ class BBCCoUkIE(InfoExtractor):
_LOGIN_URL = 'https://account.bbc.com/signin'
_NETRC_MACHINE = 'bbc'
- _MEDIASELECTOR_URLS = [
+ _MEDIA_SELECTOR_URL_TEMPL = 'https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/%s/vpid/%s'
+ _MEDIA_SETS = [
# Provides HQ HLS streams with even better quality that pc mediaset but fails
# with geolocation in some cases when it's even not geo restricted at all (e.g.
# http://www.bbc.co.uk/programmes/b06bp7lf). Also may fail with selectionunavailable.
- 'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/iptv-all/vpid/%s',
- 'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/pc/vpid/%s',
+ 'iptv-all',
+ 'pc',
]
- _MEDIASELECTION_NS = 'http://bbc.co.uk/2008/mp/mediaselection'
_EMP_PLAYLIST_NS = 'http://bbc.co.uk/2008/emp/playlist'
- _NAMESPACES = (
- _MEDIASELECTION_NS,
- _EMP_PLAYLIST_NS,
- )
-
_TESTS = [
{
'url': 'http://www.bbc.co.uk/programmes/b039g8p7',
@@ -261,8 +256,6 @@ class BBCCoUkIE(InfoExtractor):
'only_matching': True,
}]
- _USP_RE = r'/([^/]+?)\.ism(?:\.hlsv2\.ism)?/[^/]+\.m3u8'
-
def _login(self):
username, password = self._get_login_info()
if username is None:
@@ -307,22 +300,14 @@ def _extract_asx_playlist(self, connection, programme_id):
def _extract_items(self, playlist):
return playlist.findall('./{%s}item' % self._EMP_PLAYLIST_NS)
- def _findall_ns(self, element, xpath):
- elements = []
- for ns in self._NAMESPACES:
- elements.extend(element.findall(xpath % ns))
- return elements
-
def _extract_medias(self, media_selection):
- error = media_selection.find('./{%s}error' % self._MEDIASELECTION_NS)
- if error is None:
- media_selection.find('./{%s}error' % self._EMP_PLAYLIST_NS)
- if error is not None:
- raise BBCCoUkIE.MediaSelectionError(error.get('id'))
- return self._findall_ns(media_selection, './{%s}media')
+ error = media_selection.get('result')
+ if error:
+ raise BBCCoUkIE.MediaSelectionError(error)
+ return media_selection.get('media') or []
def _extract_connections(self, media):
- return self._findall_ns(media, './{%s}connection')
+ return media.get('connection') or []
def _get_subtitles(self, media, programme_id):
subtitles = {}
@@ -334,13 +319,13 @@ def _get_subtitles(self, media, programme_id):
cc_url, programme_id, 'Downloading captions', fatal=False)
if not isinstance(captions, compat_etree_Element):
continue
- lang = captions.get('{http://www.w3.org/XML/1998/namespace}lang', 'en')
- subtitles[lang] = [
+ subtitles['en'] = [
{
'url': connection.get('href'),
'ext': 'ttml',
},
]
+ break
return subtitles
def _raise_extractor_error(self, media_selection_error):
@@ -350,10 +335,10 @@ def _raise_extractor_error(self, media_selection_error):
def _download_media_selector(self, programme_id):
last_exception = None
- for mediaselector_url in self._MEDIASELECTOR_URLS:
+ for media_set in self._MEDIA_SETS:
try:
return self._download_media_selector_url(
- mediaselector_url % programme_id, programme_id)
+ self._MEDIA_SELECTOR_URL_TEMPL % (media_set, programme_id), programme_id)
except BBCCoUkIE.MediaSelectionError as e:
if e.id in ('notukerror', 'geolocation', 'selectionunavailable'):
last_exception = e
@@ -362,8 +347,8 @@ def _download_media_selector(self, programme_id):
self._raise_extractor_error(last_exception)
def _download_media_selector_url(self, url, programme_id=None):
- media_selection = self._download_xml(
- url, programme_id, 'Downloading media selection XML',
+ media_selection = self._download_json(
+ url, programme_id, 'Downloading media selection JSON',
expected_status=(403, 404))
return self._process_media_selector(media_selection, programme_id)
@@ -377,7 +362,6 @@ def _process_media_selector(self, media_selection, programme_id):
if kind in ('video', 'audio'):
bitrate = int_or_none(media.get('bitrate'))
encoding = media.get('encoding')
- service = media.get('service')
width = int_or_none(media.get('width'))
height = int_or_none(media.get('height'))
file_size = int_or_none(media.get('media_file_size'))
@@ -392,8 +376,6 @@ def _process_media_selector(self, media_selection, programme_id):
supplier = connection.get('supplier')
transfer_format = connection.get('transferFormat')
format_id = supplier or conn_kind or protocol
- if service:
- format_id = '%s_%s' % (service, format_id)
# ASX playlist
if supplier == 'asx':
for i, ref in enumerate(self._extract_asx_playlist(connection, programme_id)):
@@ -408,20 +390,11 @@ def _process_media_selector(self, media_selection, programme_id):
formats.extend(self._extract_m3u8_formats(
href, programme_id, ext='mp4', entry_protocol='m3u8_native',
m3u8_id=format_id, fatal=False))
- if re.search(self._USP_RE, href):
- usp_formats = self._extract_m3u8_formats(
- re.sub(self._USP_RE, r'/\1.ism/\1.m3u8', href),
- programme_id, ext='mp4', entry_protocol='m3u8_native',
- m3u8_id=format_id, fatal=False)
- for f in usp_formats:
- if f.get('height') and f['height'] > 720:
- continue
- formats.append(f)
elif transfer_format == 'hds':
formats.extend(self._extract_f4m_formats(
href, programme_id, f4m_id=format_id, fatal=False))
else:
- if not service and not supplier and bitrate:
+ if not supplier and bitrate:
format_id += '-%d' % bitrate
fmt = {
'format_id': format_id,
@@ -554,7 +527,7 @@ def _real_extract(self, url):
webpage = self._download_webpage(url, group_id, 'Downloading video page')
error = self._search_regex(
- r']+\bclass=["\']smp__message delta["\'][^>]*>([^<]+)<',
+ r']+\bclass=["\'](?:smp|playout)__message delta["\'][^>]*>\s*([^<]+?)\s*<',
webpage, 'error', default=None)
if error:
raise ExtractorError(error, expected=True)
@@ -607,16 +580,9 @@ class BBCIE(BBCCoUkIE):
IE_DESC = 'BBC'
_VALID_URL = r'https?://(?:www\.)?bbc\.(?:com|co\.uk)/(?:[^/]+/)+(?P[^/#?]+)'
- _MEDIASELECTOR_URLS = [
- # Provides HQ HLS streams but fails with geolocation in some cases when it's
- # even not geo restricted at all
- 'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/iptv-all/vpid/%s',
- # Provides more formats, namely direct mp4 links, but fails on some videos with
- # notukerror for non UK (?) users (e.g.
- # http://www.bbc.com/travel/story/20150625-sri-lankas-spicy-secret)
- 'http://open.live.bbc.co.uk/mediaselector/4/mtis/stream/%s',
- # Provides fewer formats, but works everywhere for everybody (hopefully)
- 'http://open.live.bbc.co.uk/mediaselector/5/select/version/2.0/mediaset/journalism-pc/vpid/%s',
+ _MEDIA_SETS = [
+ 'mobile-tablet-main',
+ 'pc',
]
_TESTS = [{
diff --git a/youtube_dlc/extractor/beampro.py b/youtube_dlc/extractor/beampro.py
deleted file mode 100644
index 86abdae00..000000000
--- a/youtube_dlc/extractor/beampro.py
+++ /dev/null
@@ -1,194 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import (
- ExtractorError,
- clean_html,
- compat_str,
- float_or_none,
- int_or_none,
- parse_iso8601,
- try_get,
- urljoin,
-)
-
-
-class BeamProBaseIE(InfoExtractor):
- _API_BASE = 'https://mixer.com/api/v1'
- _RATINGS = {'family': 0, 'teen': 13, '18+': 18}
-
- def _extract_channel_info(self, chan):
- user_id = chan.get('userId') or try_get(chan, lambda x: x['user']['id'])
- return {
- 'uploader': chan.get('token') or try_get(
- chan, lambda x: x['user']['username'], compat_str),
- 'uploader_id': compat_str(user_id) if user_id else None,
- 'age_limit': self._RATINGS.get(chan.get('audience')),
- }
-
-
-class BeamProLiveIE(BeamProBaseIE):
- IE_NAME = 'Mixer:live'
- _VALID_URL = r'https?://(?:\w+\.)?(?:beam\.pro|mixer\.com)/(?P[^/?#&]+)'
- _TEST = {
- 'url': 'http://mixer.com/niterhayven',
- 'info_dict': {
- 'id': '261562',
- 'ext': 'mp4',
- 'title': 'Introducing The Witcher 3 // The Grind Starts Now!',
- 'description': 'md5:0b161ac080f15fe05d18a07adb44a74d',
- 'thumbnail': r're:https://.*\.jpg$',
- 'timestamp': 1483477281,
- 'upload_date': '20170103',
- 'uploader': 'niterhayven',
- 'uploader_id': '373396',
- 'age_limit': 18,
- 'is_live': True,
- 'view_count': int,
- },
- 'skip': 'niterhayven is offline',
- 'params': {
- 'skip_download': True,
- },
- }
-
- _MANIFEST_URL_TEMPLATE = '%s/channels/%%s/manifest.%%s' % BeamProBaseIE._API_BASE
-
- @classmethod
- def suitable(cls, url):
- return False if BeamProVodIE.suitable(url) else super(BeamProLiveIE, cls).suitable(url)
-
- def _real_extract(self, url):
- channel_name = self._match_id(url)
-
- chan = self._download_json(
- '%s/channels/%s' % (self._API_BASE, channel_name), channel_name)
-
- if chan.get('online') is False:
- raise ExtractorError(
- '{0} is offline'.format(channel_name), expected=True)
-
- channel_id = chan['id']
-
- def manifest_url(kind):
- return self._MANIFEST_URL_TEMPLATE % (channel_id, kind)
-
- formats = self._extract_m3u8_formats(
- manifest_url('m3u8'), channel_name, ext='mp4', m3u8_id='hls',
- fatal=False)
- formats.extend(self._extract_smil_formats(
- manifest_url('smil'), channel_name, fatal=False))
- self._sort_formats(formats)
-
- info = {
- 'id': compat_str(chan.get('id') or channel_name),
- 'title': self._live_title(chan.get('name') or channel_name),
- 'description': clean_html(chan.get('description')),
- 'thumbnail': try_get(
- chan, lambda x: x['thumbnail']['url'], compat_str),
- 'timestamp': parse_iso8601(chan.get('updatedAt')),
- 'is_live': True,
- 'view_count': int_or_none(chan.get('viewersTotal')),
- 'formats': formats,
- }
- info.update(self._extract_channel_info(chan))
-
- return info
-
-
-class BeamProVodIE(BeamProBaseIE):
- IE_NAME = 'Mixer:vod'
- _VALID_URL = r'https?://(?:\w+\.)?(?:beam\.pro|mixer\.com)/[^/?#&]+\?.*?\bvod=(?P[^?#&]+)'
- _TESTS = [{
- 'url': 'https://mixer.com/willow8714?vod=2259830',
- 'md5': 'b2431e6e8347dc92ebafb565d368b76b',
- 'info_dict': {
- 'id': '2259830',
- 'ext': 'mp4',
- 'title': 'willow8714\'s Channel',
- 'duration': 6828.15,
- 'thumbnail': r're:https://.*source\.png$',
- 'timestamp': 1494046474,
- 'upload_date': '20170506',
- 'uploader': 'willow8714',
- 'uploader_id': '6085379',
- 'age_limit': 13,
- 'view_count': int,
- },
- 'params': {
- 'skip_download': True,
- },
- }, {
- 'url': 'https://mixer.com/streamer?vod=IxFno1rqC0S_XJ1a2yGgNw',
- 'only_matching': True,
- }, {
- 'url': 'https://mixer.com/streamer?vod=Rh3LY0VAqkGpEQUe2pN-ig',
- 'only_matching': True,
- }]
-
- @staticmethod
- def _extract_format(vod, vod_type):
- if not vod.get('baseUrl'):
- return []
-
- if vod_type == 'hls':
- filename, protocol = 'manifest.m3u8', 'm3u8_native'
- elif vod_type == 'raw':
- filename, protocol = 'source.mp4', 'https'
- else:
- assert False
-
- data = vod.get('data') if isinstance(vod.get('data'), dict) else {}
-
- format_id = [vod_type]
- if isinstance(data.get('Height'), compat_str):
- format_id.append('%sp' % data['Height'])
-
- return [{
- 'url': urljoin(vod['baseUrl'], filename),
- 'format_id': '-'.join(format_id),
- 'ext': 'mp4',
- 'protocol': protocol,
- 'width': int_or_none(data.get('Width')),
- 'height': int_or_none(data.get('Height')),
- 'fps': int_or_none(data.get('Fps')),
- 'tbr': int_or_none(data.get('Bitrate'), 1000),
- }]
-
- def _real_extract(self, url):
- vod_id = self._match_id(url)
-
- vod_info = self._download_json(
- '%s/recordings/%s' % (self._API_BASE, vod_id), vod_id)
-
- state = vod_info.get('state')
- if state != 'AVAILABLE':
- raise ExtractorError(
- 'VOD %s is not available (state: %s)' % (vod_id, state),
- expected=True)
-
- formats = []
- thumbnail_url = None
-
- for vod in vod_info['vods']:
- vod_type = vod.get('format')
- if vod_type in ('hls', 'raw'):
- formats.extend(self._extract_format(vod, vod_type))
- elif vod_type == 'thumbnail':
- thumbnail_url = urljoin(vod.get('baseUrl'), 'source.png')
-
- self._sort_formats(formats)
-
- info = {
- 'id': vod_id,
- 'title': vod_info.get('name') or vod_id,
- 'duration': float_or_none(vod_info.get('duration')),
- 'thumbnail': thumbnail_url,
- 'timestamp': parse_iso8601(vod_info.get('createdAt')),
- 'view_count': int_or_none(vod_info.get('viewsTotal')),
- 'formats': formats,
- }
- info.update(self._extract_channel_info(vod_info.get('channel') or {}))
-
- return info
diff --git a/youtube_dlc/extractor/bfmtv.py b/youtube_dlc/extractor/bfmtv.py
new file mode 100644
index 000000000..501f69d80
--- /dev/null
+++ b/youtube_dlc/extractor/bfmtv.py
@@ -0,0 +1,103 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import extract_attributes
+
+
+class BFMTVBaseIE(InfoExtractor):
+ _VALID_URL_BASE = r'https?://(?:www\.)?bfmtv\.com/'
+ _VALID_URL_TMPL = _VALID_URL_BASE + r'(?:[^/]+/)*[^/?&#]+_%s[A-Z]-(?P\d{12})\.html'
+ _VIDEO_BLOCK_REGEX = r'(]+class="video_block"[^>]*>)'
+ BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_default/index.html?videoId=%s'
+
+ def _brightcove_url_result(self, video_id, video_block):
+ account_id = video_block.get('accountid') or '876450612001'
+ player_id = video_block.get('playerid') or 'I2qBTln4u'
+ return self.url_result(
+ self.BRIGHTCOVE_URL_TEMPLATE % (account_id, player_id, video_id),
+ 'BrightcoveNew', video_id)
+
+
+class BFMTVIE(BFMTVBaseIE):
+ IE_NAME = 'bfmtv'
+ _VALID_URL = BFMTVBaseIE._VALID_URL_TMPL % 'V'
+ _TESTS = [{
+ 'url': 'https://www.bfmtv.com/politique/emmanuel-macron-l-islam-est-une-religion-qui-vit-une-crise-aujourd-hui-partout-dans-le-monde_VN-202010020146.html',
+ 'info_dict': {
+ 'id': '6196747868001',
+ 'ext': 'mp4',
+ 'title': 'Emmanuel Macron: "L\'Islam est une religion qui vit une crise aujourd’hui, partout dans le monde"',
+ 'description': 'Le Président s\'exprime sur la question du séparatisme depuis les Mureaux, dans les Yvelines.',
+ 'uploader_id': '876450610001',
+ 'upload_date': '20201002',
+ 'timestamp': 1601629620,
+ },
+ }]
+
+ def _real_extract(self, url):
+ bfmtv_id = self._match_id(url)
+ webpage = self._download_webpage(url, bfmtv_id)
+ video_block = extract_attributes(self._search_regex(
+ self._VIDEO_BLOCK_REGEX, webpage, 'video block'))
+ return self._brightcove_url_result(video_block['videoid'], video_block)
+
+
+class BFMTVLiveIE(BFMTVIE):
+ IE_NAME = 'bfmtv:live'
+ _VALID_URL = BFMTVBaseIE._VALID_URL_BASE + '(?P(?:[^/]+/)?en-direct)'
+ _TESTS = [{
+ 'url': 'https://www.bfmtv.com/en-direct/',
+ 'info_dict': {
+ 'id': '5615950982001',
+ 'ext': 'mp4',
+ 'title': r're:^le direct BFMTV WEB \d{4}-\d{2}-\d{2} \d{2}:\d{2}$',
+ 'uploader_id': '876450610001',
+ 'upload_date': '20171018',
+ 'timestamp': 1508329950,
+ },
+ 'params': {
+ 'skip_download': True,
+ },
+ }, {
+ 'url': 'https://www.bfmtv.com/economie/en-direct/',
+ 'only_matching': True,
+ }]
+
+
+class BFMTVArticleIE(BFMTVBaseIE):
+ IE_NAME = 'bfmtv:article'
+ _VALID_URL = BFMTVBaseIE._VALID_URL_TMPL % 'A'
+ _TESTS = [{
+ 'url': 'https://www.bfmtv.com/sante/covid-19-un-responsable-de-l-institut-pasteur-se-demande-quand-la-france-va-se-reconfiner_AV-202101060198.html',
+ 'info_dict': {
+ 'id': '202101060198',
+ 'title': 'Covid-19: un responsable de l\'Institut Pasteur se demande "quand la France va se reconfiner"',
+ 'description': 'md5:947974089c303d3ac6196670ae262843',
+ },
+ 'playlist_count': 2,
+ }, {
+ 'url': 'https://www.bfmtv.com/international/pour-bolsonaro-le-bresil-est-en-faillite-mais-il-ne-peut-rien-faire_AD-202101060232.html',
+ 'only_matching': True,
+ }, {
+ 'url': 'https://www.bfmtv.com/sante/covid-19-oui-le-vaccin-de-pfizer-distribue-en-france-a-bien-ete-teste-sur-des-personnes-agees_AN-202101060275.html',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ bfmtv_id = self._match_id(url)
+ webpage = self._download_webpage(url, bfmtv_id)
+
+ entries = []
+ for video_block_el in re.findall(self._VIDEO_BLOCK_REGEX, webpage):
+ video_block = extract_attributes(video_block_el)
+ video_id = video_block.get('videoid')
+ if not video_id:
+ continue
+ entries.append(self._brightcove_url_result(video_id, video_block))
+
+ return self.playlist_result(
+ entries, bfmtv_id, self._og_search_title(webpage, fatal=False),
+ self._html_search_meta(['og:description', 'description'], webpage))
diff --git a/youtube_dlc/extractor/bibeltv.py b/youtube_dlc/extractor/bibeltv.py
new file mode 100644
index 000000000..56c2bfee8
--- /dev/null
+++ b/youtube_dlc/extractor/bibeltv.py
@@ -0,0 +1,30 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor
+
+
+class BibelTVIE(InfoExtractor):
+ _VALID_URL = r'https?://(?:www\.)?bibeltv\.de/mediathek/videos/(?:crn/)?(?P\d+)'
+ _TESTS = [{
+ 'url': 'https://www.bibeltv.de/mediathek/videos/329703-sprachkurs-in-malaiisch',
+ 'md5': '252f908192d611de038b8504b08bf97f',
+ 'info_dict': {
+ 'id': 'ref:329703',
+ 'ext': 'mp4',
+ 'title': 'Sprachkurs in Malaiisch',
+ 'description': 'md5:3e9f197d29ee164714e67351cf737dfe',
+ 'timestamp': 1608316701,
+ 'uploader_id': '5840105145001',
+ 'upload_date': '20201218',
+ }
+ }, {
+ 'url': 'https://www.bibeltv.de/mediathek/videos/crn/326374',
+ 'only_matching': True,
+ }]
+ BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/5840105145001/default_default/index.html?videoId=ref:%s'
+
+ def _real_extract(self, url):
+ crn_id = self._match_id(url)
+ return self.url_result(
+ self.BRIGHTCOVE_URL_TEMPLATE % crn_id, 'BrightcoveNew')
diff --git a/youtube_dlc/extractor/bongacams.py b/youtube_dlc/extractor/bongacams.py
new file mode 100644
index 000000000..180542fbc
--- /dev/null
+++ b/youtube_dlc/extractor/bongacams.py
@@ -0,0 +1,60 @@
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import compat_str
+from ..utils import (
+ int_or_none,
+ try_get,
+ urlencode_postdata,
+)
+
+
+class BongaCamsIE(InfoExtractor):
+ _VALID_URL = r'https?://(?P(?:[^/]+\.)?bongacams\d*\.com)/(?P[^/?&#]+)'
+ _TESTS = [{
+ 'url': 'https://de.bongacams.com/azumi-8',
+ 'only_matching': True,
+ }, {
+ 'url': 'https://cn.bongacams.com/azumi-8',
+ 'only_matching': True,
+ }]
+
+ def _real_extract(self, url):
+ mobj = re.match(self._VALID_URL, url)
+ host = mobj.group('host')
+ channel_id = mobj.group('id')
+
+ amf = self._download_json(
+ 'https://%s/tools/amf.php' % host, channel_id,
+ data=urlencode_postdata((
+ ('method', 'getRoomData'),
+ ('args[]', channel_id),
+ ('args[]', 'false'),
+ )), headers={'X-Requested-With': 'XMLHttpRequest'})
+
+ server_url = amf['localData']['videoServerUrl']
+
+ uploader_id = try_get(
+ amf, lambda x: x['performerData']['username'], compat_str) or channel_id
+ uploader = try_get(
+ amf, lambda x: x['performerData']['displayName'], compat_str)
+ like_count = int_or_none(try_get(
+ amf, lambda x: x['performerData']['loversCount']))
+
+ formats = self._extract_m3u8_formats(
+ '%s/hls/stream_%s/playlist.m3u8' % (server_url, uploader_id),
+ channel_id, 'mp4', m3u8_id='hls', live=True)
+ self._sort_formats(formats)
+
+ return {
+ 'id': channel_id,
+ 'title': self._live_title(uploader or uploader_id),
+ 'uploader': uploader,
+ 'uploader_id': uploader_id,
+ 'like_count': like_count,
+ 'age_limit': 18,
+ 'is_live': True,
+ 'formats': formats,
+ }
diff --git a/youtube_dlc/extractor/brightcove.py b/youtube_dlc/extractor/brightcove.py
index c6ca939dd..6022076ac 100644
--- a/youtube_dlc/extractor/brightcove.py
+++ b/youtube_dlc/extractor/brightcove.py
@@ -28,6 +28,7 @@
parse_iso8601,
smuggle_url,
str_or_none,
+ try_get,
unescapeHTML,
unsmuggle_url,
UnsupportedError,
@@ -470,18 +471,18 @@ def _extract_urls(ie, webpage):
def _parse_brightcove_metadata(self, json_data, video_id, headers={}):
title = json_data['name'].strip()
+ num_drm_sources = 0
formats = []
- sources_num = len(json_data.get('sources'))
- key_systems_present = 0
- for source in json_data.get('sources', []):
+ sources = json_data.get('sources') or []
+ for source in sources:
container = source.get('container')
ext = mimetype2ext(source.get('type'))
src = source.get('src')
- # https://apis.support.brightcove.com/playback/references/playback-api-video-fields-reference.html
- if source.get('key_systems'):
- key_systems_present += 1
+ # https://support.brightcove.com/playback-api-video-fields-reference#key_systems_object
+ if container == 'WVM' or source.get('key_systems'):
+ num_drm_sources += 1
continue
- elif ext == 'ism' or container == 'WVM':
+ elif ext == 'ism':
continue
elif ext == 'm3u8' or container == 'M2TS':
if not src:
@@ -539,23 +540,14 @@ def build_format_id(kind):
})
formats.append(f)
- if sources_num == key_systems_present:
- raise ExtractorError('This video is DRM protected', expected=True)
-
if not formats:
- # for sonyliv.com DRM protected videos
- s3_source_url = json_data.get('custom_fields', {}).get('s3sourceurl')
- if s3_source_url:
- formats.append({
- 'url': s3_source_url,
- 'format_id': 'source',
- })
-
- errors = json_data.get('errors')
- if not formats and errors:
- error = errors[0]
- raise ExtractorError(
- error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
+ errors = json_data.get('errors')
+ if errors:
+ error = errors[0]
+ raise ExtractorError(
+ error.get('message') or error.get('error_subcode') or error['error_code'], expected=True)
+ if sources and num_drm_sources == len(sources):
+ raise ExtractorError('This video is DRM protected.', expected=True)
self._sort_formats(formats)
@@ -609,24 +601,27 @@ def _real_extract(self, url):
store_pk = lambda x: self._downloader.cache.store('brightcove', policy_key_id, x)
def extract_policy_key():
- webpage = self._download_webpage(
- 'http://players.brightcove.net/%s/%s_%s/index.min.js'
- % (account_id, player_id, embed), video_id)
-
- policy_key = None
+ base_url = 'http://players.brightcove.net/%s/%s_%s/' % (account_id, player_id, embed)
+ config = self._download_json(
+ base_url + 'config.json', video_id, fatal=False) or {}
+ policy_key = try_get(
+ config, lambda x: x['video_cloud']['policy_key'])
+ if not policy_key:
+ webpage = self._download_webpage(
+ base_url + 'index.min.js', video_id)
- catalog = self._search_regex(
- r'catalog\(({.+?})\);', webpage, 'catalog', default=None)
- if catalog:
- catalog = self._parse_json(
- js_to_json(catalog), video_id, fatal=False)
+ catalog = self._search_regex(
+ r'catalog\(({.+?})\);', webpage, 'catalog', default=None)
if catalog:
- policy_key = catalog.get('policyKey')
-
- if not policy_key:
- policy_key = self._search_regex(
- r'policyKey\s*:\s*(["\'])(?P.+?)\1',
- webpage, 'policy key', group='pk')
+ catalog = self._parse_json(
+ js_to_json(catalog), video_id, fatal=False)
+ if catalog:
+ policy_key = catalog.get('policyKey')
+
+ if not policy_key:
+ policy_key = self._search_regex(
+ r'policyKey\s*:\s*(["\'])(?P.+?)\1',
+ webpage, 'policy key', group='pk')
store_pk(policy_key)
return policy_key
diff --git a/youtube_dlc/extractor/canvas.py b/youtube_dlc/extractor/canvas.py
index 8667a0d04..8b76a0200 100644
--- a/youtube_dlc/extractor/canvas.py
+++ b/youtube_dlc/extractor/canvas.py
@@ -7,12 +7,12 @@
from .gigya import GigyaBaseIE
from ..compat import compat_HTTPError
from ..utils import (
+ extract_attributes,
ExtractorError,
strip_or_none,
float_or_none,
int_or_none,
merge_dicts,
- parse_iso8601,
str_or_none,
url_or_none,
)
@@ -37,6 +37,7 @@ class CanvasIE(InfoExtractor):
'url': 'https://mediazone.vrt.be/api/v1/canvas/assets/mz-ast-5e5f90b6-2d72-4c40-82c2-e134f884e93e',
'only_matching': True,
}]
+ _GEO_BYPASS = False
_HLS_ENTRY_PROTOCOLS_MAP = {
'HLS': 'm3u8_native',
'HLS_AES': 'm3u8',
@@ -47,29 +48,34 @@ def _real_extract(self, url):
mobj = re.match(self._VALID_URL, url)
site_id, video_id = mobj.group('site_id'), mobj.group('id')
- # Old API endpoint, serves more formats but may fail for some videos
- data = self._download_json(
- 'https://mediazone.vrt.be/api/v1/%s/assets/%s'
- % (site_id, video_id), video_id, 'Downloading asset JSON',
- 'Unable to download asset JSON', fatal=False)
+ data = None
+ if site_id != 'vrtvideo':
+ # Old API endpoint, serves more formats but may fail for some videos
+ data = self._download_json(
+ 'https://mediazone.vrt.be/api/v1/%s/assets/%s'
+ % (site_id, video_id), video_id, 'Downloading asset JSON',
+ 'Unable to download asset JSON', fatal=False)
# New API endpoint
if not data:
+ headers = self.geo_verification_headers()
+ headers.update({'Content-Type': 'application/json'})
token = self._download_json(
'%s/tokens' % self._REST_API_BASE, video_id,
- 'Downloading token', data=b'',
- headers={'Content-Type': 'application/json'})['vrtPlayerToken']
+ 'Downloading token', data=b'', headers=headers)['vrtPlayerToken']
data = self._download_json(
'%s/videos/%s' % (self._REST_API_BASE, video_id),
- video_id, 'Downloading video JSON', fatal=False, query={
+ video_id, 'Downloading video JSON', query={
'vrtPlayerToken': token,
'client': '%s@PROD' % site_id,
}, expected_status=400)
- message = data.get('message')
- if message and not data.get('title'):
- if data.get('code') == 'AUTHENTICATION_REQUIRED':
- self.raise_login_required(message)
- raise ExtractorError(message, expected=True)
+ if not data.get('title'):
+ code = data.get('code')
+ if code == 'AUTHENTICATION_REQUIRED':
+ self.raise_login_required()
+ elif code == 'INVALID_LOCATION':
+ self.raise_geo_restricted(countries=['BE'])
+ raise ExtractorError(data.get('message') or code, expected=True)
title = data['title']
description = data.get('description')
@@ -205,20 +211,24 @@ def _real_extract(self, url):
class VrtNUIE(GigyaBaseIE):
IE_DESC = 'VrtNU.be'
- _VALID_URL = r'https?://(?:www\.)?vrt\.be/(?Pvrtnu)/(?:[^/]+/)*(?P[^/?#&]+)'
+ _VALID_URL = r'https?://(?:www\.)?vrt\.be/vrtnu/a-z/(?:[^/]+/){2}(?P[^/?#&]+)'
_TESTS = [{
# Available via old API endpoint
- 'url': 'https://www.vrt.be/vrtnu/a-z/postbus-x/1/postbus-x-s1a1/',
+ 'url': 'https://www.vrt.be/vrtnu/a-z/postbus-x/1989/postbus-x-s1989a1/',
'info_dict': {
- 'id': 'pbs-pub-2e2d8c27-df26-45c9-9dc6-90c78153044d$vid-90c932b1-e21d-4fb8-99b1-db7b49cf74de',
+ 'id': 'pbs-pub-e8713dac-899e-41de-9313-81269f4c04ac$vid-90c932b1-e21d-4fb8-99b1-db7b49cf74de',
'ext': 'mp4',
- 'title': 'De zwarte weduwe',
- 'description': 'md5:db1227b0f318c849ba5eab1fef895ee4',
+ 'title': 'Postbus X - Aflevering 1 (Seizoen 1989)',
+ 'description': 'md5:b704f669eb9262da4c55b33d7c6ed4b7',
'duration': 1457.04,
'thumbnail': r're:^https?://.*\.jpg$',
- 'season': 'Season 1',
- 'season_number': 1,
+ 'series': 'Postbus X',
+ 'season': 'Seizoen 1989',
+ 'season_number': 1989,
+ 'episode': 'De zwarte weduwe',
'episode_number': 1,
+ 'timestamp': 1595822400,
+ 'upload_date': '20200727',
},
'skip': 'This video is only available for registered users',
'params': {
@@ -300,69 +310,25 @@ def _login(self):
def _real_extract(self, url):
display_id = self._match_id(url)
- webpage, urlh = self._download_webpage_handle(url, display_id)
-
- info = self._search_json_ld(webpage, display_id, default={})
-
- # title is optional here since it may be extracted by extractor
- # that is delegated from here
- title = strip_or_none(self._html_search_regex(
- r'(?ms)(.+?)
',
- webpage, 'title', default=None))
-
- description = self._html_search_regex(
- r'(?ms)(.+?)
',
- webpage, 'description', default=None)
-
- season = self._html_search_regex(
- [r'''(?xms)\s*
- seizoen\ (.+?)\s*
- ''',
- r'