Wittytv.it broken after updating to version 2022.06.22.1

[Salvaju29ro]

Checklist

• I'm reporting a broken site
• I've verified that I'm running yt-dlp version 2022.06.22.1 (update instructions) or later (specify commit)
• I've checked that all provided URLs are playable in a browser with the same IP and same login details
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
• I've read the guidelines for opening an issue
• I've read about sharing account credentials and I'm willing to share it if required

Region

Italy

Description

Hi, after updating to version 2022.06.22.1 the wittytv.it site has stopped working, I tried to re-download the old version (2022.05.18) to verify that it was not a problem with the site, and it actually works

Verbose log

yt-dlp 2022.06.22.1:

.\yt-dlp.exe -vU https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/
[debug] Command-line config: ['-vU', 'https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.06.22.1 [a86e01e] (win32_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: avconv -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] Checking exe version: avprobe -bsfs
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.14.1, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.06.22.1, Current version: 2022.06.22.1
yt-dlp is up to date (2022.06.22.1)
[debug] [generic] Extracting URL: https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Requesting header
WARNING: [generic] Falling back on generic information extractor.
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Downloading webpage
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Extracting information
[debug] Looking for video embeds
[debug] Identified a JSON LD
[debug] [generic] Extracting URL: https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/#__youtubedl_smuggle=%7B%22force_videoid%22%3A+%22est-ce-que-tu-maimes-gabriele-5-dicembre-copia%22%2C+%22to_generic%22%3A+true%7D
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Requesting header
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Downloading webpage
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Extracting information
[debug] Looking for video embeds
ERROR: Unsupported URL: https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1427, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1497, in __extract_info
  File "yt_dlp\extractor\common.py", line 647, in extract
  File "yt_dlp\extractor\generic.py", line 4136, in _real_extract
yt_dlp.utils.UnsupportedError: Unsupported URL: https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/

Old Version yt-dlp 2022.05.18

.\yt-dlp.exe -v https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/
[debug] Command-line config: ['-v', 'https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.05.18 [b14d523] (win_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffprobe -bsfs
[debug] Checking exe version: avprobe -bsfs
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: avconv -bsfs
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.14.1, brotli-1.0.9, certifi-2021.10.08, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] [generic] Extracting URL: https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Requesting header
WARNING: [generic] Falling back on generic information extractor.
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Downloading webpage
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Extracting information
[debug] Looking for video embeds
[debug] Identified a twitter:player iframe
[debug] [generic] Extracting URL: https://www.video.mediaset.it/player/playerIFrameTwitter.shtml?id=580214&playrelated=false&autoplay=false&related=true&hidesocial=true
[generic] playerIFrameTwitter: Requesting header
[redirect] Following redirect to https://static3.mediasetplay.mediaset.it/player/index.html?appKey=5bd1ceefa0e845001aa4d83d&programGuid=FD00000000153323&id=580214&playrelated=false&autoplay=false&related=true&hidesocial=true
[debug] [Mediaset] Extracting URL: https://static3.mediasetplay.mediaset.it/player/index.html?appKey=5bd1ceefa0e845001aa4d83d&programGuid=FD00000000153323&id=580214&playrelated=false&autoplay=false&related=true&hidesocial=true
[Mediaset] FD00000000153323: Downloading JSON metadata
[Mediaset] FD00000000153323: Downloading MPEG4 SMIL data
[Mediaset] FD00000000153323: Checking video URL
[Mediaset] FD00000000153323: Downloading M3U SMIL data
[Mediaset] FD00000000153323: Downloading m3u8 information
[Mediaset] FD00000000153323: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[Mediaset] FD00000000153323: Downloading JSON metadata
[debug] Default format spec: best/bestvideo+bestaudio
[info] FD00000000153323: Downloading 1 format(s): http-1
[debug] Invoking http downloader on "https://vod08.msf.cdn.mediaset.net/farmunica/2020/07/632113_17352660b5c65b/17352660b5c65b-07_0.mp4"
[download] Est ce que tu m'aimes - Gabriele - 5 dicembre [FD00000000153323].mp4 has already been downloaded
[download] 100% of 11.25MiB

(In the second I removed the -U to prevent it from updating to the new version)

[pukkandan]

@dirkf Continuing from #4016 (comment),

So the problem page in #4141 had embedURL in a ld+json block, with an unsupported URL format, which was extracted in preference to the desired twitter:player format

yes

(or at least before it, if yt-dlp's generic is pulling all known URLs now)?

That was in #12 and was never merged

But wasn't that actually a problem with the Mediaset extractor which didn't like a URL format with a player version .../player/v2/index.html (ie, the URL pattern needed to have .../player(?:/v\d+)?/...?

There is no v2 in the mediaset URL

❯ yt-dlp https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/ -v
[debug] Command-line config: ['--ignore-config', 'https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/', '-v']
[debug] Encodings: locale cp65001, fs utf-8, pref cp65001, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.06.22.1 [a86e01e74] (source)
[debug] Lazy loading extractors is disabled
[debug] Plugins: ['SamplePluginIE', 'SamplePluginPP']
[debug] Git HEAD: 71086cfaa
[debug] Python version 3.10.4 (CPython 64bit) - Windows-10-10.0.22000-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-106550-g072101bd52-20220410 (fdk,setts), ffprobe N-106624-g391ce570c8-20220415, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.14.1, brotli-1.0.9, certifi-2021.10.08, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.1
[debug] Proxy map: {}
[debug] [generic] Extracting URL: https://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Requesting header
WARNING: [generic] Falling back on generic information extractor.
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Downloading webpage
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Extracting information
[debug] Looking for video embeds
[debug] Identified a twitter:player iframe
[debug] [generic] Extracting URL: https://www.video.mediaset.it/player/playerIFrameTwitter.shtml?id=580214&playrelated=false&autoplay=false&related=true&hidesocial=true
[generic] playerIFrameTwitter: Requesting header
[redirect] Following redirect to https://static3.mediasetplay.mediaset.it/player/index.html?appKey=5bd1ceefa0e845001aa4d83d&programGuid=FD00000000153323&id=580214&playrelated=false&autoplay=false&related=true&hidesocial=true
[debug] [Mediaset] Extracting URL: https://static3.mediasetplay.mediaset.it/player/index.html?appKey=5bd1ceefa0e845001aa4d83d&programGuid=FD00000000153323&id=580214&playrelated=false&autoplay=false&related=true&hidesocial=true
[Mediaset] FD00000000153323: Downloading JSON metadata
[Mediaset] FD00000000153323: Downloading MPEG4 SMIL data
[Mediaset] FD00000000153323: Checking video URL
[Mediaset] FD00000000153323: Downloading M3U SMIL data
[Mediaset] FD00000000153323: Downloading m3u8 information
[Mediaset] FD00000000153323: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[Mediaset] FD00000000153323: Downloading JSON metadata
[debug] Default format spec: bestvideo*+bestaudio/best
[info] FD00000000153323: Downloading 1 format(s): hls-2269+hls-64-1
[debug] Invoking hlsnative downloader on "https://vod01.msf.cdn.mediaset.net/farmunica/2020/07/632113_17352660b5c65b/hlsrc/w6/sd_no_mpl.m3u8"
[hlsnative] Downloading m3u8 manifest
WARNING: Live HLS streams are not supported by the native downloader. If this is a livestream, please add "--downloader ffmpeg --hls-use-mpegts" to your command
[hlsnative] Total fragments: 30
[download] Destination: Est ce que tu m'aimes - Gabriele - 5 dicembre [FD00000000153323].fhls-2269.mp4
[debug] File locking is not supported. Proceeding without locking
[download]  13.3% of ~25.57MiB at  517.35KiB/s ETA 00:49 (frag 4/30)
ERROR: Interrupted by user
Exception ignored in: <_io.FileIO name="Est ce que tu m'aimes - Gabriele - 5 dicembre [FD00000000153323].fhls-2269.mp4.part" mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedWriter name="Est ce que tu m'aimes - Gabriele - 5 dicembre [FD00000000153323].fhls-2269.mp4.part">

[dirkf]

In the failing log:

Unsupported URL: https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/

yt-dl master:

$ python3 -m youtube_dl -v -F 'https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/'
[debug] System config: ['--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-F', 'https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 0059ec748
[debug] Python version 3.5.2 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Requesting header
WARNING: Falling back on generic information extractor.
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Downloading webpage
[generic] est-ce-que-tu-maimes-gabriele-5-dicembre-copia: Extracting information
ERROR: Unsupported URL: https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/
Traceback (most recent call last):
  File "/home/df/Documents/src/youtube-dl/youtube_dl/YoutubeDL.py", line 816, in wrapper
    return func(self, *args, **kwargs)
  File "/home/df/Documents/src/youtube-dl/youtube_dl/YoutubeDL.py", line 837, in __extract_info
    ie_result = ie.extract(url)
  File "/home/df/Documents/src/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/home/df/Documents/src/youtube-dl/youtube_dl/extractor/generic.py", line 3489, in _real_extract
    raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/

yt-dl with added (?:/v\d+)? in URL pattern:

$ python3 -m youtube_dl -v -F 'https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/'
[debug] System config: ['--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '-F', 'https://static3.mediasetplay.mediaset.it/player/v2/index.html?partnerId=wittytv&configId=&programGuid=FD00000000153323&autoplay=true&purl=http://www.wittytv.it/amici/est-ce-que-tu-maimes-gabriele-5-dicembre-copia/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 0059ec748
[debug] Python version 3.5.2 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[Mediaset] FD00000000153323: Downloading JSON metadata
[Mediaset] FD00000000153323: Downloading MPEG4 SD SMIL data
[Mediaset] FD00000000153323: Checking video URL
[Mediaset] FD00000000153323: Downloading MPEG-DASH SD SMIL data
[Mediaset] FD00000000153323: Downloading MPD manifest
[Mediaset] FD00000000153323: Downloading MPD manifest
[Mediaset] FD00000000153323: Downloading MPD manifest
[Mediaset] FD00000000153323: Downloading M3U SD SMIL data
[Mediaset] FD00000000153323: Downloading m3u8 information
[Mediaset] FD00000000153323: Downloading MPEG4 HD SMIL data
[Mediaset] FD00000000153323: Downloading JSON metadata
[info] Available formats for FD00000000153323:
format code    extension  resolution note
hls-audio-ita  mp4        audio only [ita] 
hls-64         mp4        audio only   64k , mp4a.40.5
dash-0_0-0     m4a        audio only [ita] DASH audio   64k , m4a_dash container, mp4a.40.5 (48000Hz)
dash-0_0-1     m4a        audio only [ita] DASH audio   64k , m4a_dash container, mp4a.40.5 (48000Hz)
dash-0_0-2     m4a        audio only [ita] DASH audio   64k , m4a_dash container, mp4a.40.5 (48000Hz)
dash-1_4-0     mp4        768x432    DASH video 1005k , mp4_dash container, avc1.640028, video only
dash-1_4-1     mp4        768x432    DASH video 1005k , mp4_dash container, avc1.640028, video only
hls-1070       mp4        unknown    1070k , avc1.640028, video only
dash-1_5-0     mp4        1024x576   DASH video 1606k , mp4_dash container, avc1.640028, video only
dash-1_5-1     mp4        1024x576   DASH video 1606k , mp4_dash container, avc1.640028, video only
dash-1_5-2     mp4        1024x576   DASH video 1606k , mp4_dash container, avc1.640028, video only
hls-1670       mp4        unknown    1670k , avc1.640028, video only
dash-1_6-0     mp4        1024x576   DASH video 2205k , mp4_dash container, avc1.640028, video only
dash-1_6-1     mp4        1024x576   DASH video 2205k , mp4_dash container, avc1.640028, video only
dash-1_6-2     mp4        1024x576   DASH video 2205k , mp4_dash container, avc1.640028, video only
hls-2269       mp4        unknown    2269k , avc1.640028, video only
http-1         mp4        unknown       0k  (best)
$ 

[pukkandan]

Huh, interesting. Maybe it depends on geo-location? What do you get on yt-dlp master?

[dirkf]

Same result. I'd say that there's just a new (additional) path format for static3.mediasetplay.mediaset.it/player/..., maybe only in partner embeds?

--- old/yt_dlp/extractor/mediaset.py
+++ new/yt_dlp/extractor/mediaset.py
@@ -23,7 +23,7 @@
                             (?:(?:www|static3)\.)?mediasetplay\.mediaset\.it/
                             (?:
                                 (?:video|on-demand|movie)/(?:[^/]+/)+[^/]+_|
-                                player/index\.html\?.*?\bprogramGuid=
+                                player(?:/v\d+)?/index\.html\?.*?\bprogramGuid=
                             )
                     )(?P<id>[0-9A-Z]{16,})
                     '''