bilibili 部分地址流还是会返回 466 错误 #89

Justsoos · 2018-03-12T06:21:39Z

lulu -d https://www.bilibili.com/video/av20383055/
[DEBUG] http://cn-sdqd-cu-v-06.acgvideo.com:80 "GET /vg4/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1520842500&platform=pc&ssig=bm3bBhWBPwIAdA2DHylJRQ&oi=2937426531&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023046943&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI= HTTP/1.1" 466 0

看来新 API 也不全覆盖？

Justsoos · 2018-03-12T06:25:12Z

奇怪的是，使用同样 api （？）的 annie 没问题
annie.exe https://www.bilibili.com/video/av20383055/

Site: 哔哩哔哩 bilibili.com
Title: 周刊哔哩哔哩排行榜#402
Type: video
Quality: 高清 720P
Size: 184.37 MiB (193327647 Bytes)

5.44 MiB / 184.37 MiB [===>-

Justsoos · 2018-03-12T07:55:33Z

lulu -d https://www.bilibili.com/video/av20523785
lulu -d https://www.bilibili.com/video/av19052070/
突然发现，好多B站下载都 466 错误了。但还是有不少可用。
我用的 0.5.0

iawia002 · 2018-03-12T12:12:21Z

奇怪，466 错误是没加 User-Agent 导致的，这不应该啊

Lulu/lulu/extractor.py

Line 268 in 9254e49

headers['User-Agent'] = self.ua

Lulu/lulu/extractors/bilibili.py

Line 144 in 9254e49

self.ua = FAKE_HEADERS['User-Agent']

[DEBUG] http://upos-hz-mirrorkodo.acgvideo.com:80 "GET /upgcxcode/11/07/33310711/33310711-1-64.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520863239&gen=playurl&um_deadline=1520863239&rate=0&um_sign=28616bd1d27874ea26042f2548b1424d&dynamic=1&os=kodo&oi=2883425376&upsig=bbbe70186fa69ab4d32dbfc06d0ad355 HTTP/1.1" 200 49967484

我一个都复现不了

Justsoos · 2018-03-12T12:29:55Z

lulu -d https://www.bilibili.com/video/av20383055/

[DEBUG] url_save: https://www.bilibili.com/video/av20383055/::{'Accept-Charset': 'UTF-8,;q=0.5', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0', 'Accept-Language': 'en-US,en;q=0.8', 'Accept-Encoding': 'gzip,deflate,sdch', 'Referer': 'https://www.bilibili.com/video/av20383055/', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,/*;q=0.8'}::{'caption': True, 'av': False, 'info_only': False, 'json_output': False, 'password': None}
[DEBUG] Starting new HTTP connection (1): cn-sdqd-cu-v-06.acgvideo.com
[DEBUG] http://cn-sdqd-cu-v-06.acgvideo.com:80 "GET /vg4/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1520864700&platform=pc&ssig=Y9MI4ungPho_8y4V0cnQRw&oi=2937436276&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023051471&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI= HTTP/1.1" 466 0

重装几次测试了 lulu 0.5.0，上面是在这里加 loggin 的结果

Lulu/lulu/common.py

Line 371 in 9254e49

):

logging.debug('url_save: {}::{}::{}'.format(refer, headers, kwargs))

Justsoos · 2018-03-12T12:44:24Z

lulu -d https://www.bilibili.com/video/av20652842/

[DEBUG] url_save: referer=https://www.bilibili.com/video/av20652842/
[DEBUG] Starting new HTTP connection (1): cn-hljheb2-cu-v-05.acgvideo.com
[DEBUG] http://cn-hljheb2-cu-v-05.acgvideo.com:80 "GET /vg4/upgcxcode/78/65/33796578/33796578-1-80.flv?expires=1520865600&platform=pc&ssig=Cjix2C4UQwGxrsRK_eJ2zg&oi=2937436276&nfa=ISqg+Kb2veed34+xb90rig==&dynamic=1&hfa=2023052581&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI= HTTP/1.1" 200 85462183
[DEBUG] Starting new HTTP connection (2): cn-hljheb2-cu-v-05.acgvideo.com

这个页面就没问题。哈哈哈，太神奇了

Justsoos · 2018-03-12T12:48:28Z

好像发现了点什么，好用的都是单独一个文件下载，出问题的，都是多线程下载？

iawia002 · 2018-03-12T12:54:34Z

这个没道理吧，难不成 B 站解析出来的 CDN 地址不一样，他们 CDN 的屏蔽规则不同？

Justsoos · 2018-03-12T16:31:22Z

这几天工具多了，嘿嘿。看看把lulu -u 解析出来的 url 交给 annie 测试:

annie -d -r 'https://www.bilibili.com/video/av20383055/' 'http://cn-sdqd-cu-v-06.acgvideo.com/vg4/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1520879100&platform=pc&ssig=WHG-W7DY4izvguOwW9hvpA&oi=3708392083&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023065893&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI='

annie doesn't support this URL by now, but it will try to download it directly

URL: http://cn-sdqd-cu-v-06.acgvideo.com/vg4/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1520879100&platform=pc&ssig=WHG-W7DY4izvguOwW9hvpA&oi=3708392083&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023065893&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=
Method: GET
Headers: map[Accept-Encoding:[gzip,deflate,sdch] Accept-Language:[en-US,en;q=0.8] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36] Referer:[https://www.bilibili.com/video/av20383055/] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Accept-Charset:[UTF-8,*;q=0.5]]
Status Code: 466

URL: http://cn-sdqd-cu-v-06.acgvideo.com/vg4/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1520879100&platform=pc&ssig=WHG-W7DY4izvguOwW9hvpA&oi=3708392083&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023065893&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=
Method: GET
Headers: map[User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36] Referer:[https://www.bilibili.com/video/av20383055/] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Accept-Charset:[UTF-8,*;q=0.5] Accept-Encoding:[gzip,deflate,sdch] Accept-Language:[en-US,en;q=0.8]]
Status Code: 466

Site: Universal
Title: 33310711-1-64
Type:
Size: 0.00 MiB (0 Bytes)

0 B / ? [---------------------------------------------------------------------------------------------------------------------------------------------------------=]33310711-1-64.flv: file already exists, skipping
0 B / ? [------------------------------------------------------------------------------------------------------------------------------------------------------=] 0s

也同样是466.

另外发现 annie 一个bug，参数不是认 -r 后面，而是谁在前面认谁。
annie -r aaa bbb refer 是aaa，annie aaa -r bbb refer 还是 aaa

Justsoos · 2018-03-12T16:34:20Z

但同样环境，直接用 annie 下，是没问题的。

annie -d https://www.bilibili.com/video/av20383055/

URL: https://www.bilibili.com/video/av20383055/
Method: GET
Headers: map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Accept-Charset:[UTF-8,*;q=0.5] Accept-Encoding:[gzip,deflate,sdch] Referer:[https://www.bilibili.com/video/av20383055/] Accept-Language:[en-US,en;q=0.8] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36]]
Status Code: 200

URL: https://interface.bilibili.com/v2/playurl?appkey=84956560bc028eb7&cid=33310711&otype=json&qn=116&quality=116&type=&sign=09bf3ceeca68dca6ecc8a62d5def7184
Method: GET
Headers: map[Accept-Charset:[UTF-8,;q=0.5] Accept-Encoding:[gzip,deflate,sdch] Accept-Language:[en-US,en;q=0.8] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,/*;q=0.8] Referer:[https://interface.bilibili.com/v2/playurl?appkey=84956560bc028eb7&cid=33310711&otype=json&qn=116&quality=116&type=&sign=09bf3ceeca68dca6ecc8a62d5def7184]]
Status Code: 200

Site: 哔哩哔哩 bilibili.com
Title: 周刊哔哩哔哩排行榜#402
Type: video
Quality: 高清 720P
Size: 184.37 MiB (193327647 Bytes)

0 B / 184.37 MiB [-----------------------------------------------------------------------------------------------------------------------------------------] 0.00%
URL: http://cn-sdqd-cu-v-06.acgvideo.com/vg4/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1520879400&platform=pc&ssig=1uZtrPTo-yUK1IasZ9XlyQ&oi=3708392083&nfa=hIhq6gZ5VhoCTqaFO9OkTw==&dynamic=1&hfa=2023067446&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=
Method: GET
Headers: map[Accept-Encoding:[gzip,deflate,sdch] Accept-Language:[en-US,en;q=0.8] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36] Referer:[https://www.bilibili.com/video/av20383055/] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Accept-Charset:[UTF-8,*;q=0.5]]
Status Code: 200

URL: http://cn-sdqd-cu-v-06.acgvideo.com/vg0/upgcxcode/11/07/33310711/33310711-2-64.flv?expires=1520879400&platform=pc&ssig=FhEOQy46RCFNRuUj5YXSTw&oi=3708392083&nfa=0IC9+c6W2KQLwirHaGDLFA==&dynamic=1&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=
Method: GET
Headers: map[Referer:[https://www.bilibili.com/video/av20383055/] Accept-Charset:[UTF-8,;q=0.5] Accept-Encoding:[gzip,deflate,sdch] Accept-Language:[en-US,en;q=0.8] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8]]
Status Code: 200
0 B / 184.37 MiB [-----------------------------------------------------------------------------------------------------------------------------------------] 0.00%
URL: http://cn-sdqd-cu-v-06.acgvideo.com/vg2/upgcxcode/11/07/33310711/33310711-3-64.flv?expires=1520879400&platform=pc&ssig=cufGtImsERsJipW7_O0o9g&oi=3708392083&nfa=M6bUBugBUd+gl1Pi7CMklQ==&dynamic=1&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=
Method: GET
Headers: map[Accept-Language:[en-US,en;q=0.8] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Referer:[https://www.bilibili.com/video/av20383055/] Accept-Charset:[UTF-8,;q=0.5] Accept-Encoding:[gzip,deflate,sdch]]
Status Code: 200
192.16 KiB / 184.37 MiB [>-------------------------------------------------------------------------------------------------------------] 0.10% 184.94 KiB/s 16m59s
URL: http://cn-sdqd-cu-v-06.acgvideo.com/vg0/upgcxcode/11/07/33310711/33310711-4-64.flv?expires=1520879400&platform=pc&ssig=NS57R0apxfGtKroiKBQl2g&oi=3708392083&nfa=8L3Zptf+tjzlyDHs1VXo8w==&dynamic=1&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=
Method: GET
Headers: map[User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8] Accept-Charset:[UTF-8,*;q=0.5] Accept-Encoding:[gzip,deflate,sdch] Accept-Language:[en-US,en;q=0.8] Referer:[https://www.bilibili.com/video/av20383055/]]
Status Code: 200
8.95 MiB / 184.37 MiB [=====>-----------------------------------------------------------------------------------------------------------] 4.85% 877.52 KiB/s 3m24s

Justsoos · 2018-03-12T16:59:11Z

annie 大战 lulu …… 不知 Bilibili 是不是因为板蓝根输死了，搞什么鬼，搞什么鬼！

Justsoos · 2018-03-12T17:02:36Z

上面几个，都出现了同一个 cdn，可见并不太可能是 cdn 的 IP 黑名单问题。
http://cn-sdqd-cu-v-06.acgvideo.com

iawia002 · 2018-03-12T23:42:25Z

好像 lulu 和 Annie 解析出来的地址后面的参数不一样？

iawia002 · 2018-03-12T23:50:40Z

它们用的 api 不是完全一样，上次 lulu 的只能算小更新，Annie 用的是现在 B 站网页端用的 api

Justsoos · 2018-03-13T08:07:34Z

http://cn-nmghhht-cu-v-10.acgvideo.com/vg1/upgcxcode/34/97/31079734/31079734-1-80.flv?expires=1520935200&platform=pc&ssig=bi3OXBbMa4WmBst1RRHXog&oi=3708392083&nfa=UEav3bCY+XfNyw5l/+27kw==&dynamic=1&hfa=2023007538&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=

http://cn-nmghhht-cu-v-10.acgvideo.com/vg1/upgcxcode/34/97/31079734/31079734-1-80.flv?expires=1520935200&platform=pc&ssig=BiBQFPr5PFC3fIrxNnR3sw&oi=3708392083&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023007458&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=

这参数基本看不出来有啥不一样。这是刚生成的两个。但奇怪的是，你那边 lulu 好用。

iawia002 · 2018-03-13T09:40:53Z

直接看我还以为少了参数，自从加了 headers，我从没遇到过这些问题

Justsoos · 2018-03-13T12:36:16Z

嘿嘿，开了个新centos7虚拟机，全新装 lulu 测试，
这几个还是一样，466

lulu -d https://www.bilibili.com/video/av20383055/
lulu -d https://www.bilibili.com/video/av20523785
lulu -d https://www.bilibili.com/video/av19052070/

这个就没问题：
https://www.bilibili.com/video/av20652842/
挂了代理，也是一样。哈~~~

Justsoos · 2018-03-13T12:44:54Z

我估计是 b 站技术对 cdn 部署问题引发 bug，因为上面用 annie 抓到的流地址，主机和参数格式，全都一样的，但 annie 的200，lulu 的就是466，说明是服务器上 xml 里存的 token 加密key就是错的。
有空我再看看youtube-dl，他们也有一个看似不同的 api 算法，最后下载是 mp4

Justsoos · 2018-03-13T19:39:25Z

我差点没看出问题来，你确定这是你拿到的流地址么？？
#89 (comment)

[DEBUG] http://upos-hz-mirrorkodo.acgvideo.com:80 "GET /upgcxcode/11/07/33310711/33310711-1-64.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520863239&gen=playurl&um_deadline=1520863239&rate=0&um_sign=28616bd1d27874ea26042f2548b1424d&dynamic=1&os=kodo&oi=2883425376&upsig=bbbe70186fa69ab4d32dbfc06d0ad355 HTTP/1.1" 200 49967484

这样的有很长参数 e= 流地址，以及缺乏&hfb，却有 &um_sign 的流地址，总之，各种不一样啊，我的 lulu 从来没取出来过。你哪里搞来的？你看一下上面我发的流地址，没一个和你这个一样，我用的annie 取出来地址也和你这个不同。
这是你自己的小灶版本？？

Justsoos · 2018-03-13T20:17:02Z

查到问题了，原来这个鬼在lulu 和you-get 取 api xml 地址机制里！

我直接从一个多次测算都是 466 的b 站页面，用 lulu 内提取 api 地址的 xml 里面，找没有&hfb 参数，但有 e= 的流地址，这次没有466，成功了。
https://www.bilibili.com/video/av19052070/

#../annie -r http://www.bilibili.com/ "http://upos-hz-mirroross.acgvideo.com/upgcxcode/34/97/31079734/31079734-1-32.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520978256&gen=playurl&um_deadline=1520978256&rate=0&um_sign=f1190bda62456c285f10c1c1eaf15842&dynamic=1&os=oss&oi=2000036556&upsig=e4f0558c317118fe450d9705a256734d"

又从这个xml 里取了其他带 &hfb 参数的测，基本都是466错误。看来这个bug 就在lulu 的取地址机制里。

下面是 xml 内容：

{'flv': {'src': ['http://cn-nmghhht-cu-v-10.acgvideo.com/vg1/upgcxcode/34/97/31079734/31079734-1-80.flv?expires=1520978100&platform=pc&ssig=_lhgEk2dQ6iVSSu06F2ASA&oi=2000036556&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023427176&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI='], 'size': 32169987, 'container': 'flv'}}

{'flv': {'src': ['http://cn-nmghhht-cu-v-10.acgvideo.com/vg1/upgcxcode/34/97/31079734/31079734-1-80.flv?expires=1520978100&platform=pc&ssig=_lhgEk2dQ6iVSSu06F2ASA&oi=2000036556&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023427176&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI='], 'size': 32169987, 'container': 'flv'}, 'flv720': {'src': ['http://upos-hz-mirrorks3.acgvideo.com/upgcxcode/34/97/31079734/31079734-1-64.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520978255&gen=playurl&um_deadline=1520978255&rate=1000&um_sign=0a19b27440816ca4a917516e22b35c5d&dynamic=1&os=ks3&oi=2000036556&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=&upsig=25facc9d47b76c2e64a4aab20539caa5'], 'size': 24714815, 'container': 'flv'}}

{'flv': {'src': ['http://cn-nmghhht-cu-v-10.acgvideo.com/vg1/upgcxcode/34/97/31079734/31079734-1-80.flv?expires=1520978100&platform=pc&ssig=_lhgEk2dQ6iVSSu06F2ASA&oi=2000036556&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023427176&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI='], 'size': 32169987, 'container': 'flv'}, 'flv720': {'src': ['http://upos-hz-mirrorks3.acgvideo.com/upgcxcode/34/97/31079734/31079734-1-64.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520978255&gen=playurl&um_deadline=1520978255&rate=1000&um_sign=0a19b27440816ca4a917516e22b35c5d&dynamic=1&os=ks3&oi=2000036556&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=&upsig=25facc9d47b76c2e64a4aab20539caa5'], 'size': 24714815, 'container': 'flv'}, 'flv480': {'src': ['http://upos-hz-mirroross.acgvideo.com/upgcxcode/34/97/31079734/31079734-1-32.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520978256&gen=playurl&um_deadline=1520978256&rate=0&um_sign=f1190bda62456c285f10c1c1eaf15842&dynamic=1&os=oss&oi=2000036556&upsig=e4f0558c317118fe450d9705a256734d'], 'size': 16669358, 'container': 'flv'}}

{'flv': {'src': ['http://cn-nmghhht-cu-v-10.acgvideo.com/vg1/upgcxcode/34/97/31079734/31079734-1-80.flv?expires=1520978100&platform=pc&ssig=_lhgEk2dQ6iVSSu06F2ASA&oi=2000036556&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023427176&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI='], 'size': 32169987, 'container': 'flv'}, 'flv720': {'src': ['http://upos-hz-mirrorks3.acgvideo.com/upgcxcode/34/97/31079734/31079734-1-64.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520978255&gen=playurl&um_deadline=1520978255&rate=1000&um_sign=0a19b27440816ca4a917516e22b35c5d&dynamic=1&os=ks3&oi=2000036556&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=&upsig=25facc9d47b76c2e64a4aab20539caa5'], 'size': 24714815, 'container': 'flv'}, 'flv480': {'src': ['http://upos-hz-mirroross.acgvideo.com/upgcxcode/34/97/31079734/31079734-1-32.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520978256&gen=playurl&um_deadline=1520978256&rate=0&um_sign=f1190bda62456c285f10c1c1eaf15842&dynamic=1&os=oss&oi=2000036556&upsig=e4f0558c317118fe450d9705a256734d'], 'size': 16669358, 'container': 'flv'}, 'mp4': {'src': ['http://upos-hz-mirrorkodo.acgvideo.com/upgcxcode/34/97/31079734/31079734-1-16.mp4?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1520978256&gen=playurl&um_deadline=1520978256&rate=0&um_sign=c7410dd1336cab2f8a632488b2d11503&dynamic=1&os=kodo&oi=2000036556&upsig=144656a58136a7aa1557f4d048cb5083'], 'size': 7542099, 'container': 'mp4'}}

you-get 没这个 466 问题。

Justsoos · 2018-03-13T20:29:05Z

试了 you-get 那边给出 api 下三个流地址链接，都可以正确下载，没有466。。。
soimort/you-get#2579
差别这么大么？？

{
"from": "local",
"result": "suee",
"quality": 64,
"format": "flv720",
"timelength": 417000,
"accept_format": "flv720,flv480,mp4",
"accept_quality": [
64,
32,
16
],
"accept_description": [
"高清 720P",
"清晰 480P",
"流畅 360P"
],
"seek_param": "start",
"seek_type": "offset",
"durl": [
{
"order": 1,
"length": 417000,
"size": 46179119,
"url": "http://cn-jlcc3-cu.acgvideo.com/vg3/upgcxcode/86/04/33250486/33250486-1-64.flv?expires=1520979600&platform=pc&ssig=PNfAngJEMzmQjOoeAdYSkA&oi=2000036556&nfa=hK2nzLyWfNzc0SC7ej5bEw==&dynamic=1&hfa=2023428502&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=",
"backup_url": [
"http://cn-sdqd-cu-v-03.acgvideo.com/vg3/upgcxcode/86/04/33250486/33250486-1-64.flv?expires=1520979600&platform=pc&ssig=PNfAngJEMzmQjOoeAdYSkA&oi=2000036556&nfa=hK2nzLyWfNzc0SC7ej5bEw==&dynamic=1&hfa=2023428502&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=",
"http://cn-nmghhht-cu-v-08.acgvideo.com/vg5/upgcxcode/86/04/33250486/33250486-1-64.flv?expires=1520979600&platform=pc&ssig=PMYy2e4B0LZGMtU4N_ztbA&oi=2000036556&nfa=hK2nzLyWfNzc0SC7ej5bEw==&dynamic=1&hfa=2023428502&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI="
]
}
]
}

Justsoos · 2018-03-13T21:03:37Z

放两个链接，不知道你到时候还能用否，上面是 you-get 的，可用，下面是 lulu 生成的，466 错误

http://cn-sdqd-cu-v-06.acgvideo.com/vg4/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1520981700&platform=pc&ssig=rg2EGXIC3EMovVaQeEPTOw&oi=2000036556&nfa=hIhq6gZ5VhoCTqaFO9OkTw==&dynamic=1&hfa=2023431916&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=


http://cn-sdqd-cu-v-06.acgvideo.com/vg4/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1520982000&platform=pc&ssig=QwvFtqF5eNRK-LMvQJg7fA&oi=2000036556&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023431991&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI=

是这个屡次测试过不去的页面 https://www.bilibili.com/video/av20383055/

iawia002 · 2018-03-13T22:10:02Z

我直接换成 Annie 那边的 api 算了

Justsoos · 2018-03-14T05:00:07Z

我还没测，但感觉目前这个API可能还是可用，但不知道哪个item是好的，以及选择标准。
你可以遍历一下这xml，挨个requests get 试试。

Justsoos · 2018-03-14T08:41:42Z

因为有 expires 或者 ip 绑定，或者 token 失效，我这 466 错误的流链接不能发给你重现。
你可以从这里

Lulu/lulu/extractors/bilibili.py

Line 104 in 9254e49

return xml_str

取 xml 里地址来试试，看看是不是也一样 466，排除是只有我遇到的特例。
我就奇怪了，为啥我和你安装一样，我这里每次遇到某些页面，取出来的都不能用，而你的 lulu 取出的地址都能用？？？
你不觉得这事儿太神奇，而不可能了么？？ @iawia002

iawia002 · 2018-03-14T10:01:53Z

可能是我们 B 站返回的 CDN 都不一样？我再测试一下你的那几个 URL

iawia002 · 2018-03-14T10:05:18Z

site:                哔哩哔哩 bilibili.com
title:               周刊哔哩哔哩排行榜#402
stream:
    - format:        flv720
      container:     flv
      size:          184.4 MiB (193327647 bytes)
    # download-with: lulu --format=flv720 [URL]

Downloading 周刊哔哩哔哩排行榜#402.flv ...
 0.0% (  0.0/184.4MB) ├────────────────────────────────────────┤[1/4] [DEBUG] Starting new HTTP connection (1): upos-hz-mirrorkodo.acgvideo.com
[DEBUG] http://upos-hz-mirrorkodo.acgvideo.com:80 "GET /upgcxcode/11/07/33310711/33310711-1-64.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1521029029&gen=playurl&um_deadline=1521029029&rate=0&um_sign=185a4739644418323537def37c4895c8&dynamic=1&os=kodo&oi=2883425376&upsig=09de07728968cad6ae4f71a113a8b27d HTTP/1.1" 200 49967484
[DEBUG] Starting new HTTP connection (2): upos-hz-mirrorkodo.acgvideo.com
[DEBUG] http://upos-hz-mirrorkodo.acgvideo.com:80 "GET /upgcxcode/11/07/33310711/33310711-1-64.flv?e=ig8g5X10ugNcXBlqNxHxNEVE5XREto8KqJZHUa6m5J0SqE85tZvEuENvNC8xNEVE9EKE9IMvXBvE2ENvNCImNEVEK9GVqJIwqa80WXIekXRE9IMvXBvEuENvNCImNEVEua6m2jIxux0CkF6s2JZv5x0DQJZY2F8SkXKE9IB5to8euxZM2rNcNbUVhwdVhoM1hwdVhwdVNCM=&platform=pc&uipk=5&uipv=5&deadline=1521029029&gen=playurl&um_deadline=1521029029&rate=0&um_sign=185a4739644418323537def37c4895c8&dynamic=1&os=kodo&oi=2883425376&upsig=09de07728968cad6ae4f71a113a8b27d HTTP/1.1" 200 49967484
 7.0% ( 12.8/184.4MB) ├███─────────────────────────────────────┤[1/4]   13 MB/s^

site:                哔哩哔哩 bilibili.com
title:               周刊哔哩哔哩排行榜#402
stream:
    - format:        flv720
      container:     flv
      size:          184.4 MiB (193327647 bytes)
    # download-with: lulu --format=flv720 [URL]

Downloading 周刊哔哩哔哩排行榜#402.flv ...
 0.0% (  0.0/184.4MB) ├────────────────────────────────────────┤[1/4] [DEBUG] Starting new HTTP connection (1): cn-scnc2-dx.acgvideo.com
[DEBUG] http://cn-scnc2-dx.acgvideo.com:80 "GET /vg7/upgcxcode/11/07/33310711/33310711-1-64.flv?expires=1521028800&platform=pc&ssig=mHpvvI5VebOgCtu4ST9lfA&oi=2883425376&nfa=iKQJtLw3Fy5f05Q/mDvFYw==&dynamic=1&hfa=2023478935&hfb=Yjk5ZmZjM2M1YzY4ZjAwYTMzMTIzYmIyNWY4ODJkNWI= HTTP/1.1" 466 0

复现了一次

iawia002 · 2018-03-14T10:10:19Z

它的 API 有一个 URL 字段，还有 backup url 的字段，讲道理，URL 应该就是第一选择吧

Justsoos · 2018-03-14T10:10:23Z

哈哈，我说么，不可能是我这死活不行的。
当 B 站也真够乱的，一个 xml 里混合这么多可用，不可用的，到底啥意思呢？

Justsoos · 2018-03-15T01:29:20Z

我下载了类似文件，每个链接有6个...

https://interface.bilibili.com:443 "GET /v2/playurl?cid=33310711&player=1&qn=116&quality=116&ts=1521074265&sign=83ca47642a00471e73f1d6e98b858cbe

把其中&quality=116 换成 74, 80, 64, 32, 15，分别6个文件，看返回 xml ，都是几乎一样的内容。不知道这个参数在这到底算是什么作用？或者，根本就是已经被废弃了吧？我看 you-get 这个参数轮询取 0-4 个值。

https://github.com/mq-liu/you-get/blob/1900f7608cc2756d5460c99eb792c8e0eb42e7f4/src/you_get/extractors/bilibili.py#L100

每个文件里内容，相当于 lulu 取到一个文件里的4分之一（改一下，是4分之一）。但没好用的。。。。一个466，全都466

现在lulu 这个 api 还是可用的，其中很大部分 <url></url> 可用，返回 200 非 466
只是还没看出来是怎么个选择规则

Justsoos · 2018-03-15T05:04:28Z

哈哈，发现问题所在了。
lulu 频繁访问 6 个接口（相比you-get 只访问4个，youtube 不访问，自己编 url 所以不容易被封），可能导致其被短暂封 IP 的原因！！

我把每次访问 21 个 cdn 的 url 全都 requests.get ，并列表。如下：
first.txt 这个文件内，只有最后4个 [200] 可用，其他全部 466 !!!
但我反复测试后，几乎所有流cdn服务器都返回 466 了。我估计是因为b站的反拖站技术在这里玩猫腻。

然后，我加了个 time.sleep(2)，到每个访问之间，试过几分钟后。有一部分200可用了。这保存到 second.txt
下面我继续测测。

first.txt
second.txt

Justsoos · 2018-03-15T05:09:19Z

你也可以来测测，看看你那 IP 被 b 站态度如何。
在这行下面插入代码：

Lulu/lulu/extractors/bilibili.py

Line 114 in 9254e49

url = durl.getElementsByTagName('url')[0]

            from ..common import urlopen_with_retry
            FAKE_HEADERS['Referer'] = self.url
            headers = FAKE_HEADERS
            u = url.firstChild.nodeValue
            print(u,':::', urlopen_with_retry(u, headers=headers))
            time.sleep(2)

执行：
lulu -du https://www.bilibili.com/video/av20383055/ > first.txt

@iawia002

Justsoos · 2018-03-15T05:18:57Z

这真是一个挺变态的行为。。。用前几天楼上发现怎么都不会 466 的网址测：
lulu -du https://www.bilibili.com/video/av20652842/ > 4th.txt
还真的是怎么蹂躏都不466，坚挺无比的200 .... 但也看出来，这个网址在 cdn 上每个cdn 服务器只有一个 &quality 楼上那个间歇性不好用的，每个 cdn 有 4 个&quality 条目。
4th.txt

Justsoos · 2018-03-15T06:27:00Z

顺手测一下 you-get，对于这个 lulu 在我这频繁466 的页面：
you-get -du https://www.bilibili.com/video/av20383055/ > you-get-1st.txt
https://github.com/soimort/you-get/blob/1900f7608cc2756d5460c99eb792c8e0eb42e7f4/src/you_get/extractors/bilibili.py#L82
下面插入代码：

            import requests
            sess = requests.Session()
            u = url.firstChild.nodeValue
            r = sess.get(u, stream=True, verify=False, headers={'referer': self.url, 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36'})
            print(u,':::', r)
            #time.sleep(2)

全200，真是好成绩啊。
you-get-1st.txt

Justsoos · 2018-03-15T06:41:17Z

lulu -du https://www.bilibili.com/video/av20383055/ > 6th.txt
全家桶，把 63 个cdn 地址全测一遍 <Response [200]> 不一定出现在那个位置，没看出规律来。
6th.txt

这的确不好办了。你有空也可以看看，是否现在这个 api 还能继续用。

Justsoos · 2018-03-15T06:52:00Z

you-get 全家桶测试：

you-get -du https://www.bilibili.com/video/av20383055/ > you-get-2.txt
you-get -du https://www.bilibili.com/video/av20652842/ > you-get-3.txt

一个 466 都没有！我看，lulu 就换 you-get 现在这个 api 就行啊

you-get-2.txt
you-get-3.txt

Justsoos · 2018-03-15T07:25:34Z

全家桶，就是加了个遍历，把 xml 读出的全部 url 都测一遍。

            from ..common import urlopen_with_retry
            FAKE_HEADERS['Referer'] = self.url
            headers = FAKE_HEADERS
            for i in durl.getElementsByTagName('url'):
                u = i.firstChild.nodeValue
                print(u,':::', urlopen_with_retry(u, headers=headers))
            #time.sleep(1)

iawia002 · 2018-03-15T08:00:16Z

you-get 那个 API 不对，只改了半截的，只拿得到一种清晰度的。V2 的清晰度不是以前那样 4-0 那样取的

Justsoos · 2018-03-15T08:08:26Z

哦，这我倒没测，只是用全家桶都试了一遍，b 站 api 背后逻辑的确难猜。可能是最近360周鸿祎搞的拖库副作用吧。

Justsoos · 2018-03-15T08:17:03Z

还有个折中，坑 b 站的方式，轮询 xml，把63个全家桶都测一遍，那个好用，那个就交给用户下载。哈哈哈，会把这 api 搞死

iawia002 · 2018-03-15T08:18:01Z

我空了直接把它改成 Annie 的 api 看看，不用 xml 了

Justsoos · 2018-03-15T08:19:24Z

lulu -di https://www.bilibili.com/video/av10813231/
这个，和其他几个页面多次测，63+ 个地址，全部 200，反复都没有466
看来就是这个结论： b 站这个 api 没错，cdn 配置有问题。

iawia002 · 2018-03-15T08:19:28Z

我说朋友，你不上课或者上班之类的吗😂感觉你住 Github 上啊

Justsoos · 2018-03-15T08:20:08Z

我只是随便看一眼 git 提示，然后干别的去。哈哈

Justsoos · 2018-03-15T08:39:49Z

我这几天已经把golang 搞定了，开始改annie了，哈哈，你说我收获大不大？

iawia002 · 2018-03-15T08:58:14Z

Justsoos · 2018-03-17T13:25:26Z

新 api 大致测了几个页面全家桶，都是 200，没一个 466 。
看来之前就是 b 站狙击周鸿祎故意搞的迷魂阵的老 api

Justsoos · 2018-03-17T13:26:40Z

哎，我有点感觉心痒痒了，b 站这么多 cdn 全曝光，这不是给多线程下载最好礼物么？哇哈哈~~
要不要做做？ @iawia002

iawia002 · 2018-03-17T13:53:48Z

lulu 对分了片的视频支持多线程（-T），annie 也对分片的视频用 goroutine 来下载，我觉得可以跑满带宽

iawia002 added the bilibili label Mar 12, 2018

Justsoos mentioned this issue Mar 13, 2018

是不是bilibili 对流作了反盗播设置？一直是ffmpeg: HTTP error 453 #54

Closed

iawia002 added the bug Something isn't working label Mar 14, 2018

iawia002 mentioned this issue Mar 15, 2018

extractors/bilibili: new api #90

Merged

iawia002 closed this as completed in #90 Mar 15, 2018

bilibili 部分地址流还是会返回 466 错误 #89

bilibili 部分地址流还是会返回 466 错误 #89

Comments

Justsoos commented Mar 12, 2018

Justsoos commented Mar 12, 2018

Justsoos commented Mar 12, 2018

iawia002 commented Mar 12, 2018 • edited Loading

Justsoos commented Mar 12, 2018

Justsoos commented Mar 12, 2018 • edited Loading

Justsoos commented Mar 12, 2018

iawia002 commented Mar 12, 2018

Justsoos commented Mar 12, 2018

Justsoos commented Mar 12, 2018

Justsoos commented Mar 12, 2018

Justsoos commented Mar 12, 2018

iawia002 commented Mar 12, 2018

iawia002 commented Mar 12, 2018

Justsoos commented Mar 13, 2018

iawia002 commented Mar 13, 2018

Justsoos commented Mar 13, 2018

Justsoos commented Mar 13, 2018

Justsoos commented Mar 13, 2018

Justsoos commented Mar 13, 2018

Justsoos commented Mar 13, 2018

Justsoos commented Mar 13, 2018 • edited Loading

iawia002 commented Mar 13, 2018

Justsoos commented Mar 14, 2018

Justsoos commented Mar 14, 2018

iawia002 commented Mar 14, 2018

iawia002 commented Mar 14, 2018

iawia002 commented Mar 14, 2018

Justsoos commented Mar 14, 2018

Justsoos commented Mar 15, 2018 • edited Loading

Justsoos commented Mar 15, 2018 • edited Loading

Justsoos commented Mar 15, 2018

Justsoos commented Mar 15, 2018

Justsoos commented Mar 15, 2018

Justsoos commented Mar 15, 2018

Justsoos commented Mar 15, 2018

Justsoos commented Mar 15, 2018

iawia002 commented Mar 15, 2018

Justsoos commented Mar 15, 2018

Justsoos commented Mar 15, 2018

iawia002 commented Mar 15, 2018

Justsoos commented Mar 15, 2018

iawia002 commented Mar 15, 2018

Justsoos commented Mar 15, 2018

Justsoos commented Mar 15, 2018

iawia002 commented Mar 15, 2018

Justsoos commented Mar 17, 2018

Justsoos commented Mar 17, 2018

iawia002 commented Mar 17, 2018 • edited Loading

iawia002 commented Mar 12, 2018 •

edited

Loading

Justsoos commented Mar 12, 2018 •

edited

Loading

Justsoos commented Mar 13, 2018 •

edited

Loading

Justsoos commented Mar 15, 2018 •

edited

Loading

Justsoos commented Mar 15, 2018 •

edited

Loading

iawia002 commented Mar 17, 2018 •

edited

Loading