Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] reddit user scraper no longer downloads redgif urls submitted after 2PM PST #3397

Closed
jadedgnome opened this issue Dec 12, 2022 · 2 comments
Labels

Comments

@jadedgnome
Copy link

jadedgnome commented Dec 12, 2022

I tried to download this user profile https://www.reddit.com/user/TitBreast

and the latest NSFW post thats 4 hours old wasn't even acknowledged by gallery-dl

https://www.reddit.com/r/WatchItForThePlot/comments/zjecbw/shoka_oshima_enjoying_rin_asukas_plot_in_woman/

it should have downloaded https://redgifs.com/watch/starchytrimhornbill but it didn't even though the link is present in the reddit json. https://www.reddit.com/user/TitBreast/.json?limit=100&raw_json=1

`gallery-dl https://www.reddit.com/user/TitBreast -d TitBreast -v
[gallery-dl][debug] Version 1.24.1
[gallery-dl][debug] Python 3.8.3 - Windows-10-10.0.17763-SP0
[gallery-dl][debug] requests 2.28.1 - urllib3 1.26.13
[gallery-dl][debug] Configuration Files []
[gallery-dl][debug] Starting DownloadJob for 'https://www.reddit.com/user/TitBreast'
[reddit][debug] Using RedditUserExtractor for 'https://www.reddit.com/user/TitBreast'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): oauth.reddit.com:443
[urllib3.connectionpool][debug] https://oauth.reddit.com:443 "GET /user/TitBreast/.json?limit=100&raw_json=1 HTTP/1.1" 200 72657
[redgifs][debug] Using RedgifsImageExtractor for 'https://redgifs.com/watch/primewrychihuahua'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.redgifs.com:443
[urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/auth/temporary HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/gifs/primewrychihuahua HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): thumbs4.redgifs.com:443
[urllib3.connectionpool][debug] https://thumbs4.redgifs.com:443 "GET /PrimeWryChihuahua.mp4?expires=1670819400&signature=f27e68d84d76aae30a95cdefb90a216b9a2acdecac6fba035a73b8cc7231717b&for=8.8.8.8&c=1823c31f7d3-745a-6589-0005-d8e8fe0a44c2 HTTP/1.1" 206 46828430
[downloader.http][debug] Resuming download at byte 7307280

  • TitBreast\redgifs\redgifs_primewrychihuahua.mp4
    [redgifs][debug] Using RedgifsImageExtractor for 'https://redgifs.com/watch/tornunfoldedasiantrumpetfish'
    [urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/gifs/tornunfoldedasiantrumpetfish HTTP/1.1" 200 None
    [urllib3.connectionpool][debug] https://thumbs4.redgifs.com:443 "GET /TornUnfoldedAsiantrumpetfish.mp4?expires=1670819400&signature=c3774b084c79d1b084c55debf5d2b0de256d9064e4ece3324d2ed92ef2aa2443&for=8.8.8.8&c=1823c31f7d3-745a-6589-0005-d8e8fe0a44c2 HTTP/1.1" 200 16889434
  • TitBreast\redgifs\redgifs_tornunfoldedasiantrumpetfish.mp4
    [redgifs][debug] Using RedgifsImageExtractor for 'https://redgifs.com/watch/notableorderlyjabiru'
    [urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/gifs/notableorderlyjabiru HTTP/1.1" 200 None
    [urllib3.connectionpool][debug] https://thumbs4.redgifs.com:443 "GET /NotableOrderlyJabiru.mp4?expires=1670819400&signature=a8910ae37c91d6885e2d2114a9f5aa096290f68889c2e921da8dcd1280066403&for=8.8.8.8&c=1823c31f7d3-745a-6589-0005-d8e8fe0a44c2 HTTP/1.1" 200 38040996
  • TitBreast\redgifs\redgifs_notableorderlyjabiru.mp4
    [redgifs][debug] Using RedgifsImageExtractor for 'https://www.redgifs.com/watch/wornhelplessbluebreastedkookaburra'
    [urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/gifs/wornhelplessbluebreastedkookaburra HTTP/1.1" 200 None
    [urllib3.connectionpool][debug] https://thumbs4.redgifs.com:443 "GET /WornHelplessBluebreastedkookaburra.mp4?expires=1670819400&signature=6151232589d40d9efc018160aafacf4ce8de017bb326da3e3ace4c9b75d854ac&for=8.8.8.8&c=1823c31f7d3-745a-6589-0005-d8e8fe0a44c2 HTTP/1.1" 200 34375090
  • TitBreast\redgifs\redgifs_wornhelplessbluebreastedkookaburra.mp4
    [redgifs][debug] Using RedgifsImageExtractor for 'https://www.redgifs.com/watch/handsomebarequagga'
    [urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v2/gifs/handsomebarequagga HTTP/1.1" 200 None
    [urllib3.connectionpool][debug] https://thumbs4.redgifs.com:443 "GET /HandsomeBareQuagga.mp4?expires=1670819400&signature=e2b87d03994acec86f6610bd7e05a069e076dc592ca6cec3bfd156708c50443f&for=8.8.8.8&c=1823c31f7d3-745a-6589-0005-d8e8fe0a44c2 HTTP/1.1" 200 53191187
  • TitBreast\redgifs\redgifs_handsomebarequagga.mp4`
@enduser420
Copy link
Contributor

default id_max seems to be causing the issue

id_max = self._parse_id("id-max", 2147483647)

if (date_min <= post["created_utc"] <= date_max and
id_min <= self._decode(post["id"]) <= id_max):

$ gdl -s https://old.reddit.com/user/TitBreast
id: 0 <= 2148898028 <= 2147483647 False
date: 0 <= 1670801033.0 <= 2147483647 True
skipped [2148898028] Shoka Oshima enjoying Rin Asuka's plot in 'Woman Who Keeps a Murderer' (2019)
id: 0 <= 2118153174 <= 2147483647 True
date: 0 <= 1669048555.0 <= 2147483647 True
# redgifs_primewrychihuahua.mp4
id: 0 <= 2115324071 <= 2147483647 True    
date: 0 <= 1668872558.0 <= 2147483647 True
# redgifs_tornunfoldedasiantrumpetfish.mp4
id: 0 <= 2112576860 <= 2147483647 True    
date: 0 <= 1668695594.0 <= 2147483647 True
# redgifs_notableorderlyjabiru.mp4

@mikf mikf added the bug label Dec 12, 2022
mikf added a commit that referenced this issue Dec 12, 2022
@mikf
Copy link
Owner

mikf commented Dec 12, 2022

Fixed in 2952add

Use -o id-max=zzzzzzzz as a workaround until the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants