Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Downloading all Fantia Posts from one creator results in 403 #41

Closed
Aaeeschylus opened this issue Mar 11, 2023 · 4 comments
Closed

[Bug]: Downloading all Fantia Posts from one creator results in 403 #41

Aaeeschylus opened this issue Mar 11, 2023 · 4 comments
Assignees
@KJHJason
Labels
bug Something isn't working

Comments

@Aaeeschylus
Copy link

Which program are you seeing this problem on?

Cultured Downloader (exe), Cultured Downloader (py)

Program Version

4.2.2

Steps to reproduce the bug

  1. Go to "Download all Fantia posts from creator(s)" section
  2. Put in link for creator and select multiple pages (ie. 1-12)
  3. Enter Y for Download Images, thumbnail and attachments
  4. Start downloading

Expected behaviour

All images/attachments/thumbnails from all pages entered should be downloaded.

Relevant log output (If any)

----------------------------------------------------------------------------------------------------
2023-03-12 10:05:46,513 [Cultured Downloader V4.2.2] [ERROR]: 
Uncaught TypeError
----------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "utils\download.py", line 95, in async_download_file
  File "httpx\_models.py", line 749, in raise_for_status
httpx.HTTPStatusError: Client error '403 Forbidden' for url 'https://cc.fantia.jp/uploads/post_content_photo/file/8723128/9c63c89f-9c09-41de-b86f-c02ca249c65c.jpg?Key-Pair-Id=APKAIOCKYZS7WKBB6G7A&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9jYy5mYW50aWEuanAvdXBsb2Fkcy9wb3N0X2NvbnRlbnRfcGhvdG8vZmlsZS84NzIzMTI4LzljNjNjODlmLTljMDktNDFkZS1iODZmLWMwMmNhMjQ5YzY1Yy5qcGciLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzg1NzUyNDV9fX1dfQ__&Signature=dxXDCPVukZMbq3YerrxpGYmreBPu1lmSlEDgtyqIg8NI0JOtKdApmiABLvtqVDFL54TKsMmNoe8k1th-cMvlXV0RHn9mdDwUSSrQXw4fnUdHb243XUuJwBNKrNVab4WQFFKO0L7PT7mu0lEKC9U6sUljO63jcUiY3pCvkuQV2i1FbgMvp5ZKHPoQqgCpBn7JQ5kUYsZ1FfE~Qx0QYMfE0vDLZvkIFFn~toykcExGykrIhqixnroNjPGNlaTcNNCrALpyQWSUttgcUPw7EN1V802JQOSmpb1kO2UJ7japjbBN-7h8v-3hbLCccN8tQqY10IXpDVK8kA6U1Kh9oc~sgw__'
For more information check: https://httpstatuses.com/403

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cultured_downloader.py", line 540, in <module>
  File "cultured_downloader.py", line 468, in main
  File "cultured_downloader.py", line 463, in initialise
  File "cultured_downloader.py", line 215, in main_program
  File "cultured_downloader.py", line 195, in nested_menu
  File "cultured_downloader.py", line 164, in download_process
  File "asyncio\runners.py", line 190, in run
  File "asyncio\runners.py", line 118, in run
  File "asyncio\base_events.py", line 653, in run_until_complete
  File "utils\web_driver.py", line 687, in execute_download_process
  File "utils\functional.py", line 1011, in check_download_tasks
  File "utils\functional.py", line 983, in call_async_result
  File "utils\download.py", line 107, in async_download_file
  File "concurrent\futures\thread.py", line 58, in run
  File "<frozen genericpath>", line 19, in exists
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

It downloads the first page or so correctly, but once it hits the 403, it stops downloading anything else and closes.

Screenshots (Optional)

No response
@Aaeeschylus Aaeeschylus added the bug Something isn't working label Mar 11, 2023
@KJHJason
Copy link
Owner

KJHJason commented Mar 13, 2023 •
edited
Loading

how long did it take for it to download the first page?

@Aaeeschylus
Copy link
Author

how long did it take for it to download the first page?

Not very long at all. about 5-10 minutes? I didn't time it.

@KJHJason
Copy link
Owner

Thanks for raising this issue.
This is an oversight from my end as it looks like Fantia has a short expiry duration of their signed AWS S3 URLs.

I can reduce the chance of the signed URL from expiring but not eliminating the problem entirely.
However, the fix will be in the upcoming v5.0.0 of this Python program where it uses the upcoming v1.1.1 of the CLI tool for its download logic. Please wait till then!

KJHJason added a commit to KJHJason/Cultured-Downloader-CLI that referenced this issue Mar 13, 2023
@KJHJason
Optimisations and bug fixes
e46fc0d 
- Mainly memory optimisations
- Addresses issue [Bug]: Downloading all Fantia Posts from one creator results in 403 [#41](KJHJason/Cultured-Downloader#41) on the main Python program repository
KJHJason added a commit that referenced this issue Mar 18, 2023
@KJHJason
fix runtime error #41 and bump up ver
7377e5c
KJHJason added a commit that referenced this issue Mar 18, 2023
@KJHJason @dependabot
Fix runtime error for #41 (#44)
04f04fe 
* Bump pydantic from 1.10.5 to 1.10.6 (#40) (#42)

Bumps [pydantic](https://github.com/pydantic/pydantic) from 1.10.5 to 1.10.6.
- [Release notes](https://github.com/pydantic/pydantic/releases)
- [Changelog](https://github.com/pydantic/pydantic/blob/v1.10.6/HISTORY.md)
- [Commits](pydantic/pydantic@v1.10.5...v1.10.6)

---
updated-dependencies:
- dependency-name: pydantic
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* add cli repo

* fix runtime error #41 and bump up ver

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@KJHJason KJHJason mentioned this issue Mar 27, 2023
Merged
KJHJason added a commit to KJHJason/Cultured-Downloader-CLI that referenced this issue Mar 27, 2023
@KJHJason @dependabot
update to v1.2.0 (#23)
f6f5638 
* minor edits and added delays to pixiv api calls

* touch up on ffmpeg args and edit flag description

* added .apng and .webp to ugoira conversion

* mostly done

* added thumbnail, images, attachments flags and readme

* fixed a bug

* minor changes

* a small bug fix

* minor edits, considering to add pixiv mobile API

* removed a line

* pixiv mobile wip

* pixiv mobile done

* edit readme

* added winres files for building for windows

* moved files and remove winres.json

* Delete winres.json

* fixed a bug where folder with no content are being created

* edit readme

* removed flags for cobra and some code refactor

* bug fixes

* code refactor wip

- In the midst of removing most of the codes that used `panic` to handle errors
- Added error codes to make debugging easier
- Have yet to improve the logging function

* code refactor mostly done, requires testing

* bug fixes, minor optimsations, and code refactor

- Added a function to log slices of errors or a channel of errors
- Some optimisations
- Fixed a potential race condition in the Pixiv Mobile RefreshAccessToken function

* Added cookie file path support

* oops

* code cleanup and added netscape cookie validations

* code refactor and optimisations

* added spinner to be integrated with the main program

* integrated custom made spinner

* bug fixes and optimisations

* minor edit

* Code Refactor and Addition of Feature

- Minor optimisations
- Resolved #13
- Separated the Cobra CMDs into their own files
- Added a .bat file to compile the program

* cleanup codes and added bug report template

* update pull req template

* edit pull req template again

* removed whitespaces in pull req template

* Code cleanup

- Remove deprecated Golang codes
    - rand.Seed(seed) to rand.New(rand.NewSource(seed))
- Improve code readability and maintainability

* fixed bugs with pixiv fanbox downloads

* fixed bugs

* bug fixes and edit readme

* edit readme

* edit readme and overwrite flag desc

* edit readme

* fix merge conflicts and improve maintainability

* improve maintainability

* Rename LICENSE.md to LICENSE

* replaced bat file with ps1 file for building the program

* Addition of feature and prep for v1.1.1

- Added page number support for Pixiv illustrator downloads
- Changed how page number download logic works, now it will download all pages if no page number is specified
    - Mostly affects Pixiv tag search downloads
- Some code clean-ups
- Updated flag descriptions
- Edited README.md with updated flag descriptions

* Graceful exit and improved readability for HTTP requests

- Graceful exit when user press Ctrl+C
    - Will now stop all downloads and delete any incomplete downloads
- Refactored the requests code to use a struct instead of passing around multiple arguments to improve readability

* Optimisations and bug fixes

- Mainly memory optimisations
- Addresses issue [Bug]: Downloading all Fantia Posts from one creator results in 403 [#41](KJHJason/Cultured-Downloader#41) on the main Python program repository

* Bump github.com/fatih/color from 1.14.1 to 1.15.0 in /src

Bumps [github.com/fatih/color](https://github.com/fatih/color) from 1.14.1 to 1.15.0.
- [Release notes](https://github.com/fatih/color/releases)
- [Commits](fatih/color@v1.14.1...v1.15.0)

---
updated-dependencies:
- dependency-name: github.com/fatih/color
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Add #19 and validated for duplicates

* edit README.md and add #21

* edit README.md

* added flag for custom user agent

* maintainability improvement and some optimisations

* some optimisations for reading file

* Maintainability improvement and addition of website

- added struct for json instead of interface
- adding kemono party (wip)

* bug fixes and added debug mode to save a copy of the JSON

* disable debug and minor optimisation

* organise file for kemono (wip)

* kemono prep

* reduce the number of unnecessary API calls

* improve logging

* improved maintainability

* kemono party wip

* improve maintainability + kemono party wip

* improved maintainability

* kemono wip

* kemono party logic mostly done

* integrated kemono cmd

* Added more features, bug fixes, and integration

- Added gdrive downloads to Fantia
- Integrated spinner progress with Kemono Party downloads
- Fixed bug with gdrive client validation

* fixed ugoira dl bug for pixiv mobile

* fixed header and user-agent bug

* improve maintainability

* Create

* fix bug-report.yaml

* fix bug-report.yaml hopefully

* update ver

* improve maintainability

* improve maintainability

* improve maintainability

* added http3 flag checks and improved maintainability

* edit logger prefix

* fixed bug for kemono party downloads

* Bug fixes and minor changes

- Changed logger prefix
- Fixed bugs for Kemono Party downloads
- Add gdrive_api_key flag to Fantia and Kemono Party Cobra CMD (forgor to add it before)
- Other bug fixes like request ctx and other minor changes

* fixed bugs with logging

* fixed bugs

* improve maintainability and censored gdrive api key from being logged

* added flags for logging URLs

* bug fixes and changed dir structures for kemono

* minor optimisations

* bug fixes for fantia

* add comment

* fixed bugs for fanbox downloads and minor changes to err checks

* disable debug mode

* minor edit to ugoira struct

* fixed bugs in verification of cookies

* minor edits

* remove support for 32 bit systems

* fixed runtime bug

* edit cmd desc and readme

* update .rc ver

* edit readme

* fixed a potential race condition and minor edits

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
KJHJason added a commit that referenced this issue Mar 28, 2023
@KJHJason
fixed runtime error in #41
5c26d97
@KJHJason KJHJason mentioned this issue Aug 25, 2023
Closed
@KJHJason
Copy link
Owner

Fixed in https://github.com/KJHJason/Cultured-Downloader/releases/tag/5.0.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@KJHJason KJHJason

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Aaeeschylus @KJHJason