Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wget exiting with error - not creating folder? #12

Closed
Rob-T opened this issue Sep 25, 2020 · 8 comments
Closed

wget exiting with error - not creating folder? #12

Rob-T opened this issue Sep 25, 2020 · 8 comments
Labels
bug Something isn't working

Comments

@Rob-T
Copy link

Rob-T commented Sep 25, 2020

  • magzdb version: 1.1.2
  • Python version: 3.7.3
  • Operating System: Debian 10.5

Description

When running with --downloader wget, magzdb exits with error. It seems to be that wget isn't creating the correct folders.

What I Did

I ran the following,

user01@s107487:~/books/temp_magazines$ /home/user01/.local/bin/magzdb -i 1593 -e 4056530 --downloader wget --debug
2020-09-25 06:01:11.903 | INFO     | magzdb.magzdb:download:199 - Found 1 editions of Everyday_practical_electronics
2020-09-25 06:01:11.903 | INFO     | magzdb.magzdb:download:204 - Downloading year 2020 issue 8
2020-09-25 06:01:11.904 | DEBUG    | magzdb.magzdb:_print:52 - Issue ID: 4056530
2020-09-25 06:01:11.988 | DEBUG    | magzdb.magzdb:_print:52 - Download Link ID: 675386
2020-09-25 06:01:12.084 | DEBUG    | magzdb.magzdb:_print:52 - Download URL: http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf
2020-09-25 06:01:12.085 | DEBUG    | magzdb.magzdb:_print:52 - wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"
Traceback (most recent call last):
  File "/home/user01/.local/bin/magzdb", line 10, in <module>
    sys.exit(main())
  File "/home/user01/.local/lib/python3.7/site-packages/magzdb/cli.py", line 96, in main
    filter=args.filter,
  File "/home/user01/.local/lib/python3.7/site-packages/magzdb/magzdb.py", line 241, in download
    subprocess.run(command)
  File "/usr/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"': 'wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"'

If I first run the command without --downloader, as expected the file downloads and I get...

user01@s107487:~/books/temp_magazines$ /home/user01/.local/bin/magzdb -i 1593 -e 4056530 --debug
2020-09-25 06:50:37.830 | WARNING  | magzdb.cli:main:81 - Use of external downloader like wget or aria2 is recommended
2020-09-25 06:50:38.245 | INFO     | magzdb.magzdb:download:199 - Found 1 editions of Everyday_practical_electronics
2020-09-25 06:50:38.245 | INFO     | magzdb.magzdb:download:204 - Downloading year 2020 issue 8
2020-09-25 06:50:38.245 | DEBUG    | magzdb.magzdb:_print:52 - Issue ID: 4056530
2020-09-25 06:50:38.328 | DEBUG    | magzdb.magzdb:_print:52 - Download Link ID: 675386
2020-09-25 06:50:38.448 | DEBUG    | magzdb.magzdb:_print:52 - Download URL: http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf
2020-09-25 06:50:38.926 | DEBUG    | magzdb.magzdb:_print:52 - Downloading to /home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf

Without removing the file/folder created in the last step, If I then run the wget command that DEBUG printed in the first attempt,

user01@s107487:~/books/temp_magazines$ wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"
--2020-09-25 06:51:41--  http://file.magzdb.org/ul/1593/Practical%20Electronics%20-%202020-08.pdf
Resolving file.magzdb.org (file.magzdb.org)... 84.39.241.145
Connecting to file.magzdb.org (file.magzdb.org)|84.39.241.145|:80... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

    The file is already fully retrieved; nothing to do.

Again, without deleting the file/folder created and running the first command again, it exits with the same error,

user01@s107487:~/books/temp_magazines$ /home/user01/.local/bin/magzdb -i 1593 -e 4056530 --downloader wget --debug
2020-09-25 06:55:39.899 | INFO     | magzdb.magzdb:download:199 - Found 1 editions of Everyday_practical_electronics
2020-09-25 06:55:39.900 | INFO     | magzdb.magzdb:download:204 - Downloading year 2020 issue 8
2020-09-25 06:55:39.900 | DEBUG    | magzdb.magzdb:_print:52 - Issue ID: 4056530
2020-09-25 06:55:39.976 | DEBUG    | magzdb.magzdb:_print:52 - Download Link ID: 675386
2020-09-25 06:55:40.090 | DEBUG    | magzdb.magzdb:_print:52 - Download URL: http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf
2020-09-25 06:55:40.090 | DEBUG    | magzdb.magzdb:_print:52 - wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"
Traceback (most recent call last):
  File "/home/user01/.local/bin/magzdb", line 10, in <module>
    sys.exit(main())
  File "/home/user01/.local/lib/python3.7/site-packages/magzdb/cli.py", line 96, in main
    filter=args.filter,
  File "/home/user01/.local/lib/python3.7/site-packages/magzdb/magzdb.py", line 241, in download
    subprocess.run(command)
  File "/usr/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"': 'wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"'
@issue-label-bot
Copy link

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.92. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

@issue-label-bot issue-label-bot bot added the bug Something isn't working label Sep 25, 2020
@skyme5
Copy link
Owner

skyme5 commented Sep 25, 2020

@Rob-T, I think the issue is related to wget, since wget would not create directory when downloading.
In your case directory Everyday_practical_electronics needs to exist.

To resolve this issue you can create the Everyday_practical_electronics or use aria2 downloader until the fix is released.

@skyme5
Copy link
Owner

skyme5 commented Sep 25, 2020 •
edited
Loading

@Rob-T I've fixed this bug in the latest release v1.1.3
Thank you for this report.

@Rob-T
Copy link
Author

Rob-T commented Sep 25, 2020

I've updated to v1.1.3 and also installed aria2, but am still having issues (with both wget and aria2 actually).

With wget...

user01@s107487:~/books/temp_magazines$ /home/user01/.local/bin/magzdb -i 1593 -e 4056530 --downloader wget --debug
2020-09-25 08:55:56.433 | INFO     | magzdb.magzdb:download:202 - Found 1 editions of Everyday_practical_electronics
2020-09-25 08:55:56.433 | INFO     | magzdb.magzdb:download:207 - Downloading year 2020 issue 8
2020-09-25 08:55:56.433 | DEBUG    | magzdb.magzdb:_print:52 - Issue ID: 4056530
2020-09-25 08:55:56.606 | DEBUG    | magzdb.magzdb:_print:52 - Download Link ID: 675386
2020-09-25 08:55:56.699 | DEBUG    | magzdb.magzdb:_print:52 - Download URL: http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf
2020-09-25 08:55:56.699 | DEBUG    | magzdb.magzdb:_print:52 - wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"
Traceback (most recent call last):
  File "/home/user01/.local/bin/magzdb", line 10, in <module>
    sys.exit(main())
  File "/home/user01/.local/lib/python3.7/site-packages/magzdb/cli.py", line 96, in main
    filter=args.filter,
  File "/home/user01/.local/lib/python3.7/site-packages/magzdb/magzdb.py", line 244, in download
    subprocess.run(command)
  File "/usr/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"': 'wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"'

With Aria2...

user01@s107487:~/books/temp_magazines$ /home/user01/.local/bin/magzdb -i 1593 -e 4056530 --downloader aria2 --debug
2020-09-25 08:56:29.388 | INFO     | magzdb.magzdb:download:202 - Found 1 editions of Everyday_practical_electronics
2020-09-25 08:56:29.388 | INFO     | magzdb.magzdb:download:207 - Downloading year 2020 issue 8
2020-09-25 08:56:29.388 | DEBUG    | magzdb.magzdb:_print:52 - Issue ID: 4056530
2020-09-25 08:56:29.465 | DEBUG    | magzdb.magzdb:_print:52 - Download Link ID: 675386
2020-09-25 08:56:29.534 | DEBUG    | magzdb.magzdb:_print:52 - Download URL: http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf
2020-09-25 08:56:29.534 | DEBUG    | magzdb.magzdb:_print:52 - aria2c -c --dir="/home/user01/books/temp_magazines/Everyday_practical_electronics" --out="Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"
Traceback (most recent call last):
  File "/home/user01/.local/bin/magzdb", line 10, in <module>
    sys.exit(main())
  File "/home/user01/.local/lib/python3.7/site-packages/magzdb/cli.py", line 96, in main
    filter=args.filter,
  File "/home/user01/.local/lib/python3.7/site-packages/magzdb/magzdb.py", line 244, in download
    subprocess.run(command)
  File "/usr/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'aria2c -c --dir="/home/user01/books/temp_magazines/Everyday_practical_electronics" --out="Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"': 'aria2c -c --dir="/home/user01/books/temp_magazines/Everyday_practical_electronics" --out="Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"'

The folder /home/user01/books/temp_magazines/Everyday_practical_electronics exists when the above was run and currently has the file Practical_Electronics_-_2020-08.pdf in it. If I remove the folder and run it again, the script creates the folder but the download does not happen (the folder is empty).

@skyme5
Copy link
Owner

skyme5 commented Sep 25, 2020

Can you check if there is a permission issue? Unfortunately I don't have a linux system.

@Rob-T
Copy link
Author

Rob-T commented Sep 25, 2020

I'm no linux master, but it looks like folders and script both have owner and group as user01, and have drwxr-xr-x attributes (read, write, execute). There doesn't seem to be an issue that I can see. The script has no issues other than when using a downloader.

@skyme5
Copy link
Owner

skyme5 commented Sep 25, 2020

@Rob-T Can you check if error is resolved by passing shell=True in magzdb/magzdb.py#L244 like this,

- subprocess.run(command)
+ subprocess.run(command, shell=True)

In your case the file is located at "/home/user01/.local/lib/python3.7/site-packages/magzdb/magzdb.py"

@Rob-T
Copy link
Author

Rob-T commented Sep 25, 2020

Yes, that seems to fix it. I ran it with both the folder/file existing and not, and the result was a success either way.

user01@s107487:~/books/temp_magazines$ /home/user01/.local/bin/magzdb -i 1593 -e 4056530 --downloader wget --debug
2020-09-25 13:06:01.793 | INFO     | magzdb.magzdb:download:202 - Found 1 editions of Everyday_practical_electronics
2020-09-25 13:06:01.794 | INFO     | magzdb.magzdb:download:207 - Downloading year 2020 issue 8
2020-09-25 13:06:01.794 | DEBUG    | magzdb.magzdb:_print:52 - Issue ID: 4056530
2020-09-25 13:06:01.883 | DEBUG    | magzdb.magzdb:_print:52 - Download Link ID: 675386
2020-09-25 13:06:01.974 | DEBUG    | magzdb.magzdb:_print:52 - Download URL: http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf
2020-09-25 13:06:01.974 | DEBUG    | magzdb.magzdb:_print:52 - wget -c -O "/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf" "http://file.magzdb.org/ul/1593/Practical Electronics - 2020-08.pdf"
--2020-09-25 13:06:01--  http://file.magzdb.org/ul/1593/Practical%20Electronics%20-%202020-08.pdf
Resolving file.magzdb.org (file.magzdb.org)... 84.39.241.145
Connecting to file.magzdb.org (file.magzdb.org)|84.39.241.145|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16782219 (16M) [application/pdf]
Saving to: ?/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf?

/home/user01/books/temp_magazines/Everyda 100%[====================================================================================>]  16.00M   336KB/s    in 55s     

2020-09-25 13:06:57 (295 KB/s) - ?/home/user01/books/temp_magazines/Everyday_practical_electronics/Practical_Electronics_-_2020-08.pdf? saved [16782219/16782219]

Good work!!

@skyme5 skyme5 closed this as completed in 0a4bf1a Sep 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Rob-T @skyme5