Generate cover [a.k.a. thumbnail/poster] for video manually uploaded #196

deldesir · 2024-06-20T03:02:49Z

Covers are generated using ffmpeg
Tested on Ubuntu 24.04

Display video/bookshelf original URL (or canonical URL if possible!) to all users [ASIDE: upload of offline video works, despite red error "Failed to Move Cover File" in web UI] #107 (comment)

deldesir · 2024-06-20T03:20:52Z

cps/uploader.py

For the same video

deldesir

Safe enough @codewiz?

deldesir · 2024-06-20T03:27:27Z

cps/uploader.py

@@ -326,8 +327,11 @@ def generate_video_cover(tmp_file_path):
    ffmpeg_args = [
        ffmpeg_executable,
        '-i', tmp_file_path,


Defensive programming reminder: Sanitize the file path to prevent command injection

I'll use shlex for now. Should be safe on POSIX compliant shells. (Done in f8c39f2)

I added a general warning that shlex.quote() is not bulletproof:

3a965f8

(Though it's probably good enough for path sanitizing!)

holta · 2024-06-20T11:37:51Z

@codewiz: review quickly if you can?

This PR seeks to resolve:

Display video/bookshelf original URL (or canonical URL if possible!) to all users [ASIDE: upload of offline video works, despite red error "Failed to Move Cover File" in web UI] #107 (comment)

holta · 2024-06-20T12:31:08Z

"2025 request"

While probably not needed in 2024, the "Upload" (Upload to IIAB) button should eventually flag overweight videos that cause congestion for everyone using a school's IIAB WiFi hotspot.

▶️ We should try to warn and educate teachers/parents/students on how to compress their own locally-created overweight videos, BEFORE they're uploaded to IIAB.
▶️ Smartphone videos can usually be compressed 10X or even 20X. This works because smartphones record video at very high resolution (a.k.a. "HD" high definition, which is rarely needed during playback).
▶️ Almost any teacher worldwide can send a just-recorded video to themselves via WhatsApp, in order to force this compression to happen without much hassle.

ASIDE: Occasionally overweight videos arrive from the Internet too — e.g. when the video author/publisher has not bothered to compress their own work for easy/broad distribution...

codewiz · 2024-06-20T14:16:19Z

cps/uploader.py

 def generate_video_cover(tmp_file_path):
    ffmpeg_executable = os.getenv('FFMPEG_PATH', 'ffmpeg')
-    ffmpeg_output_file = os.path.splitext(tmp_file_path)[0] + '.cover.jpg'
+    if not ffmpeg_executable:


Because getenv() returns "ffmpeg" as default, this check will never succeed.

Either remove this check, or change the default to "".

codewiz · 2024-06-20T14:21:09Z

cps/uploader.py

+    sanitized_input_path = sanitize_path(tmp_file_path)
+    output_file_path = os.path.splitext(tmp_file_path)[0] + '.cover.jpg'
+    sanitized_output_path = sanitize_path(output_file_path)


Sanitizing this path is unnecessary because it's not controlled by the user.

The path is constructed at line 301:

tmp_file_path = os.path.join(tmp_dir, md5)

Also, I think process.Popen invokes the program directly, avoiding all problems related to shell expansion.

The path is constructed at line 301:

You mean line 391, ok:

calibre-web/cps/uploader.py

Line 391 in 46f0e6c

tmp_file_path = os.path.join(tmp_dir, md5)

Also, I think process.Popen invokes the program directly, avoiding all problems related to shell expansion.

Clarify a specific recommendation?

Presumably @codewiz is talking about:

calibre-web/cps/subproc_wrapper.py

Line 38 in 07e2a05

return subprocess.Popen(exc_command, shell=False, stdout=sout, stderr=serr, universal_newlines=newlines, env=env) # nosec

@deldesir please go ahead and revise (i.e. tighten up) using Bernie's recommendations, Thanks!

@holta, I did not use Calibre-Web's subprocess wrapper here because I thought it's more clear to just call it and use run from it.

@holta, I did not use Calibre-Web's subprocess wrapper here because I thought it's more clear to just call it and use run from it.

Ah, interesting!

@codewiz is this clean/solid enough, in your opinion?

deldesir

Recommendations applied in 2bdd296

holta

Add explanatory comments back

Awesome! Allowing for easier tweaking of ffmpeg in future, if anybody chooses!

cps/uploader.py

Co-authored-by: Bernie Innocenti <bernie@codewiz.org>

holta · 2024-06-21T19:36:35Z

@deldesir can you tell why @nzola's fresh VM install is not working? (http://box/books shows "502 Bad Gateway")

iiab-diagnostics: https://dpaste.com/FKWVETW67
@deldesir can you eliminate the ffmpeg-related SyntaxWarning: invalid escape sequence going forward? As seen in his logs below:

[2024-06-21 12:08:26,370] WARN {py.warnings:110} /usr/local/calibre-web-py3/lib/python3.12/site-packages/flask_limiter/extension.py:337: UserWarning: Using the in-memory storage for tracking rate limits as no storage was explicitly specified. This is not recommended for production use. See: https://flask-limiter.readthedocs.io#configuring-a-storage-backend for documentation about configuring the storage backend.
warnings.warn(

[2024-06-21 12:08:26,688] WARN {py.warnings:110} /usr/local/calibre-web-py3/cps/uploader.py:338: SyntaxWarning: invalid escape sequence ','
'-vf', 'fps=1,thumbnail,select=gt(scene,0.1),scale=-1:720', # apply filters to avoid black frames and scale

[2024-06-21 12:08:26,822] WARN {py.warnings:110} /usr/local/calibre-web-py3/cps/metadata_provider/lubimyczytac.py:140: SyntaxWarning: invalid escape sequence '?'
characters_to_remove = "?()/"

holta · 2024-06-21T20:26:28Z

FYI @nzola a fresh install of IIAB Calibre-Web onto Ubuntu 24.04 (VM) just worked for me.

As confirmed by iiab-diagnostics: https://dpaste.com/9DWAB47VX

nzola · 2024-06-21T21:40:40Z

FYI @nzola a fresh install of IIAB Calibre-Web onto Ubuntu 24.04 (VM) just worked for me.

As confirmed by iiab-diagnostics: https://dpaste.com/9DWAB47VX

ubuntu@box:~$ pastebinit -b dpaste.com /var/log/nginx/error.log
https://dpaste.com/9JANY2EA6
ubuntu@box:~$ journalctl -u nginx | pastebinit -b dpaste.com
https://dpaste.com/3S2KYHC2A
ubuntu@box:~$ journalctl -u calibre-web | pastebinit -b dpaste.com
https://dpaste.com/3RASF2AMN
ubuntu@box:~$

holta · 2024-06-21T23:49:47Z

@nzola you tried two erroneous URLs:

http://box/books (wrong box, it shows NGINX 1.22.1, which is not possible with an Ubuntu 24.04 VM, so the "502 Bad Gateway" error is irrelevant!) ❌
http://172.23.47.39/home showing "403 Forbidden" ❌

You need to try:

http://172.23.47.39/books ✅

nzola · 2024-06-22T07:50:07Z

@nzola you tried two erroneous URLs:

http://box/books (wrong box, it shows NGINX 1.22.1, which is not possible with an Ubuntu 24.04 VM, so the "502 Bad Gateway" error is irrelevant!) ❌

http://172.23.47.39/home showing "403 Forbidden" ❌

You need to try:

http://172.23.47.39/books ✅

My vm/iiab is working now.
I turned [off] the box Lokole/iiab downstairs and now I can browse with http://172.23.47.39/books as well with http://box/books.

Generate cover for video manually updated

d81458a

deldesir added the enhancement New feature or request label Jun 20, 2024

deldesir requested a review from holta June 20, 2024 03:02

deldesir self-assigned this Jun 20, 2024

deldesir changed the title ~~Generate cover for video manually updated~~ Generate cover for video manually uploaded Jun 20, 2024

Enhance frame selection

0416359

deldesir commented Jun 20, 2024

View reviewed changes

cps/uploader.py

Copy link

Collaborator Author

deldesir Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the same video

deldesir added 3 commits June 19, 2024 23:24

Add explanatory comments

9ef5a1d

Sanitize input and output path

f8c39f2

Set ffmpeg failure as error

46f0e6c

deldesir commented Jun 20, 2024

View reviewed changes

deldesir added the bug Something isn't working label Jun 20, 2024

holta changed the title ~~Generate cover for video manually uploaded~~ Generate cover [a.k.a. thumbnail/poster] for video manually uploaded Jun 20, 2024

holta mentioned this pull request Jun 20, 2024

Display video/bookshelf original URL (or canonical URL if possible!) to all users [ASIDE: upload of offline video works, despite red error "Failed to Move Cover File" in web UI] #107

Open

codewiz reviewed Jun 20, 2024

View reviewed changes

Enhance cover/poster generation

2bdd296

deldesir commented Jun 21, 2024

View reviewed changes

deldesir and others added 4 commits June 20, 2024 21:16

Add explanatory comments back

3379ee8

uploader.py: Just a warning about shlex.quote() risks

3a965f8

Remove unnecessary sanitizing function

90f0b50

Preserve shlex for future use of sanitize_path, even if commented out (:

cbaad2b

holta reviewed Jun 21, 2024

View reviewed changes

codewiz reviewed Jun 21, 2024

View reviewed changes

cps/uploader.py Outdated Show resolved Hide resolved

deldesir and others added 3 commits June 21, 2024 12:01

use subprocess directly

e7d9636

Co-authored-by: Bernie Innocenti <bernie@codewiz.org>

Remove subprocess import

686d6a3

Import subprocess

10296a7

holta merged commit 121feb5 into iiab:master Jun 21, 2024

EMG70 mentioned this pull request Jun 28, 2024

Uploading videos via Calibre-Web fails - 500 Internal Server Error [if filename contains square brackets] #203

Closed

deldesir deleted the deldesir-patch-48 branch July 1, 2024 21:40

holta mentioned this pull request Jul 6, 2024

Uploading of a video results in no cover/thumbnail — "Failed to Move Cover file" — "[Errno 2] No such file of directory" #214

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate cover [a.k.a. thumbnail/poster] for video manually uploaded #196

Generate cover [a.k.a. thumbnail/poster] for video manually uploaded #196

deldesir commented Jun 20, 2024 •

edited by holta

Loading

deldesir Jun 20, 2024

deldesir left a comment

deldesir Jun 20, 2024

deldesir Jun 20, 2024

holta Jun 21, 2024

holta commented Jun 20, 2024 •

edited

Loading

holta commented Jun 20, 2024 •

edited

Loading

codewiz Jun 20, 2024

deldesir Jun 20, 2024

codewiz Jun 20, 2024

codewiz Jun 20, 2024

holta Jun 20, 2024

holta Jun 20, 2024

holta Jun 20, 2024

deldesir Jun 21, 2024

holta Jun 21, 2024

deldesir left a comment •

edited

Loading

holta left a comment •

edited

Loading

holta commented Jun 21, 2024

holta commented Jun 21, 2024

nzola commented Jun 21, 2024 •

edited by holta

Loading

holta commented Jun 21, 2024

nzola commented Jun 22, 2024 •

edited by holta

Loading

Generate cover [a.k.a. thumbnail/poster] for video manually uploaded #196

Generate cover [a.k.a. thumbnail/poster] for video manually uploaded #196

Conversation

deldesir commented Jun 20, 2024 • edited by holta Loading

Choose a reason for hiding this comment

deldesir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holta commented Jun 20, 2024 • edited Loading

holta commented Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deldesir left a comment • edited Loading

Choose a reason for hiding this comment

holta left a comment • edited Loading

Choose a reason for hiding this comment

holta commented Jun 21, 2024

holta commented Jun 21, 2024

nzola commented Jun 21, 2024 • edited by holta Loading

holta commented Jun 21, 2024

nzola commented Jun 22, 2024 • edited by holta Loading

deldesir commented Jun 20, 2024 •

edited by holta

Loading

holta commented Jun 20, 2024 •

edited

Loading

holta commented Jun 20, 2024 •

edited

Loading

deldesir left a comment •

edited

Loading

holta left a comment •

edited

Loading

nzola commented Jun 21, 2024 •

edited by holta

Loading

nzola commented Jun 22, 2024 •

edited by holta

Loading