Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor lb-wrapper for command separation #110

Merged
merged 15 commits into from
Jan 31, 2024
Merged
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 19 additions & 10 deletions scripts/lb-wrapper
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@ XKLB_EXECUTABLE="${XKLB_EXECUTABLE:-lb}"
VERBOSITY="-vv"
TMP_DOWNLOADS_DIR="/library/downloads/calibre-web"
XKLB_DB_FILE="/library/calibre-web/xklb-metadata.db"
URL="$1"
XKLB_INTERNAL_CMD="$1"
URL="$2" # Use $2 for the URL argument

# Or download largest possible HD-style / UltraHD videos, to try to force
# out-of-memory "502 Bad Gateway" for testing of issues like #37 and #79
Expand All @@ -16,7 +17,6 @@ XKLB_FULL_CMD="${XKLB_EXECUTABLE} tubeadd ${XKLB_DB_FILE} ${URL} ${VERBOSITY} \
&& ${XKLB_EXECUTABLE} dl ${XKLB_DB_FILE} --prefix ${TMP_DOWNLOADS_DIR} \
--video ${URL} ${FORMAT_OPTIONS} --write-thumbnail ${VERBOSITY}"


mkdir -p ${TMP_DOWNLOADS_DIR}

# Function to log messages e.g. w/ level "Info", "Warning" or "Error"
Expand All @@ -26,8 +26,8 @@ log() {
echo "$(date +'%Y-%m-%d %H:%M:%S') - [$level] $message" | tee -a ${LOG_FILE}
}

if [ $# -eq 0 ]; then
log "Error" "No arguments provided. Please provide a URL to download."
if [ $# -ne 2 ]; then
log "Error" "Two arguments are required. Please provide a command (tubeadd/dl) and a URL."
exit 1
fi

Expand All @@ -40,21 +40,30 @@ fi
# exit 1
# fi

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validation on Lines 43-44 should definitely be restored in my opinion:

if ! command -v "${XKLB_EXECUTABLE}"; then
    log "Error" "xklb could not be found. Please install xklb and try again."

(...followed by a blank line if possible!)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

if ! command -v "${XKLB_EXECUTABLE}"; then
log "Error" "xklb could not be found. Please install xklb and try again."
if [ "$XKLB_INTERNAL_CMD" == "tubeadd" ]; then
XKLB_FULL_CMD="${XKLB_FULL_CMD%%&&*}"
elif [ "$XKLB_INTERNAL_CMD" == "dl" ]; then
XKLB_FULL_CMD="${XKLB_FULL_CMD##*&& }"
else
log "Error" "Invalid command. Please choose 'tubeadd' or 'dl'."
exit 1
fi

# 2024-01-21: Not needed as XKLB reports its own version# to /var/log/xklb.log
# log "Info" "xklb version: $(${XKLB_EXECUTABLE} --version)"

log "Info" "Running xklb commands: ${XKLB_FULL_CMD}"
# 2024-01-24: Not needed as we don't discard the database anymore
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please (re)delete lines 55-58 per JC suggestion (:

As was merged with PR #109 here:

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

# if mv ${XKLB_DB_FILE} ${XKLB_DB_FILE}.$(date +%F_%T_%Z) 2> /dev/null; then
# log "Info" "Old ${XKLB_DB_FILE} moved aside."
# fi

log "Info" "Running xklb command: ${XKLB_FULL_CMD}"

# >(...) "process substitution" explained at https://unix.stackexchange.com/a/324170
# 1>&2 redirect back-to-STDERR to avoid nested (repeat) logging, explained at https://stackoverflow.com/a/15936384
eval "${XKLB_FULL_CMD}" \
> >(while read -r line; do if [[ $line == downloading* ]]; then echo "$line"; else log "Info" "$line"; fi; done) \
2> >(while read -r line; do if [[ $line == downloading* ]]; then echo "$line"; else log "Debug" "$line" 1>&2; fi; done) &
> >(while read -r line; do if [[ $line == downloading* ]]; then echo "$line"; else log "Info" "$line"; fi; done) \
2> >(while read -r line; do if [[ $line == downloading* ]]; then echo "$line"; else log "Debug" "$line" 1>&2; fi; done) &
# 2024-01-11: HOW THIS WORKS...
# 0) xklb sends a flood of yt-dlp status message lines like "downloading 59.8% 2.29MiB/s" to STDERR.
# 1) Then, "2> >(...)" reroutes (only those raw lines!) to STDIN, instead of logging them (as "Debug" lines).
Expand All @@ -71,7 +80,7 @@ rc=$?

# Check return code (exit status)
if [ $rc -eq 0 ]; then
log "Info" "lb-wrapper's xklb commands (download) completed successfully."
log "Info" "lb-wrapper's xklb command (${XKLB_INTERNAL_CMD}) completed successfully."
else
log "Error" "Error $rc occurred while running lb-wrapper's xklb commands."
exit 1
Expand Down