Skip to content

Home

Jump to bottom Edit New page
Wikinaut edited this page Mar 7, 2024 · 300 revisions

Code or bash sniplets and useful scripts

Table of Contents

Linux

Rename filenames using Regular Expressions

Rename filenames (without spaces) "from-*.*" to "to-*.*" using Regular Expressions 

  for i in *; do mv "$i" "`echo $i | sed "s/from-/to-/"`"; done

Rename filenames from file creation dates

Rename filenames (without spaces) to filenames with creation date (YYYYMMDD-HHMMSS) 

  for i in *; do mv "$i" "`stat -c %Y $i | awk '{print strftime("%Y%m%d-%H%M%S", $1)}'`".${i#*.}; done

Rename media from their image creation dates 

  exiftool -d %Y%m%d-%H%M%S%%-c.%%e "-FileName<DateTimeOriginal" <filename>

Update any photo that doesn't have DateTimeOriginal to have it based on file modify date 

  exiftool '-datetimeoriginal<filemodifydate' -if '(not $datetimeoriginal or ($datetimeoriginal eq "0000:00:00 00:00:00")) and ($filetype eq "JPEG")' .

To move all files to elsewhere, but not folders

If you are interested in moving all files (but not folders) from Downloads folder to Videos folder, use this command 
find ~/Downloads/ -type f -print0 | xargs -0 mv -t ~/Videos

delete (or mark with ".keep") doublure files with same md5 but the oldest

find . -type f -iname "*.umap" -print0 | xargs -0 -i md5sum {} | sort | uniq -w 32 --all-repeated=separate | uniq -w 32 | awk 'NF {print $2}' | xargs -I{} echo mv {} {}.keep

add a prefix and directory name to filenames 

for folder in *; do
  if [ -d "$folder" ]; then
    cd "$folder"
    for file in *; do
      mv -- "$file" "${folder}_$file"
    done
    cd ..
  fi
done

Renumbering files with leading zeroes 

  j=0;for i in *;do ((j++));mv $i $(printf "%04d\n" $j).jpg; done

Rename files with dates dd.mm.yyyy to yyyymmdd_ 

  for i in *; do mv "$i" "`echo $i | sed -e "s/^\(.*\)\([0-9]\{2\}\)\.\([0-9]\{2\}\)\.\([0-9]\{4\}\)\(.*\)$/\4\3\2_\0/;s/PDF/pdf/"`"; done

copy all .pdf files (also with spaces in filenames) to cwd 

 IFS=$'\n';for i in $(find . -type f -iname "*.pdf"); do cp "$i" .; done

Change spaces in filenames to underscores recursively

requires /usr/bin/rename (a Perl script): 
 sudo find . -depth -name "* *" -execdir rename 's/ /_/g' "{}" \;

How to add line numbers "#### " to every line in a text file 

nl -w4 -n rz -s ' ' text.txt

-w4: 4 digits
-n rz: leading zeroes
-s ' ': separated by a single space

Delete files older than …

Delete files older than 30 days 

 find . -mtime +30 -print0 | xargs -0 rm
 find . -mtime +30 -print0 | xargs -0 rm -Rf (recursive, force = dangerous)

See http://stackoverflow.com/questions/16758525/use-xargs-with-filenames-containing-whitespaces for the use of -print0 and -0 options to allow filenames with white spaces.

Look up files with a certain date 

 find .  -type f -name "*" -newermt 2016-01-17 ! -newermt 2016-01-18 -print0 | xargs -0 --no-run-if-empty ls -lsa

Excluding Directories from 'locate' (updatedb)

Edit the /etc/updatedb.conf file. Add the paths to the PRUNEPATHS line and then re-run updatedb. For example: 

 PRUNEPATHS="/tmp /timeshift /var/spool /media /mnt"

grep in gz files 

find . -name "*.gz" -print0 | xargs -0 zgrep -i -E "(ne.dle1|needle2)"

Show heavy directories

Show "heavy" directories deeper than /this-dir.... ordered by size, show size in Megabytes (M) or Gigabytes (G) 

 sudo du -BM / | sort -nr | head -n 40
 sudo du -BG / | sort -nr | head -n 40

Show top level directory size only, and print on printer 

 du -h --max-depth=1 | lpr

Show number of files in a subdirectory 

 find /this-dir -type f | wc -l
 find . -type f | wc -l

List files only 

 ls -p | grep -v /

List only one level of subdirectories 

 find . -maxdepth 1 -type d

concatenate files sorted 

 cat `ls -- files*.ext | sort -V` > files-all.ext

sort files by dates dd.mm.yyyy in the filename 

ls | sed -e 's/^\(.*\)\([0-9]\{2\}\)\.\([0-9]\{2\}\)\.\([0-9]\{4\}\)\(.*\)$/\4\3\2_\0/;s/PDF/pdf/' | sort -r

Delete duplicate lines - make files unique 

sort infile | uniq > outfile

List files in file1 which are not in file2 

comm -23 <(sort file1) <(sort file2)

Zip all files older than …

Zip (or zip and move) all files older than x days
Usage: 

 ./zip-all-files-older-than.sh number-of-days [move]

#!/bin/bash
# 20121217

if [ $1 ] ; then

  UNTIL="$(date "+%Y%m%d" -d "$1 days ago")"

  if [ $2 ] && [ $2 == move ] ; then
    echo "zip and move all files older than $1 days (until $UNTIL)"
    find . -daystart -mtime +$(($1-1)) | xargs zip -m files-$UNTIL.zip
    ls -lh files-$UNTIL.zip
    df -h
  else
    echo "zip all files older than $1 days (until $UNTIL)"
    find . -daystart -mtime +$(($1-1)) | xargs zip files-$UNTIL.zip
    ls -lh files-$UNTIL.zip
    df -h
  fi

else
   echo zip all files older than n days
   echo Usage: $0 n [move]
fi

exit

Number of CPUs, Cores 

echo; echo "PHYSICAL CPUs:"; 
grep "physical id" /proc/cpuinfo | sort | uniq | wc -l;

echo; echo "CPU Cores:";
grep "cpu cores" /proc/cpuinfo | uniq;

echo; echo  "VIRTUAL PROCESSORS:";
grep "^processor" /proc/cpuinfo

Show version of operation system, kernel and distribution

see also http://www.sysadminslife.com/linux/quicktipp-installierte-linux-distribution-anzeigen-und-version-herausfinden/
https://linuxconfig.org/check-what-debian-version-you-are-running-on-your-linux-system
 cat /proc/version
 cat /etc/os-release
 uname -a
 uname -mrs
 lsb_release -a
 hostnamectl

Show BIOS version 

 sudo dmidecode -t bios

remount read/write 

mount -o remount,rw /media/user/device /media/user/device

flush DNS cache 

sudo systemd-resolve --flush-caches

CPU benchmark for the command line 

 time echo "scale=4000; a(1)*4" | bc -l

or use sysbench, see http://developer-blog.net/hardware/benchmark/ 

 sysbench --test=cpu --cpu-max-prime=20000 run

or vanitygen.

Full system backup with rsync

Rsync has two major modes of operation: the modes are dependent on the absence or presence of the trailing slash on the source directory. You need to add the trailing slash, otherwise, a folder called "x" is created inside the "x" folder that we specify as destination.

Boot requirements: see also https://wiki.archlinux.org/index.php/Full_system_backup_with_rsync#Boot_requirements 
 ; show overall progress info and transfer speed instead of huge list of files.
 rsync -aAX --info=progress2 --log-file="/home/benutzer/rsync.log" --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /path-to-backup/ /path/to/backup/folder

 ; show huge list of files
 rsync -aAXv --log-file="/home/benutzer/rsync.log" --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /path-to-backup/ /path/to/backup/folder

 sudo rsync --dry-run -aAXv --log-file="/root/rsync.log" --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/home/**/.thumbnails/*","/home/**/Trash/*","/home/**/.cache"} / .

 ; Using rsync to move (not copy) files between directories
 ; https://unix.stackexchange.com/questions/43957/using-rsync-to-move-not-copy-files-between-directories
 ; https://wiki.ubuntuusers.de/rsync/
 ; https://wiki.archlinux.org/index.php/rsync
 rsync -avh -e ssh --progress --log-file="/home/benutzer/rsync.log" --remove-source-files /home/user/mystuff/* user@external.server:/path/to/backup
 find . -depth -type d -empty -delete

TYM Time Machine using rsync

Convert *.rpm to *.deb packages

  • for x64 CPUs and with execution of embedded scripts
 sudo alien -c --target=amd64 package.rpm

dpkg: how to remove all packages marked as rc

https://linuxprograms.wordpress.com/2010/05/12/remove-packages-marked-rc/

using Timeshift with RAID 

just use
sudo timeshift --list-devices to list availables devices (including raid partitions)
and then
sudo timeshift --snapshot-device /dev/mdXXX with correct device name to set your backup destination

The interface still doesn't list the partition but the backup seems ok

sed

delete string at line ends and remove the line break 

sed ':a;N;$!ba;s/string\n//g' <file>

RAID1

How to replace a failed disk

MySQL

How to set UTF-8 as default character set

How to reset your root MySQL password

MySQL: add database and database user

When you use MySQL or MariaDB, you can create a new database "owncloud" and a new user "ownclouduser" with these lines: 
shell> mysql -u root -p
mysql> CREATE USER 'ownclouduser'@'localhost' IDENTIFIED BY 'password';
       CREATE DATABASE IF NOT EXISTS owncloud;
       GRANT ALL PRIVILEGES ON owncloud.* TO 'ownclouduser'@'localhost' IDENTIFIED BY 'password';
or
       GRANT CREATE,ALTER,SELECT,INSERT,UPDATE,DELETE ON owncloud.* TO 'ownclouduser'@'localhost' IDENTIFIED BY 'password';

       FLUSH PRIVILEGES;
       EXIT;

Dump MySQL table schema (table structure) without data 

mysqldump -u root -p --no-data dbname > schema.sql

MySQL dump and zip all databases 

#!/bin/bash

echo dump and zip all databases
# http://serverfault.com/questions/331057/mysqldump-unknown-table-engine-performance-schema
# http://www.sysadminslife.com/linux/mysql-warning-skipping-the-data-of-table-mysql-event-debian-ubuntu/
NOW="$(date "+%Y%m%d")"
mysqldump -u root -p \
--all-databases \
--events \
--default-character-set=utf8 --skip-set-charset \
--ignore-table=performance_schema.cond_instances \
--ignore-table=performance_schema.events_waits_current \
--ignore-table=performance_schema.cond_instances \
--ignore-table=performance_schema.events_waits_history \
--ignore-table=performance_schema.events_waits_history_long  \
--ignore-table=performance_schema.events_waits_summary_by_instance \
--ignore-table=performance_schema.events_waits_summary_by_thread_by_event_name \
--ignore-table=performance_schema.events_waits_summary_global_by_event_name \
--ignore-table=performance_schema.file_instances \
--ignore-table=performance_schema.file_summary_by_event_name \
--ignore-table=performance_schema.file_summary_by_instance  \
--ignore-table=performance_schema.mutex_instances \
--ignore-table=performance_schema.performance_timers \
--ignore-table=performance_schema.rwlock_instances \
--ignore-table=performance_schema.setup_consumers \
--ignore-table=performance_schema.setup_instruments \
--ignore-table=performance_schema.setup_timers \
--ignore-table=performance_schema.threads | gzip - > mysqldump-alldatabases_$NOW.gz

add debian-sys-maint user

poor men's file encryption e.g. for database dumps

Sequence to generate a compressed and encrypted database dump: 

mysqldump <dbname> -u root -p | gzip - | openssl aes-256-cbc -salt -k <password> -out <dbdump>

Example: 

mysqldump test -u root -p | gzip - | openssl aes-256-cbc -salt -k SeCrEt -out test.gz.aes

This creates test.gz.aes.

To decode this on our server use 

gunzip <dbdump> --stdout | openssl aes-256-cbc -d -k <password> > <dbname>.sql

Example: 

gunzip test.gz.aes --stdout | openssl aes-256-cbc -d -k SeCrEt > test.sql

Import the database dump in MySQL: 

mysql <dbname> -u root -p < <dbname>.sql

or, run the command directly in MySQL: 

mysql> use <dbname>;
mysql> SOURCE <dbname>.sql

git

prune history: make current commit the only (initial) commit in a Git repository 

git checkout --orphan newBranch
git add -A  # Add all files and commit them
git commit
git branch -D master  # Deletes the master branch
git branch -m master  # Rename the current branch to master
git push -f origin master  # Force push master branch to github
git gc --aggressive --prune=all     # remove the old files

borg (borgbackup)

Multi-archive search for $pattern in $repo 

for i in $(borg list --short $repo);do borg list --short $repo::$i "re:.*$pattern";done
for i in $(borg list --short $repo);do borg list --format="{archivename} {mode} {user:6} {group:6} {size:8d} {isomtime} {path}{extra}{NEWLINE}" $repo::$i "re:.*$pattern";done

borg mount -o versions 

 borg mount -o versions $repo /tmp/mountpoint

ownCloud

convert Sqlite to MySQL database

Read https://github.com/owncloud/core/issues/5139#issuecomment-28285731 about resulting issues "Failed to update | Multiple primary key for oc_locks table" from the Sqlite to MySQL conversion:

GnuPG

GnuPG (gpg) undocumented commands 

# show all built-in options (~320 options)
gpg --dump-options
gpg --dump-options | wc -l

# show the fingerprint of a keyfile without importing it first
# --with-fingerprint is an option, and not a regular command. But the default command (list key data) is used for the keyfile
gpg --with-fingerprint KEYFILE

# show internal key structures
gpg --list-packets KEYFILE

# generate public key from imported old PGP private keys
gpg --import --allow-non-selfsigned-uid KEYFILE

# use an old key for encryption
gpg --e --allow-non-selfsigned-uid recipient

# generate message digests (checksum, file hashes) in all available formats
# the "*" must be quoted to avoid command line expansion (globbing) on operating systems like Linux
gpg --print-md "*" FILENAME

# set better hash defaults in ~/.gnupg/gpg.conf 
personal-digest-preferences SHA512
cert-digest-algo SHA512
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

# create and verify a detached ascii armored signature file for FILENAME
gpg --detach-sign -a FILENAME
gpg --verify -v FILENAME.asc

# clear the passphrase cache.  Call that before leaving the computer.
gpgconf --reload gpg-agent

How to compile GnuPG (gpg) from the sources

see https://www.gnupg.org/download/cvs_access.html and http://git.gnupg.org/
This requires automake version 1.11 (see http://lists.gnupg.org/pipermail/gnupg-devel/2014-June/028537.html and see inside the script how to compile automake 1.11). The exemplary script shows what you need to compile GnuPG; a few commands require root privileges. 
#!/bin/bash

echo clone GnuPG with all dependencies from git and compile

# Before actually starting the compilation, 
# the following two lines can create an ad-hoc backup of the libraries
# with the date-time as part of the backup filename.
# These files can be deleted afterwards if everything went fine.
#
# NOW="$(date "+%Y%m%d")"
# tar -czf git-$NOW.tgz npth libgpg-error libgcrypt libassuan libksba pinentry gnupg

# GnuPG compilation requires automake version 1.11
#
# please make sure that automake 1.11 is installed (automake-1.11 on Debian). 
# Newer automakes are not compatible, because
# - these have parallel regression tests enabled by default
# - old automakes do not have an option for disabling
#
# if you have a recent automake:
# for compiling GnuPG install automake 1.11 using the following steps:
#
# wget http://ftp.gnu.org/gnu/automake/automake-1.11.6.tar.gz
# wget http://ftp.gnu.org/gnu/automake/automake-1.11.6.tar.gz.sig
# wget ftp://ftp.gnu.org/gnu/gnu-keyring.gpg
# gpg --verify --keyring ./gnu-keyring.gpg automake-1.11.6.tar.gz.sig
# tar -xzf automake-1.11.6.tar.gz
# cd automake-1.11.6/
# ./configure 
# make
# AUTOMAKE_SUFFIX=1.11
# check if automake version == 1.11
# automake --version

rm -rf npth libgpg-error libgcrypt libassuan libksba pinentry gnupg

for i in npth libgpg-error libgcrypt libassuan libksba pinentry gnupg ; do

  git clone --depth 1 git://git.gnupg.org/$i.git

  # Alternative location in case the other cannot be accessed:
  # git clone http://git.spodhuis.org/gnupg.org/$i.git
  
  cd $i
  ./autogen.sh

  if [ $i -eq gnupg ] ; then
    ./configure --enable-maintainer-mode
  else
    ./configure --enable-maintainer-mode
  fi

  make
  sudo make install
  cd ..

done

sudo pkill gpg-agent

cd /usr/local/src
./make-gnupg.sh

# check the compiled gpg2 version
cd /usr/local/src/gnupg/g10
./gpg2 --version

how to prune (trim) your public key

(not tested)

  1. backup the .gnupg directory
  2. export your key and store it in a safe place
  3. start key-editor: gpg --edit-key YourID
  4. selected expired subkeys (e.g. key 1), the selected subkey is marked with "*"
  5. delete this subkey with "delkey"
  6. remove with "uid number" and "deluid" not needed further User-IDs
  7. exit the gpg key-editor
  8. re-import your own key which is now of minimal size
  9. re-import your own private key from step 2
  10. check whether your key is complete. If not: restore the backup from step 1

Enigmail

How to compile Enigmail for Thunderbird from the sources 

git clone git://git.code.sf.net/p/enigmail/source enigmail ; or
git clone --branch enigmail-1-4-3 git://git.code.sf.net/p/enigmail/source enigmail

make distclean
./configure
make
make xpi

# this is no longer needed for cross-compilation (since 28.01.2014)
# ./configure --host=i686-pc-mingw32 ; for cross-compilation with http://mxe.cc/
#
# make distclean
# ./configure --host=i686-pc-mingw32 ; for cross-compilation with http://mxe.cc/
# make
# make xpi
#
# When cross-compiling for windows:
# the problem is that install.rdf file is not adapted properly. 
# You'll have to unzip enigmail.xpi
# remove a line in file install.rdf:
#
# 
# cd build
# unzip enigmail-1.6-win32-x86-msvc.xpi 
# ( edit install.rdf )
# ( remove the line "<em:targetPlatform>...</em:platform>" )
# zip -r enigmail-1.6-win32-x86-msvc.xpi *
# ^these steps are no longer needed.

mailvelope

How to compile Mailvelope for Firefox from the github sources 

#!/bin/bash

# based on https://github.com/toberndo/mailvelope

NOW="$(date "+%Y%m%d-%H%M%S")"

echo Make a fresh and shallow clone of Mailvelope with all dependencies from git and compile
echo Create a backup of local repositories in mailvelope-$NOW.tgz
tar -czf mailvelope-$NOW.tgz mailvelope addon-sdk

rm -r mailvelope addon-sdk

git clone --depth 1 git://github.com/toberndo/mailvelope.git
cd mailvelope
git submodule init
git submodule update
make build

cd ..
git clone --depth 1 git://github.com/mozilla/addon-sdk.git
cd addon-sdk
source bin/activate

cd ../mailvelope
make dist-ff

cd /usr/local/src
./make-mailvelope.sh
# check the resulting file
dir /usr/local/src/mailvelope/dist/mailvelope.firefox.xpi

Audio and Video transcoding and decoding

OGG => WAV => MP3 

# script:
# ogg2mp3.sh
ogg123 -d wav x.ogg -f out.wav
ogg123 -d wav x.ogg -f - | lame -q0 -V2 -ms - out.mp3
ogg123 -d wav x.ogg -f - | lame -q0 -b 256 -ms - out.mp3

WMA or MP3 => WAV 

# use mplayer!
# script wma2wav.bat
mplayer -vc null -vo null -af resample=44100 -ao pcm:fast -ao pcm:waveheader -ao pcm:file="out.wav" "in.wma"
mplayer -vc null -vo null -af resample=44100 -ao pcm:fast -ao pcm:waveheader -ao pcm:file="out.wav" "in.mp3"

WMA => OGG or MP3 

# use wma2ogg.pl which is an mplayer tool script in /usr/lib/mplayer
# see https://help.ubuntu.com/community/MPlayerTips
# script wma2ogg.pl
/usr/lib/mplayer/wma2ogg.pl -lame -t mp3 -f <filename.WMA>

Extract aac audio from video without transcoding 

ffmpeg -i input.mp4 -c copy -map 0:a:0 output.aac (or output.m4a)

concatenate mp4 

ffmpeg -f concat -safe 0 -i <(for f in ./*.mp4; do echo "file '$PWD/$f'"; done) -c copy temp.mp4; ffmpeg -r 25 -i temp.mp4 -strict -2 -c copy out.mp4

optimize for streaming 

ffmpeg -i in.mp4 -movflags faststart -acodec copy -vcodec copy out.mp4
The -movflags faststart parameter is what tells ffmpeg to reorder the MP4 video atoms so that moov is at the start. We are also instructing ffmpeg to copy the video and audio data instead of re-encoding them, so nothing gets altered.

remove or set mp4 metadata information 

ffmpeg -i in.mp4 -c:v copy -c:a copy -metadata encoder="" -fflags +bitexact out.mp4

flip a video 

ffmpeg -i in.mp4 -vf vflip -c:a copy out.mp4

convert mkv to mp4 

ffmpeg -i infile.mkv -c copy outfile.mp4

transcode video for twitter 

ffmpeg -i in.mov -vcodec libx264 -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2" -pix_fmt yuv420p -strict experimental -r 30 -t 2:20 -acodec aac -vb 1024k -minrate 1024k -maxrate 1024k -bufsize 1024k -ar 44100 -ac 2 out.mp4

transcode an audio file into a stillimage as pseudo video file for twitter 

ffmpeg -loop 1 -i stillimage.png -i audio.mp3 -filter_complex "loop=-1:1:0" -c:v libx264 -x264-params keyint=1 -shortest -pix_fmt yuv420p -c:a aac video-for-twitter.mp4

shortest video file 

fmpeg -loop 1 -framerate 25 -i stillimage.png -i audio.wav -filter_complex "loop=-1:1:0" -c:v libx264 -x264-params keyint=100 -shortest -pix_fmt yuv420p -c:a aac video-for-twitter.mp4

recreate timestamps 

ffmpeg -r 25 -i in.webm -strict -2 -c copy out.mp4

Partially downloading videos from Youtube with ffmpeg

https://til.simonwillison.net/macos/downloading-partial-youtube-videos

Ocean Wave sound

Here's another fun variation if you like ocean wave sounds (it's not perfect). 
play -n synth brownnoise synth pinknoise mix synth 0 0 0 15 40 80 trapezium amod 0.2 20

Spooky noise 

play -n synth sin 900 bend 0.3,800,1 bend 1.2,-150,1 trim 0 3.5 reverb -w

Alarm sound 

play -n synth 0.5 saw 300-2000 repeat - vol 0.5

text-to-speech 

espeak -f <filename.txt> -s 150 -vde --stdout | lame -h -V2 - "filename.mp3"

Google Android TTS https://elinux.org/RPi_Text_to_Speech_(Speech_Synthesis) 

sudo apt-get install libttspico-utils
pico2wave -w out.wav -l de-DE "Deutschlandradio Kultur" && aplay out.wav

transcoding

Filme transkodieren 

cat /media/DVD_VIDEO_RECORDER/VIDEO_TS/VTS_01_*.VOB | ffmpeg -i - -strict -2 -vcodec libx264 /work/video_all.mp4

mp4 to ogg vorbis audio and video https://trac.ffmpeg.org/wiki/TheoraVorbisEncodingGuide 

ffmpeg -i input.mp4 -codec:v libtheora -qscale:v 7 -codec:a libvorbis -qscale:a 5 output.ogv

# transcoding to "-V2 -q0 -ms" example
lame -ms -V2 -q0 -mp3input in.mp3 out.mp3
# -mp3input is default for mp3 input files. so we can use
lame -ms -V2 in.mp3 out.mp3

deshaking video

Use ffmpeg with vidstab: https://github.com/georgmartius/vid.stab 

# pass1
ffmpeg -i input.mp4 -vf vidstabdetect -f null -
# pass2
ffmpeg -i input.mp4 -vf vidstabtransform=crop=black:smoothing=20,unsharp=5:5:0.8:3:3:0.4 output.mp4

remove station logos

https://www.martin-riedl.de/2019/08/10/remove-station-logo-from-video-using-ffmpeg/

How to remove two station logos, starting at 10' and processing 15 seconds only: 

ffmpeg -i in.mp4 -ss 00:10:00 -vf "delogo=x=1100:y=47:w=121:h=46,delogo=x=87:y=47:w=46:h=56" -t 15 out.mp4

subtitleing

https://stackoverflow.com/questions/8672809/use-ffmpeg-to-add-text-subtitles http://stackoverflow.com/a/13125122/731798

Requirements:

Install Aegisub program http://www.aegisub.org/ for editing subtitles. Install ffmpeg as the "Swiss Army Knife" for video processing including "hard-subbing" (to burn subtitles into the video). 

# if you have an srt file, then convert to ass
ffmpeg -i 1.srt 1.ass
# use yuv420 for compatibility with old decoders
ffmpeg -i "video-without-subtitles.mp4" -vf ass="subtitle-textfile.ass" -c:v libx264 -r 25 -pix_fmt yuv420p "video-with-subtitles.mp4"

How to burn subtitles into video 

ffmpeg -i in.mp4 -vf subtitles=in.srt out.mp4

ffmpeg -i in.mp4 -vf "subtitles=in.srt:force_style='PrimaryColour=&H00ffff&,FontSize=18,OutlineColour=H40000000&,BorderStyle=3,Spacing=1.0,BackColour=Black,Outline=1'" out.mp4

!#/bin/bash
for i in *.ass do
  # encode ("burn") subtitle files into video files
  # use yuv420 for compatibility with old decoders 
  ffmpeg -i 1.mp4 -vf ass=$i -c:v libx264 -r 25 -pix_fmt yuv420p ${i%.*}.mp4
done

convert ass to txt 

sed -e "0,/^Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text/d;s/^Dialogue:.*\([0-9]\+:[0-9]\+:[0-9]\+.[0-9]\+\).*Default.*0,,\(.*\)/\1 \2/g;s/{\\\i1}/<i>/g;s/{\\\i0}/<\/i>/g;s/\\\N/<br>/g" $1 $2

play and record streams 

 mplayer -vc null -vo null -ao pcm:fast:waveheader:file="out.wav" rtsp://stream4.rbb-online/....
 mplayer -dumpstream -dumpfile out.mp3 http://kulturradio.de/livemp3

Videos mit allen Untertiteln umkodieren 

 Videos mit Untertiteln umkodieren

Ich kodiere meine Videos regelmäßig gemäß Ihrer Anleitung aus c’t 13/2019, Seite 174 („Stummfilm auf dem Asus Media Player“) um, weil mein Mediaplayer das Audioformat EAC3 nicht beherrscht. Dazu verwende ich ffmpeg mit folgenden Optionen:

ffmpeg -i <videofile> -c:v copy -c:s copy -c:a ac3 <outfile>

Allerdings kopiert das nur je einen Audio- und Untertitel-Stream. Ich möchte aber gerne alle Untertitel bewahren, um den Film im Zweifelsfall auch mit deutschen Untertiteln abspielen zu können. Wie mache ich das?

Dazu müssen Sie ffmpeg explizit ein Mapping übergeben. Das geht einfach, indem Sie -map 0 vor dem -c:v einfügen. Damit kopiert ffmpeg alle Streams.

Alternativ können Sie auch gezielt Untertitel auswählen. Allerdings schaltet das explizite Mapping die automatische Auswahl komplett ab, sodass Sie dann auch Audio- und Video-Streams spezifizieren müssen. Mit diesem Aufruf kopieren Sie beispielsweise den ersten Video-Stream, konvertieren die erste Audiospur und kopieren die englischen und die deutschen Untertitel:

ffmpeg -i <video> -map 0:v -map 0:a -map 0:s:0 -map 0:s:15 -c:v copy -c:s copy -c:a ac3 <outfile>

Das ergibt dann folgendes Mapping:

Stream #0:0  -> #0:0 (copy)
Stream #0:1  -> #0:1 (eac3 (native) 
                      -> ac3 (native))
Stream #0:2  -> #0:2 (copy)
Stream #0:17 -> #0:3 (copy)

Zu beachten ist dabei, dass ffmpeg die Streams nach Typen getrennt zählt und dabei jeweils bei 0 beginnt. Aufgrund der vorangestellten Video- (0) und Audio-Streams (1) muss man den Stream Nummer 17 mit den deutschen Texten deshalb als 15. Untertitel-Stream (s:15) spezifizieren. Ein Aufruf von ffmpeg -i <videofile> zeigt Ihnen übrigens, was sich in welchen Streams befindet. (ju@ct.de)

Record a screencast

Gnome: Strg+Alt+Shift+R
  • gsettings get org.gnome.settings-daemon.plugins.media-keys max-screencast-length
Set max screencast length to 7200 seconds:
  • gsettings set org.gnome.settings-daemon.plugins.media-keys max-screencast-length 7200

record the browser/system audio output via command line 

# lookup where the output is sent to
# pacmd list-sinks | grep -e 'name:' -e 'index' -e 'Speakers'
parec -d alsa_output.pci-0000_00_1b.0.analog-stereo.monitor | lame -r -V0 - recorded.mp3

playing with audio pitch correction (i.e. original pitch) 

 mplayer -af scaletempo <file>
 mplayer -af scaletempo -speed 2.0 <file>
 mplayer -af scaletempo -speed 0.5 <file>

increase/decrease playing speed +/-10% [ ], by factor 2, 1/2 { }

speed up a video

https://trac.ffmpeg.org/wiki/How%20to%20speed%20up%20/%20slow%20down%20a%20video 

 ffmpeg -i infile.mp4 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" outfile.mp4

adjusting volume without transcoding 

# detect volume
ffmpeg -i infile.mp4 -af "volumedetect" -f null /dev/null
# adjust volume
ffmpeg -i infile.mp4 -vcodec copy -af "volume=20dB" output_20dB.mp4

ffmpeg concatenate equally-coded videos and timelapse the video 

# convert mp4 to ts
for i in *.mp4;do ffmpeg -i $i -c copy -bsf:v h264_mp4toannexb -f mpegts "${i%.*}.ts"; done

# concatenate ts and write mp4
ffmpeg -i "concat:1.ts|2.ts|3.ts|4.ts|5.ts|6.ts|7.ts|8.ts" -c copy -bsf:a aac_adtstoasc temp.mp4

# timelapse 50%
ffmpeg -i temp.mp4 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" out.mp4
# keyframes every 5th frame:
ffmpeg -i in.mp4 -x264-params keyint=5:scenecut=0 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" out.mp4

crop and rotate a smartphone video from 1280x720 portrait to landscape 

 ffmpeg -i in.mp4 -filter:v "crop=550:720:500:0,transpose=1" -c:a copy out.mp4

rotate a video 

 ffmpeg -i in.mp4 -metadata:s:v rotate="-90" -codec copy out.mp4

AVI => MPEG4 

ffmpeg -i "video.mpg.AVI" -acodec libfaac -ac 2 -ab 128k -vcodec libx264 -crf 21 -threads 0 "video.mp4"

How to extract every n-th frame

https://twitter.com/Wikinaut/status/1074964714675994624

How to extract every frame 

ffmpeg -i in.mp4 img%04d.jpg

How to extract every n-th frame, every keyframe, or frames every n-th second: 

ffmpeg -i in.mp4 -vf fps=1/5 img%04d.jpg -hide_banner

How to "fax a video" ?

How to overlay/watermark a movie

Add an overlay.png (here: the mask with a transparent background) to a movie (in.mp4): 
ffmpeg -i in.mp4 %04d.split.png
for i in *.split.png; do composite -gravity center overlay.png $i $i.png;done
ffmpeg -pattern_type glob -i '*.png.png' -c:v libx264 -r 25 -pix_fmt yuv420p "out.mp4"

Single line version: 

ffmpeg -i in.mp4 -i overlay.png -filter_complex "[0:v][1:v] overlay=0:0" -pix_fmt yuv420p -c:a copy ou

Cut part from video file without reencoding 

ffmpeg -ss starttime -i in.mp4 -t length -c copy out.mp4
ffmpeg -ss 2640 -i in.mp4 -t 00:05:20 -c copy out.mp4

Convert a Smartphone's Portrait Video into Landscape, with ffmpeg 

ffmpeg -i in.mp4 -filter_complex '[0:v]scale=ih*16/9:-1,boxblur=luma_radius=min(h\,w)/20:luma_power=1:chroma_radius=min(cw\,ch)/20:chroma_power=1[bg];[bg][0:v]overlay=(W-w)/2:(H-h)/2,crop=h=iw*9/16' out.mp4

length of an audio file 

 echo $(sox audio.mp3 -n stat 2>&1 | sed -n 's#^Length (seconds):[^0-9]*\([0-9.]*\)$#\1#p')

extract MP3 audio track from video 

ffmpeg -i "video.mp4" -vn -codec:a copy -f mp3 "audio-track.mp3" 
or
ffmpeg -i "video.mp4" -vn -ar 44100 -ac 2 -ab 192k -f mp3 "audio-track.mp3"

extract AAC audio track from video

make sure that ffmpeg only extracts and does not re-encode https://askubuntu.com/a/437799 https://askubuntu.com/questions/437798/how-to-extract-aac-audio-from-an-mp4-file-to-m4a 

ffmpeg -i input.mp4 -vn -c:a copy output.m4a

Export and import album cover art image to ID3 tags

Export cover image 

 ffmpeg -i file.mp3 cover-image.jpg

Import cover image 

 ffmpeg -i "infile.mp3" -i "cover-image.jpg" -codec:a copy -map_metadata 0 -map 0 -map 1 "outfile-with-image.mp3"

Concatenate MP3 files 

 cat Track{1..4}.mp3 > GESAMT.mp3
 ffmpeg -i GESAMT.mp3 -acodec copy Code or bash sniplets and useful scripts

Linux

Rename files using Regular Expressions

Rename files "from-*.*" to "to-*.*" using Regular Expressions 

  for i in *; do mv "$i" "`echo $i | sed "s/from-/to-/"`"; done

To move all files to elsewhere, but not folders

If you are interested in moving all files (but not folders) from Downloads folder to Videos folder, use this command 
find ~/Downloads/ -type f -print0 | xargs -0 mv -t ~/Videos

delete (or mark with ".keep") doublure files with same md5 but the oldest

find . -type f -iname "*.umap" -print0 | xargs -0 -i md5sum {} | sort | uniq -w 32 --all-repeated=separate | uniq -w 32 | awk 'NF {print $2}' | xargs -I{} echo mv {} {}.keep

add a prefix and directory name to filenames 

for folder in *; do
  if [ -d "$folder" ]; then
    cd "$folder"
    for file in *; do
      mv -- "$file" "${folder}_$file"
    done
    cd ..
  fi
done

Renumbering files with leading zeroes 

  j=0;for i in *;do ((j++));mv $i $(printf "%04d\n" $j).jpg; done

Rename files with dates dd.mm.yyyy to yyyymmdd_ 

  for i in *; do mv "$i" "`echo $i | sed -e "s/^\(.*\)\([0-9]\{2\}\)\.\([0-9]\{2\}\)\.\([0-9]\{4\}\)\(.*\)$/\4\3\2_\0/;s/PDF/pdf/"`"; done

copy all .pdf files (also with spaces in filenames) to cwd 

 IFS=$'\n';for i in $(find . -type f -iname "*.pdf"); do cp "$i" .; done

Change spaces in filenames to underscores recursively

requires /usr/bin/rename (a Perl script): 
 sudo find . -depth -name "* *" -execdir rename 's/ /_/g' "{}" \;

How to add line numbers "#### " to every line in a text file 

nl -w4 -n rz -s ' ' text.txt

-w4: 4 digits
-n rz: leading zeroes
-s ' ': separated by a single space

Delete files older than …

Delete files older than 30 days 

 find . -mtime +30 -print0 | xargs -0 rm
 find . -mtime +30 -print0 | xargs -0 rm -Rf (recursive, force = dangerous)

See http://stackoverflow.com/questions/16758525/use-xargs-with-filenames-containing-whitespaces for the use of -print0 and -0 options to allow filenames with white spaces.

Look up files with a certain date 

 find .  -type f -name "*" -newermt 2016-01-17 ! -newermt 2016-01-18 -print0 | xargs -0 --no-run-if-empty ls -lsa

Excluding Directories from 'locate' (updatedb)

Edit the /etc/updatedb.conf file. Add the paths to the PRUNEPATHS line and then re-run updatedb. For example: 

 PRUNEPATHS="/tmp /timeshift /var/spool /media /mnt"

grep in gz files 

find . -name "*.gz" -print0 | xargs -0 zgrep -i -E "(ne.dle1|needle2)"

Show heavy directories

Show "heavy" directories deeper than /this-dir.... ordered by size, show size in Megabytes (M) or Gigabytes (G) 

 sudo du -BM / | sort -nr | head -n 40
 sudo du -BG / | sort -nr | head -n 40

Show top level directory size only, and print on printer 

 du -h --max-depth=1 | lpr

Show number of files in a subdirectory 

 find /this-dir -type f | wc -l
 find . -type f | wc -l

List files only 

 ls -p | grep -v /

List only one level of subdirectories 

 find . -maxdepth 1 -type d

concatenate files sorted 

 cat `ls -- files*.ext | sort -V` > files-all.ext

sort files by dates dd.mm.yyyy in the filename 

ls | sed -e 's/^\(.*\)\([0-9]\{2\}\)\.\([0-9]\{2\}\)\.\([0-9]\{4\}\)\(.*\)$/\4\3\2_\0/;s/PDF/pdf/' | sort -r

Delete duplicate lines - make files unique 

sort infile | uniq > outfile

List files in file1 which are not in file2 

comm -23 <(sort file1) <(sort file2)

Zip all files older than …

Zip (or zip and move) all files older than x days
Usage: 

 ./zip-all-files-older-than.sh number-of-days [move]

#!/bin/bash
# 20121217

if [ $1 ] ; then

  UNTIL="$(date "+%Y%m%d" -d "$1 days ago")"

  if [ $2 ] && [ $2 == move ] ; then
    echo "zip and move all files older than $1 days (until $UNTIL)"
    find . -daystart -mtime +$(($1-1)) | xargs zip -m files-$UNTIL.zip
    ls -lh files-$UNTIL.zip
    df -h
  else
    echo "zip all files older than $1 days (until $UNTIL)"
    find . -daystart -mtime +$(($1-1)) | xargs zip files-$UNTIL.zip
    ls -lh files-$UNTIL.zip
    df -h
  fi

else
   echo zip all files older than n days
   echo Usage: $0 n [move]
fi

exit

Number of CPUs, Cores 

echo; echo "PHYSICAL CPUs:"; 
grep "physical id" /proc/cpuinfo | sort | uniq | wc -l;

echo; echo "CPU Cores:";
grep "cpu cores" /proc/cpuinfo | uniq;

echo; echo  "VIRTUAL PROCESSORS:";
grep "^processor" /proc/cpuinfo

Show version of operation system, kernel and distribution

see also http://www.sysadminslife.com/linux/quicktipp-installierte-linux-distribution-anzeigen-und-version-herausfinden/
https://linuxconfig.org/check-what-debian-version-you-are-running-on-your-linux-system
 cat /proc/version
 cat /etc/os-release
 uname -a
 uname -mrs
 lsb_release -a
 hostnamectl

Show BIOS version 

 sudo dmidecode -t bios

remount read/write 

mount -o remount,rw /media/user/device /media/user/device

flush DNS cache 

sudo systemd-resolve --flush-caches

CPU benchmark for the command line 

 time echo "scale=4000; a(1)*4" | bc -l

or use sysbench, see http://developer-blog.net/hardware/benchmark/ 

 sysbench --test=cpu --cpu-max-prime=20000 run

or vanitygen.

Full system backup with rsync

Rsync has two major modes of operation: the modes are dependent on the absence or presence of the trailing slash on the source directory. You need to add the trailing slash, otherwise, a folder called "x" is created inside the "x" folder that we specify as destination.

Boot requirements: see also https://wiki.archlinux.org/index.php/Full_system_backup_with_rsync#Boot_requirements 
 ; show overall progress info and transfer speed instead of huge list of files.
 rsync -aAX --info=progress2 --log-file="/home/benutzer/rsync.log" --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /path-to-backup/ /path/to/backup/folder

 ; show huge list of files
 rsync -aAXv --log-file="/home/benutzer/rsync.log" --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} /path-to-backup/ /path/to/backup/folder

 sudo rsync --dry-run -aAXv --log-file="/root/rsync.log" --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found","/home/**/.thumbnails/*","/home/**/Trash/*","/home/**/.cache"} / .

 ; Using rsync to move (not copy) files between directories
 ; https://unix.stackexchange.com/questions/43957/using-rsync-to-move-not-copy-files-between-directories
 ; https://wiki.ubuntuusers.de/rsync/
 ; https://wiki.archlinux.org/index.php/rsync
 rsync -avh -e ssh --progress --log-file="/home/benutzer/rsync.log" --remove-source-files /home/user/mystuff/* user@external.server:/path/to/backup
 find . -depth -type d -empty -delete

TYM Time Machine using rsync

Convert *.rpm to *.deb packages

  • for x64 CPUs and with execution of embedded scripts
 sudo alien -c --target=amd64 package.rpm

dpkg: how to remove all packages marked as rc

https://linuxprograms.wordpress.com/2010/05/12/remove-packages-marked-rc/

using Timeshift with RAID 

just use
sudo timeshift --list-devices to list availables devices (including raid partitions)
and then
sudo timeshift --snapshot-device /dev/mdXXX with correct device name to set your backup destination

The interface still doesn't list the partition but the backup seems ok

RAID1

How to replace a failed disk

MySQL

How to set UTF-8 as default character set

How to reset your root MySQL password

MySQL: add database and database user

When you use MySQL or MariaDB, you can create a new database "owncloud" and a new user "ownclouduser" with these lines: 
shell> mysql -u root -p
mysql> CREATE USER 'ownclouduser'@'localhost' IDENTIFIED BY 'password';
       CREATE DATABASE IF NOT EXISTS owncloud;
       GRANT ALL PRIVILEGES ON owncloud.* TO 'ownclouduser'@'localhost' IDENTIFIED BY 'password';
or
       GRANT CREATE,ALTER,SELECT,INSERT,UPDATE,DELETE ON owncloud.* TO 'ownclouduser'@'localhost' IDENTIFIED BY 'password';

       FLUSH PRIVILEGES;
       EXIT;

Dump MySQL table schema (table structure) without data 

mysqldump -u root -p --no-data dbname > schema.sql

MySQL dump and zip all databases 

#!/bin/bash

echo dump and zip all databases
# http://serverfault.com/questions/331057/mysqldump-unknown-table-engine-performance-schema
# http://www.sysadminslife.com/linux/mysql-warning-skipping-the-data-of-table-mysql-event-debian-ubuntu/
NOW="$(date "+%Y%m%d")"
mysqldump -u root -p \
--all-databases \
--events \
--default-character-set=utf8 --skip-set-charset \
--ignore-table=performance_schema.cond_instances \
--ignore-table=performance_schema.events_waits_current \
--ignore-table=performance_schema.cond_instances \
--ignore-table=performance_schema.events_waits_history \
--ignore-table=performance_schema.events_waits_history_long  \
--ignore-table=performance_schema.events_waits_summary_by_instance \
--ignore-table=performance_schema.events_waits_summary_by_thread_by_event_name \
--ignore-table=performance_schema.events_waits_summary_global_by_event_name \
--ignore-table=performance_schema.file_instances \
--ignore-table=performance_schema.file_summary_by_event_name \
--ignore-table=performance_schema.file_summary_by_instance  \
--ignore-table=performance_schema.mutex_instances \
--ignore-table=performance_schema.performance_timers \
--ignore-table=performance_schema.rwlock_instances \
--ignore-table=performance_schema.setup_consumers \
--ignore-table=performance_schema.setup_instruments \
--ignore-table=performance_schema.setup_timers \
--ignore-table=performance_schema.threads | gzip - > mysqldump-alldatabases_$NOW.gz

add debian-sys-maint user

poor men's file encryption e.g. for database dumps

Sequence to generate a compressed and encrypted database dump: 

mysqldump <dbname> -u root -p | gzip - | openssl aes-256-cbc -salt -k <password> -out <dbdump>

Example: 

mysqldump test -u root -p | gzip - | openssl aes-256-cbc -salt -k SeCrEt -out test.gz.aes

This creates test.gz.aes.

To decode this on our server use 

gunzip <dbdump> --stdout | openssl aes-256-cbc -d -k <password> > <dbname>.sql

Example: 

gunzip test.gz.aes --stdout | openssl aes-256-cbc -d -k SeCrEt > test.sql

Import the database dump in MySQL: 

mysql <dbname> -u root -p < <dbname>.sql

or, run the command directly in MySQL: 

mysql> use <dbname>;
mysql> SOURCE <dbname>.sql

git

prune history: make current commit the only (initial) commit in a Git repository 

git checkout --orphan newBranch
git add -A  # Add all files and commit them
git commit
git branch -D master  # Deletes the master branch
git branch -m master  # Rename the current branch to master
git push -f origin master  # Force push master branch to github
git gc --aggressive --prune=all     # remove the old files

borg (borgbackup)

Multi-archive search for $pattern in $repo 

for i in $(borg list --short $repo);do borg list --short $repo::$i "re:.*$pattern";done
for i in $(borg list --short $repo);do borg list --format="{archivename} {mode} {user:6} {group:6} {size:8d} {isomtime} {path}{extra}{NEWLINE}" $repo::$i "re:.*$pattern";done

borg mount -o versions 

 borg mount -o versions $repo /tmp/mountpoint

ownCloud

convert Sqlite to MySQL database

Read https://github.com/owncloud/core/issues/5139#issuecomment-28285731 about resulting issues "Failed to update | Multiple primary key for oc_locks table" from the Sqlite to MySQL conversion:

GnuPG

GnuPG (gpg) undocumented commands 

# show all built-in options (~320 options)
gpg --dump-options
gpg --dump-options | wc -l

# show the fingerprint of a keyfile without importing it first
# --with-fingerprint is an option, and not a regular command. But the default command (list key data) is used for the keyfile
gpg --with-fingerprint KEYFILE

# show internal key structures
gpg --list-packets KEYFILE

# generate public key from imported old PGP private keys
gpg --import --allow-non-selfsigned-uid KEYFILE

# use an old key for encryption
gpg --e --allow-non-selfsigned-uid recipient

# generate message digests (checksum, file hashes) in all available formats
# the "*" must be quoted to avoid command line expansion (globbing) on operating systems like Linux
gpg --print-md "*" FILENAME

# set better hash defaults in ~/.gnupg/gpg.conf 
personal-digest-preferences SHA512
cert-digest-algo SHA512
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 ZLIB BZIP2 ZIP Uncompressed

# create and verify a detached ascii armored signature file for FILENAME
gpg --detach-sign -a FILENAME
gpg --verify -v FILENAME.asc

# clear the passphrase cache.  Call that before leaving the computer.
gpgconf --reload gpg-agent

How to compile GnuPG (gpg) from the sources

see https://www.gnupg.org/download/cvs_access.html and http://git.gnupg.org/
This requires automake version 1.11 (see http://lists.gnupg.org/pipermail/gnupg-devel/2014-June/028537.html and see inside the script how to compile automake 1.11). The exemplary script shows what you need to compile GnuPG; a few commands require root privileges. 
#!/bin/bash

echo clone GnuPG with all dependencies from git and compile

# Before actually starting the compilation, 
# the following two lines can create an ad-hoc backup of the libraries
# with the date-time as part of the backup filename.
# These files can be deleted afterwards if everything went fine.
#
# NOW="$(date "+%Y%m%d")"
# tar -czf git-$NOW.tgz npth libgpg-error libgcrypt libassuan libksba pinentry gnupg

# GnuPG compilation requires automake version 1.11
#
# please make sure that automake 1.11 is installed (automake-1.11 on Debian). 
# Newer automakes are not compatible, because
# - these have parallel regression tests enabled by default
# - old automakes do not have an option for disabling
#
# if you have a recent automake:
# for compiling GnuPG install automake 1.11 using the following steps:
#
# wget http://ftp.gnu.org/gnu/automake/automake-1.11.6.tar.gz
# wget http://ftp.gnu.org/gnu/automake/automake-1.11.6.tar.gz.sig
# wget ftp://ftp.gnu.org/gnu/gnu-keyring.gpg
# gpg --verify --keyring ./gnu-keyring.gpg automake-1.11.6.tar.gz.sig
# tar -xzf automake-1.11.6.tar.gz
# cd automake-1.11.6/
# ./configure 
# make
# AUTOMAKE_SUFFIX=1.11
# check if automake version == 1.11
# automake --version

rm -rf npth libgpg-error libgcrypt libassuan libksba pinentry gnupg

for i in npth libgpg-error libgcrypt libassuan libksba pinentry gnupg ; do

  git clone --depth 1 git://git.gnupg.org/$i.git

  # Alternative location in case the other cannot be accessed:
  # git clone http://git.spodhuis.org/gnupg.org/$i.git
  
  cd $i
  ./autogen.sh

  if [ $i -eq gnupg ] ; then
    ./configure --enable-maintainer-mode
  else
    ./configure --enable-maintainer-mode
  fi

  make
  sudo make install
  cd ..

done

sudo pkill gpg-agent

cd /usr/local/src
./make-gnupg.sh

# check the compiled gpg2 version
cd /usr/local/src/gnupg/g10
./gpg2 --version

how to prune (trim) your public key

(not tested)

  1. backup the .gnupg directory
  2. export your key and store it in a safe place
  3. start key-editor: gpg --edit-key YourID
  4. selected expired subkeys (e.g. key 1), the selected subkey is marked with "*"
  5. delete this subkey with "delkey"
  6. remove with "uid number" and "deluid" not needed further User-IDs
  7. exit the gpg key-editor
  8. re-import your own key which is now of minimal size
  9. re-import your own private key from step 2
  10. check whether your key is complete. If not: restore the backup from step 1

Enigmail

How to compile Enigmail for Thunderbird from the sources 

git clone git://git.code.sf.net/p/enigmail/source enigmail ; or
git clone --branch enigmail-1-4-3 git://git.code.sf.net/p/enigmail/source enigmail

make distclean
./configure
make
make xpi

# this is no longer needed for cross-compilation (since 28.01.2014)
# ./configure --host=i686-pc-mingw32 ; for cross-compilation with http://mxe.cc/
#
# make distclean
# ./configure --host=i686-pc-mingw32 ; for cross-compilation with http://mxe.cc/
# make
# make xpi
#
# When cross-compiling for windows:
# the problem is that install.rdf file is not adapted properly. 
# You'll have to unzip enigmail.xpi
# remove a line in file install.rdf:
#
# 
# cd build
# unzip enigmail-1.6-win32-x86-msvc.xpi 
# ( edit install.rdf )
# ( remove the line "<em:targetPlatform>...</em:platform>" )
# zip -r enigmail-1.6-win32-x86-msvc.xpi *
# ^these steps are no longer needed.

mailvelope

How to compile Mailvelope for Firefox from the github sources 

#!/bin/bash

# based on https://github.com/toberndo/mailvelope

NOW="$(date "+%Y%m%d-%H%M%S")"

echo Make a fresh and shallow clone of Mailvelope with all dependencies from git and compile
echo Create a backup of local repositories in mailvelope-$NOW.tgz
tar -czf mailvelope-$NOW.tgz mailvelope addon-sdk

rm -r mailvelope addon-sdk

git clone --depth 1 git://github.com/toberndo/mailvelope.git
cd mailvelope
git submodule init
git submodule update
make build

cd ..
git clone --depth 1 git://github.com/mozilla/addon-sdk.git
cd addon-sdk
source bin/activate

cd ../mailvelope
make dist-ff

cd /usr/local/src
./make-mailvelope.sh
# check the resulting file
dir /usr/local/src/mailvelope/dist/mailvelope.firefox.xpi

Audio and Video transcoding and decoding

OGG => WAV => MP3 

# script:
# ogg2mp3.sh
ogg123 -d wav x.ogg -f out.wav
ogg123 -d wav x.ogg -f - | lame -q0 -V2 -ms - out.mp3
ogg123 -d wav x.ogg -f - | lame -q0 -b 256 -ms - out.mp3

WMA or MP3 => WAV 

# use mplayer!
# script wma2wav.bat
mplayer -vc null -vo null -af resample=44100 -ao pcm:fast -ao pcm:waveheader -ao pcm:file="out.wav" "in.wma"
mplayer -vc null -vo null -af resample=44100 -ao pcm:fast -ao pcm:waveheader -ao pcm:file="out.wav" "in.mp3"

WMA => OGG or MP3 

# use wma2ogg.pl which is an mplayer tool script in /usr/lib/mplayer
# see https://help.ubuntu.com/community/MPlayerTips
# script wma2ogg.pl
/usr/lib/mplayer/wma2ogg.pl -lame -t mp3 -f <filename.WMA>

Extract aac audio from video without transcoding 

ffmpeg -i input.mp4 -c copy -map 0:a:0 output.aac (or output.m4a)

concatenate mp4 

ffmpeg -f concat -safe 0 -i <(for f in ./*.mp4; do echo "file '$PWD/$f'"; done) -c copy temp.mp4; ffmpeg -r 25 -i temp.mp4 -strict -2 -c copy out.mp4

optimize for streaming 

ffmpeg -i in.mp4 -movflags faststart -acodec copy -vcodec copy out.mp4
The -movflags faststart parameter is what tells ffmpeg to reorder the MP4 video atoms so that moov is at the start. We are also instructing ffmpeg to copy the video and audio data instead of re-encoding them, so nothing gets altered.

remove or set mp4 metadata information 

ffmpeg -i in.mp4 -c:v copy -c:a copy -metadata encoder="" -fflags +bitexact out.mp4

transcode video for twitter 

ffmpeg -i in.mov -vcodec libx264 -vf "pad=ceil(iw/2)*2:ceil(ih/2)*2" -pix_fmt yuv420p -strict experimental -r 30 -t 2:20 -acodec aac -vb 1024k -minrate 1024k -maxrate 1024k -bufsize 1024k -ar 44100 -ac 2 out.mp4

transcode an audio file into a stillimage as pseudo video file for twitter 

ffmpeg -loop 1 -i stillimage.png -i audio.mp3 -filter_complex "loop=-1:1:0" -c:v libx264 -x264-params keyint=1 -shortest -pix_fmt yuv420p -c:a aac video-for-twitter.mp4

shortest video file 

fmpeg -loop 1 -framerate 25 -i stillimage.png -i audio.wav -filter_complex "loop=-1:1:0" -c:v libx264 -x264-params keyint=100 -shortest -pix_fmt yuv420p -c:a aac video-for-twitter.mp4

recreate timestamps 

ffmpeg -r 25 -i in.webm -strict -2 -c copy out.mp4

Ocean Wave sound

Here's another fun variation if you like ocean wave sounds (it's not perfect). 
play -n synth brownnoise synth pinknoise mix synth 0 0 0 15 40 80 trapezium amod 0.2 20

Spooky noise 

play -n synth sin 900 bend 0.3,800,1 bend 1.2,-150,1 trim 0 3.5 reverb -w

Alarm sound 

play -n synth 0.5 saw 300-2000 repeat - vol 0.5

text-to-speech 

espeak -f <filename.txt> -s 150 -vde --stdout | lame -h -V2 - "filename.mp3"

Google Android TTS https://elinux.org/RPi_Text_to_Speech_(Speech_Synthesis) 

sudo apt-get install libttspico-utils
pico2wave -w out.wav -l de-DE "Deutschlandradio Kultur" && aplay out.wav

transcoding

Filme transkodieren 

cat /media/DVD_VIDEO_RECORDER/VIDEO_TS/VTS_01_*.VOB | ffmpeg -i - -strict -2 -vcodec libx264 /work/video_all.mp4

mp4 to ogg vorbis audio and video https://trac.ffmpeg.org/wiki/TheoraVorbisEncodingGuide 

ffmpeg -i input.mp4 -codec:v libtheora -qscale:v 7 -codec:a libvorbis -qscale:a 5 output.ogv

# transcoding to "-V2 -q0 -ms" example
lame -ms -V2 -q0 -mp3input in.mp3 out.mp3
# -mp3input is default for mp3 input files. so we can use
lame -ms -V2 in.mp3 out.mp3

deshaking video

Use ffmpeg with vidstab: https://github.com/georgmartius/vid.stab 

# pass1
ffmpeg -i input.mp4 -vf vidstabdetect -f null -
# pass2
ffmpeg -i input.mp4 -vf vidstabtransform=crop=black:smoothing=20,unsharp=5:5:0.8:3:3:0.4 output.mp4

remove station logos

https://www.martin-riedl.de/2019/08/10/remove-station-logo-from-video-using-ffmpeg/

How to remove two station logos, starting at 10' and processing 15 seconds only: 

ffmpeg -i in.mp4 -ss 00:10:00 -vf "delogo=x=1100:y=47:w=121:h=46,delogo=x=87:y=47:w=46:h=56" -t 15 out.mp4

subtitleing

https://stackoverflow.com/questions/8672809/use-ffmpeg-to-add-text-subtitles http://stackoverflow.com/a/13125122/731798

Requirements:

Install Aegisub program http://www.aegisub.org/ for editing subtitles. Install ffmpeg as the "Swiss Army Knife" for video processing including "hard-subbing" (to burn subtitles into the video). 

# if you have an srt file, then convert to ass
ffmpeg -i 1.srt 1.ass
# use yuv420 for compatibility with old decoders
ffmpeg -i "video-without-subtitles.mp4" -vf ass="subtitle-textfile.ass" -c:v libx264 -r 25 -pix_fmt yuv420p "video-with-subtitles.mp4"

ffmpeg -i in.mp4 -vf subtitles=in.srt out.mp4

!#/bin/bash
for i in *.ass do
  # encode ("burn") subtitle files into video files
  # use yuv420 for compatibility with old decoders 
  ffmpeg -i 1.mp4 -vf ass=$i -c:v libx264 -r 25 -pix_fmt yuv420p ${i%.*}.mp4
done

convert ass to txt 

sed -e "0,/^Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text/d;s/^Dialogue:.*\([0-9]\+:[0-9]\+:[0-9]\+.[0-9]\+\).*Default.*0,,\(.*\)/\1 \2/g;s/{\\\i1}/<i>/g;s/{\\\i0}/<\/i>/g;s/\\\N/<br>/g" $1 $2

play and record streams 

 mplayer -vc null -vo null -ao pcm:fast:waveheader:file="out.wav" rtsp://stream4.rbb-online/....
 mplayer -dumpstream -dumpfile out.mp3 http://kulturradio.de/livemp3

Videos mit allen Untertiteln umkodieren 

 Videos mit Untertiteln umkodieren

Ich kodiere meine Videos regelmäßig gemäß Ihrer Anleitung aus c’t 13/2019, Seite 174 („Stummfilm auf dem Asus Media Player“) um, weil mein Mediaplayer das Audioformat EAC3 nicht beherrscht. Dazu verwende ich ffmpeg mit folgenden Optionen:

ffmpeg -i <videofile> -c:v copy -c:s copy -c:a ac3 <outfile>

Allerdings kopiert das nur je einen Audio- und Untertitel-Stream. Ich möchte aber gerne alle Untertitel bewahren, um den Film im Zweifelsfall auch mit deutschen Untertiteln abspielen zu können. Wie mache ich das?

Dazu müssen Sie ffmpeg explizit ein Mapping übergeben. Das geht einfach, indem Sie -map 0 vor dem -c:v einfügen. Damit kopiert ffmpeg alle Streams.

Alternativ können Sie auch gezielt Untertitel auswählen. Allerdings schaltet das explizite Mapping die automatische Auswahl komplett ab, sodass Sie dann auch Audio- und Video-Streams spezifizieren müssen. Mit diesem Aufruf kopieren Sie beispielsweise den ersten Video-Stream, konvertieren die erste Audiospur und kopieren die englischen und die deutschen Untertitel:

ffmpeg -i <video> -map 0:v -map 0:a -map 0:s:0 -map 0:s:15 -c:v copy -c:s copy -c:a ac3 <outfile>

Das ergibt dann folgendes Mapping:

Stream #0:0  -> #0:0 (copy)
Stream #0:1  -> #0:1 (eac3 (native) 
                      -> ac3 (native))
Stream #0:2  -> #0:2 (copy)
Stream #0:17 -> #0:3 (copy)

Zu beachten ist dabei, dass ffmpeg die Streams nach Typen getrennt zählt und dabei jeweils bei 0 beginnt. Aufgrund der vorangestellten Video- (0) und Audio-Streams (1) muss man den Stream Nummer 17 mit den deutschen Texten deshalb als 15. Untertitel-Stream (s:15) spezifizieren. Ein Aufruf von ffmpeg -i <videofile> zeigt Ihnen übrigens, was sich in welchen Streams befindet. (ju@ct.de)

Record a screencast

Gnome: Strg+Alt+Shift+R
  • gsettings get org.gnome.settings-daemon.plugins.media-keys max-screencast-length
Set max screencast length to 7200 seconds:
  • gsettings set org.gnome.settings-daemon.plugins.media-keys max-screencast-length 7200

record the browser/system audio output via command line 

# lookup where the output is sent to
# pacmd list-sinks | grep -e 'name:' -e 'index' -e 'Speakers'
parec -d alsa_output.pci-0000_00_1b.0.analog-stereo.monitor | lame -r -V0 - recorded.mp3

playing with audio pitch correction (i.e. original pitch) 

 mplayer -af scaletempo <file>
 mplayer -af scaletempo -speed 2.0 <file>
 mplayer -af scaletempo -speed 0.5 <file>

increase/decrease playing speed +/-10% [ ], by factor 2, 1/2 { }

speed up a video

https://trac.ffmpeg.org/wiki/How%20to%20speed%20up%20/%20slow%20down%20a%20video 

 ffmpeg -i infile.mp4 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" outfile.mp4

adjusting volume without transcoding 

# detect volume
ffmpeg -i infile.mp4 -af "volumedetect" -f null /dev/null
# adjust volume
ffmpeg -i infile.mp4 -vcodec copy -af "volume=20dB" output_20dB.mp4

ffmpeg concatenate equally-coded videos and timelapse the video 

# convert mp4 to ts
for i in *.mp4;do ffmpeg -i $i -c copy -bsf:v h264_mp4toannexb -f mpegts "${i%.*}.ts"; done

# concatenate ts and write mp4
ffmpeg -i "concat:1.ts|2.ts|3.ts|4.ts|5.ts|6.ts|7.ts|8.ts" -c copy -bsf:a aac_adtstoasc temp.mp4

# timelapse 50%
ffmpeg -i temp.mp4 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" out.mp4
# keyframes every 5th frame:
ffmpeg -i in.mp4 -x264-params keyint=5:scenecut=0 -filter_complex "[0:v]setpts=0.5*PTS[v];[0:a]atempo=2.0[a]" -map "[v]" -map "[a]" out.mp4

crop and rotate a smartphone video from 1280x720 portrait to landscape 

 ffmpeg -i in.mp4 -filter:v "crop=550:720:500:0,transpose=1" -c:a copy out.mp4

rotate a video 

 ffmpeg -i in.mp4 -metadata:s:v rotate="-90" -codec copy out.mp4

AVI => MPEG4 

ffmpeg -i "video.mpg.AVI" -acodec libfaac -ac 2 -ab 128k -vcodec libx264 -crf 21 -threads 0 "video.mp4"

How to extract every n-th frame

https://twitter.com/Wikinaut/status/1074964714675994624

How to extract every frame 

ffmpeg -i in.mp4 img%04d.jpg

How to extract every n-th frame, every keyframe, or frames every n-th second: 

ffmpeg -i in.mp4 -vf fps=1/5 img%04d.jpg -hide_banner

How to overlay/watermark a movie

Add an overlay.png (here: the mask with a transparent background) to a movie (in.mp4): 
ffmpeg -i in.mp4 %04d.split.png
for i in *.split.png; do composite -gravity center overlay.png $i $i.png;done
ffmpeg -pattern_type glob -i '*.png.png' -c:v libx264 -r 25 -pix_fmt yuv420p "out.mp4"

Single line version: 

ffmpeg -i in.mp4 -i overlay.png -filter_complex "[0:v][1:v] overlay=0:0" -pix_fmt yuv420p -c:a copy ou

Cut part from video file without reencoding 

ffmpeg -ss starttime -i in.mp4 -t length -c copy out.mp4
ffmpeg -ss 2640 -i in.mp4 -t 00:05:20 -c copy out.mp4

Convert a Smartphone's Portrait Video into Landscape, with ffmpeg 

ffmpeg -i in.mp4 -filter_complex '[0:v]scale=ih*16/9:-1,boxblur=luma_radius=min(h\,w)/20:luma_power=1:chroma_radius=min(cw\,ch)/20:chroma_power=1[bg];[bg][0:v]overlay=(W-w)/2:(H-h)/2,crop=h=iw*9/16' out.mp4

length of an audio file 

 echo $(sox audio.mp3 -n stat 2>&1 | sed -n 's#^Length (seconds):[^0-9]*\([0-9.]*\)$#\1#p')

extract MP3 audio track from video 

ffmpeg -i "video.mp4" -vn -codec:a copy -f mp3 "audio-track.mp3" 
or
ffmpeg -i "video.mp4" -vn -ar 44100 -ac 2 -ab 192k -f mp3 "audio-track.mp3"

extract AAC audio track from video

make sure that ffmpeg only extracts and does not re-encode https://askubuntu.com/a/437799 https://askubuntu.com/questions/437798/how-to-extract-aac-audio-from-an-mp4-file-to-m4a 

ffmpeg -i input.mp4 -vn -c:a copy output.m4a

Export and import album cover art image to ID3 tags

Export cover image 

 ffmpeg -i file.mp3 cover-image.jpg

Import cover image 

 ffmpeg -i "infile.mp3" -i "cover-image.jpg" -codec:a copy -map_metadata 0 -map 0 -map 1 "outfile-with-image.mp3"

Concatenate MP3 files 

 cat Track{1..4}.mp3 > GESAMT.mp3
 ffmpeg -i GESAMT.mp3 -acodec copy GESAMT_KORRIGIERT.mp3
 rm GESAMT.mp3
 id3cp Track1.mp3 GESAMT_KORRIGIERT.mp3

multiple-file commands 

# for datei in "/videos/*.mpg ; do dx=${datei##*/} ; d=${dx%.} ; ffmpeg -i $datei cat /video/*.VOB | ffmpeg -i - -str
for datei in "/videos/*.mpg ; do dx=${datei##*/} ; d=${dx%.x} ; ffmpeg -i "$datei" -strict -2 -vcodec libx264 "$d" ; done

for i in *.flv;do ffmpeg -i "${i%.*}".flv -acodec copy -vcodec libx264 -threads 0 "${i%.*}".mp4;done  

ffprobe video.mpg
ffplay video.mpg

Audio playlist tools

copy playlist files to external drive

mix audio files with crossfades into a single mix file

Convert Webp images 

Ich vergesse ständig, wie das Kommandozeilen-Tool zur Umwandlung von Webp-Bildern in ein richtiges Bildformat heißt.

Ein Aufruf nach dem Muster man -k <stichwort> zeigt alle Befehle zu einem Stichwort an, so liefert etwa man -k webp unter anderem:

cwebp (1)   - compress an image file to a WebP file
dwebp (1)   - decompress a WebP file to an image file

Der Suchbegriff funktioniert als regulärer Ausdruck, sodass Sie auch zum Beispiel nach web'.\*' suchen können. Alternativ zum man-Kommando können Sie auch den Befehl apropos verwenden. (ju@ct.de)

Copy an image out of your clipboard to a file 

$ xclip -sel clip -t image/png -o > image.png

ImageMagick/Convert Image tools

generate canonical image filenames (filename from EXIF date)

The -P option preserves the original file modification date. 

alias r2d='exiftool -P -d %Y%m%d-%H%M%S%%-c.%%e "-FileName<DateTimeOriginal"'

exiftool -ext jpg '-FileName<CreateDate' -d %Y%m%d_%H%M%S%%-c.%%e .

Create a polaroid-style overview image 

montage *.jpg -thumbnail 1200x1200 -set caption %f -background grey40 -pointsize 40 +polaroid -background white -geometry +1+1 -tile 10x20 -title "caption text" polaroid.jpg
montage *.jpg -thumbnail 1200x1200 -set caption %f -background grey40 -pointsize 40 -polaroid 0 -background white -geometry +1+1 -tile 10x20 -title "caption text" polaroid.jpg

create thumbnail views from all first PDF pages 

for i in *.pdf; do convert -thumbnail "1000x1000>" $i[0] ${i%.pdf}.png; done

convert pdf to image with high resolution 

convert -density 300 -trim in.pdf -quality 100 %d.png

create jpg file with about 2 MB size 

convert in.png -define jpeg:extent=2MB out-2MB.jpg
mogrify -define jpeg:extent=2MB *.jpg

color operations

Determine the average color of an image and create a 32x32 square with that color. 

convert input.png -colors 256  -scale 1x1\! -scale 32x32 average-color-tile-32x32.png

How to make an animated gif 

convert -delay 70 -loop 0 *.png anim.gif

manipulate and merge animated Gif sequences

Problem:

You have an animated gif and want to

  • zoom into an area A in the source sequence bound by (xa1,ya1), (xa2,ya2)
  • move and enlarge another area B (xb1,yb1), (xb2,yb2) ; and
  • merge both parts into a new output sequence.
The example has area A (250,200),(385,325) and area B (5,5),(57,27) 
convert \( $1 -coalesce -crop (xa2-xa1)x(ya2-ya1)+x1a+y1a -resize 500x500 +repage \) -coalesce null: \( $1 -coalesce -crop (xb2-xb1)x(yb2-yb1)+xb1+yb1 +repage -resize 120x120 \) -geometry +40+90 -layers Composite $2

#!/bin/bash

# see http://www.imagemagick.org/Usage/anim_mods/#merging
# convert x.gif -coalesce -crop 125x135+250+200 -resize 500x500 +repage z.gif
# convert x.gif -coalesce -crop 52x22+5+5 +repage z1.gif
# convert z.gif -coalesce null: \( z1.gif -resize 120x120 -coalesce \) -geometry +40+90 -layers Composite z2.gif

convert \( $1 -coalesce -crop 125x135+250+200 -resize 500x500 +repage \) -coalesce null: \( $1 -coalesce -crop 52x22+5+5 +repage -resize 120x120 \) -geometry +40+90 -layers Composite $2

create a movie from still images, add their directory and filenames as caption, make 10:6 and 4:3 movies 

!#/bin/bash

# rewrite filenames to contain parent directory name
# for f in */* ; do cp $f "../all/$(dirname $f) - $(basename $f)"; done

# rename filenames
# for i in *; do mv "$i" "`echo $i | sed "s/from-/to-/"`"; done

# sort
# ls -ls | sort -k9

# if you need this
# mogrify -format tif *.png

# perspective control with Shift-N
# Install http://www.shiftn.de/ . On Linux, use it under WINE !
# ShiftN can process jpg, tif, bmp but cannot process png!

# for png input images:
# env WINEPREFIX="/home/benutzer/.wine";for i in *.png;do convert "${i%.*}.png" -format tif "${i%.*}.tif";wine C:\\Program\ Files\ \(x86\)\\ShiftN\\ShiftN.exe "${i%.*}.tif" "${i%.*}" A4; done

# or for tif input images (ShiftN "A4" output is tif, see http://www.shiftn.de/ShiftN_Funktionsweise.html )
env WINEPREFIX="/home/benutzer/.wine";for i in *.tif;do wine C:\\Program\ Files\ \(x86\)\\ShiftN\\ShiftN.exe "${i%.*}.tif" "${i%.*}" A4;done


size="1920x1080"
# size="1280x1024"

# resize to letterbox format, preserve aspect ratio
for f in *.tif; do convert "$f" -resize $size -background black -compose Copy -gravity center -extent $size "$f"; done

# add filename as annotation. Use pointsize 40 for 1920x1080, pointsize 28 for 1280x1024
# to list the available font names, use $ convert -list font | grep Font
for f in *.tif; do convert "$f" -gravity South -undercolor '#88888880' -font "Consolas" -pointsize 40 -fill yellow -annotate 0 "%t" "$f"; done

# https://trac.ffmpeg.org/wiki/Create%20a%20video%20slideshow%20from%20images
# http://www.itforeveryone.co.uk/image-to-video.html

# make movie from images (use pix_fmt to create a compatible stream for older decoders)
ffmpeg -framerate 1/10 -pattern_type glob -i '*.tif' -c:v libx264 -r 25 -pix_fmt yuv420p "out-${size}.mp4"

extend or crop canvas to a multiple of 32 

#!/bin/bash

if [ $# -lt 1 ]; then
   echo "extend-crop-background <imagefile> [modulus [extent]]"
   echo "Extend or crop image to a multiple of modulus ±'extent'"
   echo
   echo "modulus: default 32"
   echo "extent: default=modulus"
   echo
   echo "Examples:"
   echo "extend-crop-background in.jpg 32"
   echo "extend-crop-background in.jpg 32 -32"
   echo "extend-crop-background in.jpg 100 -32"
   exit
fi

magick "$1" \
  -background none \
  -gravity center \
  -set option:xmod "${2:-32}" \
  -set option:zz "$3" \
  -set wx '%[fx:mod(w,xmod)==0?w:w-mod(w,xmod)+zz+xmod]' \
  -set hx '%[fx:mod(h,xmod)==0?h:h-mod(h,xmod)+zz+xmod]' \
  -extent '%[wx]x%[hx]' \
  -set filename:mod '%[xmod]' \
  -set filename:size '%wx%h' \
  -set filename:name '%t' \
  '%[filename:name]_mod%[filename:mod]-%[filename:size].png'

Convert PNG to SVG in Inkscape

https://www.youtube.com/watch?v=KRkx8kjmXZw 

* Select Path → Trace Bitmap

  → Colors: yes
  → Scans: 2
  → ok

* Drag it down to separate:

  → Top is the SVG
  → Bottom: PNG (delete)

Puppeteer

Sandbox under debian

PDF tools

collate (concatenate)

Add several PDFs into a single output PDF 

pdftk 1.pdf 2.pdf ... output collated.pdf

Sort by name and add all pdf files into a single output PDF 

pdftk $(ls *.pdf | sort -n) cat output all.pdf
https://stackoverflow.com/questions/14639206/how-can-i-pass-all-arguments-with-xargs-in-middle-of-command-in-linux

Shuffle

Shuffle (collate) two files with 1-sided scans to one 2-sided PDF. The "shuffle" command was introduced in PDFTK 1.44 - October 28, 2010, so you need at least that version (see http://www.pdflabs.com/docs/pdftk-version-history/)

file shuffle.bat: 

pdftk A=1.pdf B=2.pdf shuffle A Bend-1 output collated.pdf

Reverse page sequence 

pdftk A=1.pdf cat Aend-1 output reversed.pdf

Rotate pages 180° 

pdftk A=1.pdf cat Asouth output rotated.pdf

shuffle a paperbook which long-edge is scanned as "short-edge" to 2-sided portrait format 

pdftk A=in.pdf shuffle AoddLeft AevenRight output out.pdf

Split PDF into half (decutshuffler)

"When you scanned the two-sided DIN-A4-sheets of a DIN-A5-booklet with your scanner, and you want its single pages in the correct sequence", or "How to split PDF pages with double page layout in two, down the middle. Splitting one A4 into two A5 pages is a great example." then use this code: 

!/bin/bash
# split pdf in half

fn="${1%.*}"

# mutool is part of mupdf-tools
mutool poster -x 0.5 "$fn.pdf" "$fn.tmp.pdf"
pdftk A="$fn.tmp.pdf" cat A1-endodd Aend-2even output "$fn.split.pdf"
rm "$fn.tmp.pdf"

Other possibilities and sources:

split 1 page into 2 pages 

mutool poster -x 0.5 singlepage.pdf twopages.pdf

add additional margins to PDF 

pdfjam --keepinfo --trim "-10mm -15mm -10mm -15mm" --clip true --suffix "cropped" in.pdf

scale all pages in PDF to A4

http://stackoverflow.com/a/14538863/731798 has a working one-liner, which simply can be used as an after-burner to the output of pdf-redact-tools -merge : 

gs -o out.pdf -sDEVICE=pdfwrite -sPAPERSIZE=a4 -dFIXEDMEDIA -dPDFFitPage -dCompatibilityLevel=1.4 in.pdf

change PDF resolution and compression

Example: 

convert -density 300 -compress jpeg -quality 20 in.pdf out.pdf

Better Quality: 

convert -density 300 -compress jpeg -quality 50 in.pdf out.pdf
Place in your ~/.bash_aliases: 
function pdfresize() {
  gs -sDEVICE=pdfwrite -sPAPERSIZE=a4 -dColorConversionStrategy=/LeaveColorUnchanged -dAutoRotatePages=/None \
  -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dFIXEDMEDIA -dPDFFitPage -dNOPAUSE -dQUIET -dBATCH \
  -sOutputFile="${1%.*}.reduz.pdf" "${1}"
}
pdfresize filename

create an 8 page pocketmod booklet

pdfjam-pocketmod - make an 8-page PDF document into a pocket-sized booklet 
pdfjam --help
pdfjam-pocketmod --help
pdfjam-pocketmod <in.pdf> --outfile <out.pdf>

create a blank PDF page 

echo "" | ps2pdf -sPAPERSIZE=a4 - blank.pdf

git

How to compare branches, items...

To get to the compare view, append /compare to your repository's path.

Still a missing feature (I asked Github to implement a button, or show a link for that). The compare help article explains how to use the compare page.

git list all ignored files 

 git ls-files --others --ignored --exclude-standard

git squash the last 3 commits

→ Source: http://stackoverflow.com/questions/5189560/squash-my-last-x-commits-together-using-git http://stackoverflow.com/a/5201642 
 git reset --soft HEAD~3
 git commit --edit -m"$(git log --format=%B --reverse HEAD..HEAD@{1})"

How to remove files listed in gitignore

https://stackoverflow.com/questions/13541615/how-to-remove-files-that-are-listed-in-the-gitignore-but-still-on-the-repositor 

 git ls-files -i -z --exclude-from=.gitignore | xargs -0 git rm --cached

(github) Create a new repository on the command line 

 touch README.md
 git init
 git add README.md
 git commit -m "first commit"
 git remote add origin git@github.com:username/repo.git
 git push -u origin master

(github) Push an existing repository from the command line 

 git remote add origin git@github.com:username/repo.git
 git push -u origin master

Generate fully-flavoured apk filename with app_name, versionName, versionCode, git hash

Generate fully-flavoured apk filename with app_name, versionName, versionCode, git hash https://gist.github.com/Wikinaut/d3eb229e1dd1ec703f00

Hardware: Audio mixer circuits

Let's Encrypt 

old:
./letsencrypt-auto renew --force-renewal --agree-tos --rsa-key-size 4096

lumen to watt

Leetspeek online generator

http://www.robertecker.com/hp/research/leet-converter.php

sed

  • replace twitterurls
sed -e 's/twitter.com\/\(.*\?\)\/status\/\([0-9]\+\)/twitter.com\/i\/status\/\2/' infile

sed: sed: joining lines depending on pattern starting on the second one 

sed 'N;s/\npattern/pattern/;P;D' file

Raspberry Pi

autostart methods

http://raspberry.tips/raspberrypi-einsteiger/raspberry-pi-autostart-von-skripten-und-programmen-einrichten

Nextcloud

jobs were killed

Jobs are killed after upgrade to 21?

/etc/php/8.0/cli/conf.d/20-apcu.ini: 
apc.enable_cli=1

or in /etc/php/7.3/cli/php.ini: 

apc.enable_cli=1

CORS header

If you want to openWebsite a shared video on an NEXTCLOUD instance, a change is needed. Add the rC3-Domain/s!

In /lib/public/AppFramework/Http/ContentSecurityPolicy.php: 

/** @var array Domains which can embed this Nextcloud instance */
protected $allowedFrameAncestors = [
'\'self\'',
'test.visit.at.wa-test.rc3.cccv.de',
];
Source: https://return2.net/nextcloud-enable-external-iframe/

GESAMT_KORRIGIERT.mp3 

 rm GESAMT.mp3
 id3cp Track1.mp3 GESAMT_KORRIGIERT.mp3

multiple-file commands 

# for datei in "/videos/*.mpg ; do dx=${datei##*/} ; d=${dx%.} ; ffmpeg -i $datei cat /video/*.VOB | ffmpeg -i - -str
for datei in "/videos/*.mpg ; do dx=${datei##*/} ; d=${dx%.x} ; ffmpeg -i "$datei" -strict -2 -vcodec libx264 "$d" ; done

for i in *.flv;do ffmpeg -i "${i%.*}".flv -acodec copy -vcodec libx264 -threads 0 "${i%.*}".mp4;done  

ffprobe video.mpg
ffplay video.mpg

remove id3 Tags and library string 

ffmpeg -i in.mp3 -map_metadata -1 -write_xing 0 -id3v2_version 0 -c copy out.mp3

Audio playlist tools

copy playlist files to external drive

mix audio files with crossfades into a single mix file

Convert Webp images 

Ich vergesse ständig, wie das Kommandozeilen-Tool zur Umwandlung von Webp-Bildern in ein richtiges Bildformat heißt.

Ein Aufruf nach dem Muster man -k <stichwort> zeigt alle Befehle zu einem Stichwort an, so liefert etwa man -k webp unter anderem:

cwebp (1)   - compress an image file to a WebP file
dwebp (1)   - decompress a WebP file to an image file

Der Suchbegriff funktioniert als regulärer Ausdruck, sodass Sie auch zum Beispiel nach web'.\*' suchen können. Alternativ zum man-Kommando können Sie auch den Befehl apropos verwenden. (ju@ct.de)

ImageMagick/Convert Image tools

generate canonical image filenames (filename from EXIF date)

The -P option preserves the original file modification date. 

alias r2d='exiftool -P -d %Y%m%d-%H%M%S%%-c.%%e "-FileName<DateTimeOriginal"'

Create a polaroid-style overview image 

montage *.jpg -thumbnail 1200x1200 -set caption %f -background grey40 -pointsize 40 +polaroid -background white -geometry +1+1 -tile 10x20 -title "caption text" polaroid.jpg
montage *.jpg -thumbnail 1200x1200 -set caption %f -background grey40 -pointsize 40 -polaroid 0 -background white -geometry +1+1 -tile 10x20 -title "caption text" polaroid.jpg

create thumbnail views from all first PDF pages 

for i in *.pdf; do convert -thumbnail "1000x1000>" $i[0] ${i%.pdf}.png; done

convert pdf to image with high resolution 

convert -density 300 -trim in.pdf -quality 100 %d.png

create jpg file with about 2 MB size 

convert in.png -define jpeg:extent=2MB out-2MB.jpg
mogrify -define jpeg:extent=2MB *.jpg

color operations

Determine the average color of an image and create a 32x32 square with that color. 

convert input.png -colors 256  -scale 1x1\! -scale 32x32 average-color-tile-32x32.png

How to make an animated gif 

convert -delay 70 -loop 0 *.png anim.gif

manipulate and merge animated Gif sequences

Problem:

You have an animated gif and want to

  • zoom into an area A in the source sequence bound by (xa1,ya1), (xa2,ya2)
  • move and enlarge another area B (xb1,yb1), (xb2,yb2) ; and
  • merge both parts into a new output sequence.
The example has area A (250,200),(385,325) and area B (5,5),(57,27) 
convert \( $1 -coalesce -crop (xa2-xa1)x(ya2-ya1)+x1a+y1a -resize 500x500 +repage \) -coalesce null: \( $1 -coalesce -crop (xb2-xb1)x(yb2-yb1)+xb1+yb1 +repage -resize 120x120 \) -geometry +40+90 -layers Composite $2

#!/bin/bash

# see http://www.imagemagick.org/Usage/anim_mods/#merging
# convert x.gif -coalesce -crop 125x135+250+200 -resize 500x500 +repage z.gif
# convert x.gif -coalesce -crop 52x22+5+5 +repage z1.gif
# convert z.gif -coalesce null: \( z1.gif -resize 120x120 -coalesce \) -geometry +40+90 -layers Composite z2.gif

convert \( $1 -coalesce -crop 125x135+250+200 -resize 500x500 +repage \) -coalesce null: \( $1 -coalesce -crop 52x22+5+5 +repage -resize 120x120 \) -geometry +40+90 -layers Composite $2

create a movie from still images, add their directory and filenames as caption, make 10:6 and 4:3 movies 

!#/bin/bash

# rewrite filenames to contain parent directory name
# for f in */* ; do cp $f "../all/$(dirname $f) - $(basename $f)"; done

# rename filenames
# for i in *; do mv "$i" "`echo $i | sed "s/from-/to-/"`"; done

# sort
# ls -ls | sort -k9

# if you need this
# mogrify -format tif *.png

# perspective control with Shift-N
# Install http://www.shiftn.de/ . On Linux, use it under WINE !
# ShiftN can process jpg, tif, bmp but cannot process png!

# for png input images:
# env WINEPREFIX="/home/benutzer/.wine";for i in *.png;do convert "${i%.*}.png" -format tif "${i%.*}.tif";wine C:\\Program\ Files\ \(x86\)\\ShiftN\\ShiftN.exe "${i%.*}.tif" "${i%.*}" A4; done

# or for tif input images (ShiftN "A4" output is tif, see http://www.shiftn.de/ShiftN_Funktionsweise.html )
env WINEPREFIX="/home/benutzer/.wine";for i in *.tif;do wine C:\\Program\ Files\ \(x86\)\\ShiftN\\ShiftN.exe "${i%.*}.tif" "${i%.*}" A4;done


size="1920x1080"
# size="1280x1024"

# resize to letterbox format, preserve aspect ratio
for f in *.tif; do convert "$f" -resize $size -background black -compose Copy -gravity center -extent $size "$f"; done

# add filename as annotation. Use pointsize 40 for 1920x1080, pointsize 28 for 1280x1024
# to list the available font names, use $ convert -list font | grep Font
for f in *.tif; do convert "$f" -gravity South -undercolor '#88888880' -font "Consolas" -pointsize 40 -fill yellow -annotate 0 "%t" "$f"; done

# https://trac.ffmpeg.org/wiki/Create%20a%20video%20slideshow%20from%20images
# http://www.itforeveryone.co.uk/image-to-video.html

# make movie from images (use pix_fmt to create a compatible stream for older decoders)
ffmpeg -framerate 1/10 -pattern_type glob -i '*.tif' -c:v libx264 -r 25 -pix_fmt yuv420p "out-${size}.mp4"

extend or crop canvas to a multiple of 32 

#!/bin/bash

if [ $# -lt 1 ]; then
   echo "extend-crop-background <imagefile> [modulus [extent]]"
   echo "Extend or crop image to a multiple of modulus ±'extent'"
   echo
   echo "modulus: default 32"
   echo "extent: default=modulus"
   echo
   echo "Examples:"
   echo "extend-crop-background in.jpg 32"
   echo "extend-crop-background in.jpg 32 -32"
   echo "extend-crop-background in.jpg 100 -32"
   exit
fi

magick "$1" \
  -background none \
  -gravity center \
  -set option:xmod "${2:-32}" \
  -set option:zz "$3" \
  -set wx '%[fx:mod(w,xmod)==0?w:w-mod(w,xmod)+zz+xmod]' \
  -set hx '%[fx:mod(h,xmod)==0?h:h-mod(h,xmod)+zz+xmod]' \
  -extent '%[wx]x%[hx]' \
  -set filename:mod '%[xmod]' \
  -set filename:size '%wx%h' \
  -set filename:name '%t' \
  '%[filename:name]_mod%[filename:mod]-%[filename:size].png'

Convert PNG to SVG in Inkscape

https://www.youtube.com/watch?v=KRkx8kjmXZw 

* Select Path → Trace Bitmap

  → Colors: yes
  → Scans: 2
  → ok

* Drag it down to separate:

  → Top is the SVG
  → Bottom: PNG (delete)

Puppeteer

Sandbox under debian

PDF tools

collate (concatenate)

Add several PDFs into a single output PDF 

pdftk 1.pdf 2.pdf ... output collated.pdf

Sort by name and add all pdf files into a single output PDF 

pdftk $(ls *.pdf | sort -n) cat output all.pdf
https://stackoverflow.com/questions/14639206/how-can-i-pass-all-arguments-with-xargs-in-middle-of-command-in-linux

Shuffle

Shuffle (collate) two files with 1-sided scans to one 2-sided PDF. The "shuffle" command was introduced in PDFTK 1.44 - October 28, 2010, so you need at least that version (see http://www.pdflabs.com/docs/pdftk-version-history/)

file shuffle.bat: 

pdftk A=1.pdf B=2.pdf shuffle A Bend-1 output collated.pdf

Reverse page sequence 

pdftk A=1.pdf cat Aend-1 output reversed.pdf

Rotate pages 180° 

pdftk A=1.pdf cat Asouth output rotated.pdf

shuffle a paperbook which long-edge is scanned as "short-edge" to 2-sided portrait format 

pdftk A=in.pdf shuffle AoddLeft AevenRight output out.pdf

Split PDF into half (decutshuffler)

"When you scanned the two-sided DIN-A4-sheets of a DIN-A5-booklet with your scanner, and you want its single pages in the correct sequence", or "How to split PDF pages with double page layout in two, down the middle. Splitting one A4 into two A5 pages is a great example." then use this code: 

!/bin/bash
# split pdf in half

fn="${1%.*}"

# mutool is part of mupdf-tools
mutool poster -x 0.5 "$fn.pdf" "$fn.tmp.pdf"
pdftk A="$fn.tmp.pdf" cat A1-endodd Aend-2even output "$fn.split.pdf"
rm "$fn.tmp.pdf"

Other possibilities and sources:

split 1 page into 2 pages 

mutool poster -x 0.5 singlepage.pdf twopages.pdf

add additional margins to PDF 

pdfjam --keepinfo --trim "-10mm -15mm -10mm -15mm" --clip true --suffix "cropped" in.pdf

scale all pages in PDF to A4

http://stackoverflow.com/a/14538863/731798 has a working one-liner, which simply can be used as an after-burner to the output of pdf-redact-tools -merge : 

gs -o out.pdf -sDEVICE=pdfwrite -sPAPERSIZE=a4 -dFIXEDMEDIA -dPDFFitPage -dCompatibilityLevel=1.4 in.pdf

change PDF resolution and compression

Example: 

convert -density 300 -compress jpeg -quality 20 in.pdf out.pdf

Better Quality: 

convert -density 300 -compress jpeg -quality 50 in.pdf out.pdf
Place in your ~/.bash_aliases: 
function pdfresize() {
  gs -sDEVICE=pdfwrite -sPAPERSIZE=a4 -dColorConversionStrategy=/LeaveColorUnchanged -dAutoRotatePages=/None \
  -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dFIXEDMEDIA -dPDFFitPage -dNOPAUSE -dQUIET -dBATCH \
  -sOutputFile="${1%.*}.reduz.pdf" "${1}"
}
pdfresize filename

create an 8 page pocketmod booklet

pdfjam-pocketmod - make an 8-page PDF document into a pocket-sized booklet 
pdfjam --help
pdfjam-pocketmod --help
pdfjam-pocketmod <in.pdf> --outfile <out.pdf>

create a blank PDF page 

echo "" | ps2pdf -sPAPERSIZE=a4 - blank.pdf

git

How to compare branches, items...

To get to the compare view, append /compare to your repository's path.

Still a missing feature (I asked Github to implement a button, or show a link for that). The compare help article explains how to use the compare page.

git list all ignored files 

 git ls-files --others --ignored --exclude-standard

git squash the last 3 commits

→ Source: http://stackoverflow.com/questions/5189560/squash-my-last-x-commits-together-using-git http://stackoverflow.com/a/5201642 
 git reset --soft HEAD~3
 git commit --edit -m"$(git log --format=%B --reverse HEAD..HEAD@{1})"

How to remove files listed in gitignore

https://stackoverflow.com/questions/13541615/how-to-remove-files-that-are-listed-in-the-gitignore-but-still-on-the-repositor 

 git ls-files -i -z --exclude-from=.gitignore | xargs -0 git rm --cached

(github) Create a new repository on the command line 

 touch README.md
 git init
 git add README.md
 git commit -m "first commit"
 git remote add origin git@github.com:username/repo.git
 git push -u origin master

(github) Push an existing repository from the command line 

 git remote add origin git@github.com:username/repo.git
 git push -u origin master

Generate fully-flavoured apk filename with app_name, versionName, versionCode, git hash

Generate fully-flavoured apk filename with app_name, versionName, versionCode, git hash https://gist.github.com/Wikinaut/d3eb229e1dd1ec703f00

Hardware: Audio mixer circuits

Let's Encrypt 

old:
./letsencrypt-auto renew --force-renewal --agree-tos --rsa-key-size 4096

lumen to watt

Leetspeek online generator

http://www.robertecker.com/hp/research/leet-converter.php

sed

  • replace twitterurls
sed -e 's/twitter.com\/\(.*\?\)\/status\/\([0-9]\+\)/twitter.com\/i\/status\/\2/' infile

sed: sed: joining lines depending on pattern starting on the second one 

sed 'N;s/\npattern/pattern/;P;D' file

Raspberry Pi

autostart methods

http://raspberry.tips/raspberrypi-einsteiger/raspberry-pi-autostart-von-skripten-und-programmen-einrichten

Nextcloud

jobs were killed

Jobs are killed after upgrade to 21?

/etc/php/8.0/cli/conf.d/20-apcu.ini: 
apc.enable_cli=1

or in /etc/php/7.3/cli/php.ini: 

apc.enable_cli=1

CORS header

If you want to openWebsite a shared video on an NEXTCLOUD instance, a change is needed. Add the rC3-Domain/s!

In /lib/public/AppFramework/Http/ContentSecurityPolicy.php: 

/** @var array Domains which can embed this Nextcloud instance */
protected $allowedFrameAncestors = [
'\'self\'',
'test.visit.at.wa-test.rc3.cccv.de',
];
Source: https://return2.net/nextcloud-enable-external-iframe/
Add a custom footer
Add a custom sidebar
Clone this wiki locally