Merge video streams side by side results in audio stream missing

[blgo]

The pull request claims that #26 has been fixed but I cannot find any of the methods proposed there.

I can't get the audio stream comming throught, which does show on probe:

ffmpeg.probe('SampleVideo_360x240_1mb.mp4')

{'streams': [
{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'unknown', 'profile': '77', 'codec_type': 'video', 'codec_time_base': '1/30', 'codec_tag_string': 'avc1', 'codec_tag': '0x31637661', 'width': 320, 'height': 240, 'coded_width': 320, 'coded_height': 240, 'has_b_frames': 0, 'sample_aspect_ratio': '1:1', 'display_aspect_ratio': '4:3', 'pix_fmt': 'yuv420p', 'level': 13, 'chroma_location': 'left', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '15/1', 'avg_frame_rate': '15/1', 'time_base': '1/15360', 'start_pts': 0, 'start_time': '0.000000', 'duration_ts': 209925, 'duration': '13.666992', 'bit_rate': '229387', 'bits_per_raw_sample': '8', 'nb_frames': '205', 'disposition': {'default': 1, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}, 'tags': {'creation_time': '1970-01-01T00:00:00.000000Z', 'language': 'und', 'handler_name': 'VideoHandler'}}, 
{'index': 1, 'codec_name': 'aac', 'codec_long_name': 'unknown', 'profile': '1', 'codec_type': 'audio', 'codec_time_base': '1/48000', 'codec_tag_string': 'mp4a', 'codec_tag': '0x6134706d', 'sample_fmt': 'fltp', 'sample_rate': '48000', 'channels': 6, 'channel_layout': '5.1', 'bits_per_sample': 0, 'r_frame_rate': '0/0', 'avg_frame_rate': '0/0', 'time_base': '1/48000', 'start_pts': 0, 'start_time': '0.000000', 'duration_ts': 657408, 'duration': '13.696000', 'bit_rate': '382488', 'max_bit_rate': '416704', 'nb_frames': '642', 'disposition': {'default': 1, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}, 'tags': {'creation_time': '1970-01-01T00:00:00.000000Z', 'language': 'und', 'handler_name': 'SoundHandler'}}], 'format': {'filename': 'SampleVideo_360x240_1mb.mp4', 'nb_streams': 2, 'nb_programs': 0, 'format_name': 'mov,mp4,m4a,3gp,3g2,mj2', 'start_time': '0.000000', 'duration': '13.696000', 'size': '1053651', 'bit_rate': '615450', 'probe_score': 100, 'tags': {'major_brand': 'isom', 'minor_version': '512', 'compatible_brands': 'isomiso2avc1mp41', 'creation_time': '1970-01-01T00:00:00.000000Z', 'encoder': 'Lavf53.24.2'}}}

The only thing that works is extracting the audio using:

import subprocess
command = "ffmpeg -i SampleVideo_360x240_1mb.mp4 -ab 160k -ac 2 -ar 44100 -vn audio.wav"
subprocess.call(command, shell=True)

and adding it at the end with:

file_1 = ffmpeg.input('SampleVideo_360x240_1mb.mp4')
file_2 = ffmpeg.input('SampleVideo_360x240_1mb.mp4')
file_3 = ffmpeg.input('audio.wav"')
merged = ffmpeg.filter([file_1, file_2], 'hstack')
merged.output(file_3, 'out.mp4').run()

Am I doing anything wrong or this is a bug?

This is the log when I output without the extracted audio file.

ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 10.0.0 (clang-1000.11.45.5)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.3_1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
Input #0, matroska,webm, from 'us4zanesgfqmdjnk_CAMERA_PXYSV.webm':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2019-05-15T14:09:45.000000Z
  Duration: 00:02:58.70, start: 0.000000, bitrate: 511 kb/s
    Stream #0:0(eng): Video: vp8, yuv420p(progressive), 640x480, SAR 1:1 DAR 4:3, 29.83 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
    Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      title           : Audio
Input #1, matroska,webm, from '1hvtxe22eubmjdz6_CAMERA_TTKPX.webm':
  Metadata:
    encoder         : GStreamer matroskamux version 1.8.1.1
    creation_time   : 2019-05-15T14:09:39.000000Z
  Duration: 00:02:56.88, start: 0.000000, bitrate: 410 kb/s
    Stream #1:0(eng): Video: vp8, yuv420p(progressive), 640x480, SAR 1:1 DAR 4:3, 30 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      title           : Video
    Stream #1:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
    Metadata:
      title           : Audio
File 'out.mp4' already exists. Overwrite ? [y/N] y
Stream mapping:
  Stream #0:0 (vp8) -> scale
  Stream #1:0 (vp8) -> scale
  hstack -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[libx264 @ 0x7fb9b1065800] using SAR=1/1
[libx264 @ 0x7fb9b1065800] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x7fb9b1065800] profile High, level 2.1
[libx264 @ 0x7fb9b1065800] 264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf58.20.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 640x240 [SAR 1:1 DAR 8:3], q=-1--1, 29.83 fps, 11456 tbn, 29.83 tbc (default)
    Metadata:
      encoder         : Lavc58.35.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
frame= 5304 fps=308 q=-1.0 Lsize=    2944kB time=00:02:57.68 bitrate= 135.7kbits/s dup=530 drop=3957 speed=10.3x    
video:2883kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.118943%
[libx264 @ 0x7fb9b1065800] frame I:22    Avg QP:16.81  size: 21073
[libx264 @ 0x7fb9b1065800] frame P:1660  Avg QP:20.03  size:  1240
[libx264 @ 0x7fb9b1065800] frame B:3622  Avg QP:26.90  size:   119
[libx264 @ 0x7fb9b1065800] consecutive B-frames:  6.1%  6.8%  5.6% 81.5%
[libx264 @ 0x7fb9b1065800] mb I  I16..4:  6.5% 52.4% 41.1%
[libx264 @ 0x7fb9b1065800] mb P  I16..4:  0.2%  0.5%  0.2%  P16..4: 15.0%  6.8%  5.1%  0.0%  0.0%    skip:72.2%
[libx264 @ 0x7fb9b1065800] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  9.8%  0.5%  0.1%  direct: 0.1%  skip:89.4%  L0:32.6% L1:62.2% BI: 5.2%
[libx264 @ 0x7fb9b1065800] 8x8 transform intra:56.2% inter:59.4%
[libx264 @ 0x7fb9b1065800] coded y,uvDC,uvAC intra: 71.8% 81.9% 62.4% inter: 3.2% 3.4% 0.4%
[libx264 @ 0x7fb9b1065800] i16 v,h,dc,p: 41% 12% 11% 36%
[libx264 @ 0x7fb9b1065800] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 21% 19%  5%  8%  8%  7%  5%  7%
[libx264 @ 0x7fb9b1065800] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 22% 10%  5%  9%  9%  8%  6%  7%
[libx264 @ 0x7fb9b1065800] i8c dc,h,v,p: 48% 18% 24%  9%
[libx264 @ 0x7fb9b1065800] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x7fb9b1065800] ref P L0: 72.2% 13.4% 11.2%  3.2%
[libx264 @ 0x7fb9b1065800] ref B L0: 93.4%  5.5%  1.0%
[libx264 @ 0x7fb9b1065800] ref B L1: 96.5%  3.5%
[libx264 @ 0x7fb9b1065800] kb/s:132.82

Reproduced on MAC OS 10.13.6
ffmpeg-python 0.1.17
ffmpeg version 4.1.3

Docker image sampeka/ffmpeg-python:3.6.5 Alpine Linux v3.5
ffmpeg-python 0.1.17
ffmpeg version 4.0

[kkroening]

Processing streams with both audio and video in ffmpeg gets weird. Take a look at this example for some ideas.

In the details of #26, it's mentioned that a lot of this is just weird ffmpeg behavior and not directly ffmpeg-python, though some day we might be able to work around ffmpeg's weirdness a bit on the ffmpeg-python side.

In any case, take a look at that example and see if you can adapt the patterns to your use case.

(Bonus points if you post a working example for others to refer to)

[rsomani95]

I have a humbler example below that's reproduced from #184 , and is slightly different from the one linked above as it doesn't rely on input.audio or input.video.

vid = (
    input
    .trim(start = 100, end = 160)
    .setpts('PTS-STARTPTS')
)
aud = (
    input
    .filter_('atrim', start = 100, end = 160)
    .filter_('asetpts', 'PTS-STARTPTS')
)

joined = ffmpeg.concat(vid, aud, v=1, a=1)
output = ffmpeg.output(joined[0], joined[1], 'out.mp4')
output.run()