h264_v4l2m2m acceleration in Raspberry Pi 4 - 64 bits

[nfj25]

Describe the bug
h264_v4l2m2m acceleration is broken in Raspberry Pi 4 64 bits. When trying to use it (Exynos V4L2 MFC), ffmpeg returns the error:

[h264_v4l2m2m @ 0x5587de52e0] Encoder requires yuv420p pixel format.

This is due to the fact that jellyfin is forcing this parameter (EncodingHelper.cs):

if (string.Equals(videoEncoder, "h264_v4l2m2m", StringComparison.OrdinalIgnoreCase))
            {
                param += " -pix_fmt nv21";
            }

But testing ffmpeg with:
-pix_fmt yuv420p

Worked and in my tests was much faster than using libx264 with the same parameters.

Probably, "Exynos V4L2 MFC" wasn't meant for Raspberry Pi. But as jellyfin/jellyfin#4023 OpenMax doesn't work in 64bits and because the recomended alternative is using h264_v4l2m2m (raspberrypi/Raspberry-Pi-OS-64bit#98) maybe we could try to use this instead. By the way, VAAPI option doesn't seam to be doing anything, it uses the normal libx264.

System (please complete the following information):

• OS: Raspbian (Arm64)
• Virtualization: Docker
• Jellyfin Version: 10.6.4
• Storage: local

To Reproduce

Doesn't work:

/usr/lib/jellyfin-ffmpeg/ffmpeg -i file:"/data/movies/input.mp4" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_v4l2m2m -pix_fmt nv21 -b:v 9990684 -maxrate 9990684 -bufsize 19981368 -level 41 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -g 72 -keyint_min 72 -sc_threshold 0 -vf "scale=trunc(min(max(iw\,ih*dar)\,1920)/64)*64:trunc(ow/dar/2)*2" -start_at_zero -vsync -1 -codec:a:0 libmp3lame -ac 2 -ab 224000 -af "volume=2" -copyts -avoid_negative_ts disabled -start_number 0 output.mp4

Works:

/usr/lib/jellyfin-ffmpeg/ffmpeg -i file:"/data/movies/input.mp4" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_v4l2m2m -pix_fmt yuv420p -b:v 9990684 -maxrate 9990684 -bufsize 19981368 -level 41 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -g 72 -keyint_min 72 -sc_threshold 0 -vf "scale=trunc(min(max(iw\,ih*dar)\,1920)/64)*64:trunc(ow/dar/2)*2" -start_at_zero -vsync -1 -codec:a:0 libmp3lame -ac 2 -ab 224000 -af "volume=2" -copyts -avoid_negative_ts disabled -start_number 0 output.mp4

[mlim15]

Removing the -pix_fmt argument (or changing it) does make V4L2 work on the Raspberry Pi , at least for 8-bit files. However, V4L2 on the raspi still seems to fail for me when exposed to 10-bit files, though they work on OpenMAX. This may be a driver-related issue regarding what capabilities are exposed in the video10 and video11 devices by the bcm2835-codec module. This may help somebody more knowledgeable than me know what's up - video10 is the decoding device, video11 is the encoder:

$ v4l2-ctl --all -d /dev/video10

Driver Info:
        Driver name      : bcm2835-codec
        Card type        : bcm2835-codec-decode
        Bus info         : platform:bcm2835-codec
        Driver version   : 5.10.6
        Capabilities     : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : bcm2835-codec
        Model            : bcm2835-codec
        Serial           : 0000
        Bus info         : platform:bcm2835-codec
        Media version    : 5.10.6
        Hardware revision: 0x00000001 (1)
        Driver version   : 5.10.6
Interface Info:
        ID               : 0x0300000c
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : bcm2835-codec-decode-source
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Source
          Link 0x02000008: to remote pad 0x1000004 of entity 'bcm2835-codec-decode-proc': Data, Enabled, Immutable
Priority: 2
Format Video Capture Multiplanar:
        Width/Height      : 640/480
        Pixel Format      : 'YU12' (Planar YUV 4:2:0)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Rec. 709
        Transfer Function : Default
        YCbCr/HSV Encoding: Default
        Quantization      : Default
        Plane 0           :
           Bytes per Line : 640
           Size Image     : 460800
Format Video Output Multiplanar:
        Width/Height      : 640/480
        Pixel Format      : 'H264' (H.264)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Rec. 709
        Transfer Function : Default
        YCbCr/HSV Encoding: Default
        Quantization      : Default
        Plane 0           :
           Bytes per Line : 0
           Size Image     : 524288
Selection Video Capture: compose, Left 0, Top 0, Width 640, Height 480, Flags:
Selection Video Capture: compose_default, Left 0, Top 0, Width 640, Height 480, Flags:
Selection Video Capture: compose_bounds, Left 0, Top 0, Width 640, Height 480, Flags:

User Controls

  min_number_of_capture_buffers 0x00980927 (int)    : min=1 max=1 step=1 default=1 value=1 flags=read-only

$ v4l2-ctl --all -d /dev/video11

Driver Info:
        Driver name      : bcm2835-codec
        Card type        : bcm2835-codec-encode
        Bus info         : platform:bcm2835-codec
        Driver version   : 5.10.6
        Capabilities     : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : bcm2835-codec
        Model            : bcm2835-codec
        Serial           : 0000
        Bus info         : platform:bcm2835-codec
        Media version    : 5.10.6
        Hardware revision: 0x00000001 (1)
        Driver version   : 5.10.6
Interface Info:
        ID               : 0x0300001a
        Type             : V4L Video
Entity Info:
        ID               : 0x0000000f (15)
        Name             : bcm2835-codec-encode-source
        Function         : V4L2 I/O
        Pad 0x01000010   : 0: Source
          Link 0x02000016: to remote pad 0x1000012 of entity 'bcm2835-codec-encode-proc': Data, Enabled, Immutable
Priority: 2
Format Video Capture Multiplanar:
        Width/Height      : 640/480
        Pixel Format      : 'H264' (H.264)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Rec. 709
        Transfer Function : Default
        YCbCr/HSV Encoding: Default
        Quantization      : Default
        Plane 0           :
           Bytes per Line : 0
           Size Image     : 524288
Format Video Output Multiplanar:
        Width/Height      : 640/480
        Pixel Format      : 'YU12' (Planar YUV 4:2:0)
        Field             : None
        Number of planes  : 1
        Flags             :
        Colorspace        : Rec. 709
        Transfer Function : Default
        YCbCr/HSV Encoding: Default
        Quantization      : Default
        Plane 0           :
           Bytes per Line : 640
           Size Image     : 460800
Selection Video Output: crop, Left 0, Top 0, Width 640, Height 480, Flags:
Selection Video Output: crop_default, Left 0, Top 0, Width 640, Height 480, Flags:
Selection Video Output: crop_bounds, Left 0, Top 0, Width 640, Height 480, Flags:

Codec Controls

             video_bitrate_mode 0x009909ce (menu)   : min=0 max=1 default=0 value=0 flags=update
                                0: Variable Bitrate
                                1: Constant Bitrate
                  video_bitrate 0x009909cf (int)    : min=25000 max=25000000 step=25000 default=10000000 value=10000000
           sequence_header_mode 0x009909d8 (menu)   : min=0 max=1 default=1 value=1
                                0: Separate Buffer
                                1: Joined With 1st Frame
         repeat_sequence_header 0x009909e2 (bool)   : default=0 value=0
                force_key_frame 0x009909e5 (button) : flags=write-only, execute-on-write
            h264_i_frame_period 0x00990a66 (int)    : min=0 max=2147483647 step=1 default=60 value=60
                     h264_level 0x00990a67 (menu)   : min=0 max=13 default=11 value=11
                                0: 1
                                1: 1b
                                2: 1.1
                                3: 1.2
                                4: 1.3
                                5: 2
                                6: 2.1
                                7: 2.2
                                8: 3
                                9: 3.1
                                10: 3.2
                                11: 4
                                12: 4.1
                                13: 4.2
                   h264_profile 0x00990a6b (menu)   : min=0 max=4 default=4 value=4
                                0: Baseline
                                1: Constrained Baseline
                                2: Main
                                4: High

Actually, the above problem happens when specifying mkv output. It seems to work with an mp4 file extension. Oops. :P

Anyway, for now (until this -pix_fmt issue is ironed out), you can still get hardware acceleration on the Pi through OpenMAX (even on a 64-bit OS) by using multiarch to install jellyfin/jellyfin-ffmpeg/dependencies as armhf so they can talk to the /opt/vc libraries.

Also, as you say, there is no VAAPI capability on the Pi, despite what the current Jellyfin documentation says. I did see the post on reddit that resulted in it being added to the documentation and I did try myself on Ubuntu Server 20.04.1; it simply does not work, and I cannot find any official mention of vaapi support either. Afaik we got the /dev/dri/renderD128 node because of changes in mesa and the new KMS/DRM graphics driver stack - its presence indicates support for DRI (?) but doesn't guarantee VAAPI. Mesa drivers that implement VAAPI have an extra $driverName_drv_video.so file in /usr/lib/$arch-linux-gnu/dri that we don't have. It may be worth opening a separate issue about that.

[nfj25]

Hello,

My /opt/vc libraries for 64bit raspbian dont have the required video libraries.
Please, how can I install them? I only could find them for 32 bit raspbian.

What is "jellyfin/jellyfin-ffmpeg/dependencies"? I cant find it...

[mlim15]

My /opt/vc libraries for 64bit raspbian dont have the required video libraries.
Please, how can I install them? I only could find them for 32 bit raspbian.

Under normal circumstances I believe you can install them with rpi-update (warning that the latest firmware commit is currently bugged and will ignore your config.txt. Don't run rpi-update right now, or specify the 5.10.6 commit if you do). Alternatively, they are also provided by the libraspberrypi0 package, but this probably won't install them to /opt/vc anymore as it's considered an "old" location. The Ubuntu wiki recommends symlinking /usr and /opt/vc to work around this. (The /opt/vc path for the libraries seems to be hardcoded into the ffmpeg builds, which is what one of the bugs you linked referenced.) If you're on 64 bit raspbian, you'd have to explicitly install the 32 bit version of libraspberrypi0 so that 32 bit binaries can link to them and libopenmaxil.so will be installed.

However, none of this matters anyway, because you're using docker. It's the container's responsibility to ensure it has access to the libraries it needs. It can't just use your system's libraries. I had a look at the dockerfile and it does install libraspberrypi0 but whether the libraries in the container are then available at /opt/vc is anyone's guess, though I would assume they are since there aren't tons of open "openmax broken in docker" tickets right now. (Edit: Actually, it doesn't put them in /opt/vc. But you can enter the docker container and symlink it yourself and it works until the container is restarted. Or you can pass through your local /opt/vc as a volume, if you have the libraries locally. This symlink should be added to the dockerfile.) You could try installing the armhf (32-bit) docker container and seeing if that works. (Edit: it does, but it's not seamless because it doesn't place the libraries in /opt/vc internally by default, and hardware decoding with h264_mmal requires you also expose /dev/vcsm-cma to the container, which is not mentioned in the documentation.) Unfortunately, encoding speeds are still sometimes less than 1x when the file you are transcoding is using the high profile at levels above hardware capabilities (I tested with a high 5.1 file).

What is "jellyfin/jellyfin-ffmpeg/dependencies"? I cant find it...

I meant "jellyfin and jellyfin-ffmpeg and their dependencies". If you're using docker the equivalent is installing the armhf docker container.

There are enough caveats to this process that it would probably be better to wait for V4L2 support to be fixed in Jellyfin rather than try to get OpenMAX working.

[kls0e]

There are enough caveats to this process that it would probably be better to wait for V4L2 support to be fixed in Jellyfin rather than try to get OpenMAX working.

btdt. +1.

[nyanmisaka]

OpenMax encoder is only available on arm32 since some runtime are missing on arm64.
Pi developers are working on better V4L2 support, we will move to it when it becomes stable.
https://github.com/jc-kynesim/rpi-ffmpeg/commits/dev/4.3.1/drm_prime_1

[mlim15]

I'm sorry, is there a workaround for this without docker until it's fixed? I have issues with a specific bridge setup when I use docker, and raspberry os 32bit has a bug that won't allow huge files over samba (kinda renders the NAS I'm trying to set up unusable for me)

I spent a couple hours trying to get it to work on my arm64 Debian Bullseye install without docker and didn't have any luck. I'm not sure why it isn't working, because I can copy-paste the exact ffmpeg command that Jellyfin is running and run it myself in a shell, and that does work fine, including h264_omx and h264_mmal hardware acceleration. And the default ffmpeg on my raspi is the one installed by jellyfin. But Jellyfin's transcode seems to somehow fail. No error in the ffmpeg log.

Sorry, I'm dumb. I had some permission errors with jellyfin user accessing /dev/vchiq. In my case, by default it was owned by root:root (not root:video) and had crw------ permissions.

It works fine once I ironed out my permission errors (with jellyfin user in the video group, I changed owner of /dev/vchiq to root:video and changed permissions to crw-rw---). This can be made to persist by changing udev rules for the device. I was testing at a root shell because I couldn't hang around in /var/log/jellyfin as my own user, so obviously it worked when logged in as root...

A couple snippets showing I am running a 64-bit system, OpenMAX acceleration is being used, and the installed libopenmaxil.so and jellyfin-ffmpeg are both 32-bit:

root@hostname:/var/log/jellyfin# uname -a
Linux hostname 5.10.6-v8+ jellyfin/jellyfin#1393 SMP PREEMPT Mon Jan 11 15:13:58 GMT 2021 aarch64 GNU/Linux

root@hostname:/var/log/jellyfin# dpkg --print-foreign-architectures
armhf

root@hostname:/var/log/jellyfin# more ffmpeg-transcode-a0555203-7236-40a2-bfec-e7dab113430d.txt | grep omx
/usr/lib/jellyfin-ffmpeg/ffmpeg -i file:"/mnt/fast/TV/test.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_omx  -b:v 256000 -maxrate 256000 -bufsize 512000  -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -g 72 -keyint_min 72 -sc_threshold 0 -vf "scale=trunc(min(max(iw\,ih*dar)\,426)/2)*2:trunc(ow/dar/2)*2" -start_at_zero -vsync -1 -codec:a:0 libmp3lame -ac 1 -ab 64000  -copyts -avoid_negative_ts disabled -f hls -max_delay 5000000 -hls_time 3 -individual_header_trailer 0 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/var/lib/jellyfin/transcodes/604791b2545baa6679c43d9e48ac66f1%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/var/lib/jellyfin/transcodes/604791b2545baa6679c43d9e48ac66f1.m3u8"
  configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-shared --disable-libxcb --disable-sdl2 --disable-xlib --enable-gpl --enable-version3 --enable-static --enable-libfontconfig --enable-fontconfig --enable-gmp --enable-gnutls --enable-libass --enable-libbluray --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --toolchain=hardened --enable-cross-compile --enable-omx --enable-omx-rpi --enable-mmal --arch=armhf --cross-prefix=/usr/bin/arm-linux-gnueabihf-
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_omx))
[h264_omx @ 0x380fb10] Using OMX.broadcom.video_encode
    Stream #0:0: Video: h264 (h264_omx), yuv420p, 426x318 [SAR 212:213 DAR 4:3], q=2-31, 256 kb/s, 23.98 fps, 90k tbn, 23.98 tbc (default)
      encoder         : Lavc58.91.100 h264_omx

root@hostname:/var/log/jellyfin# file /opt/vc/lib/libopenmaxil.so 
/opt/vc/lib/libopenmaxil.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, not stripped

root@hostname:/var/log/jellyfin# file /usr/lib/jellyfin-ffmpeg/ffmpeg
/usr/lib/jellyfin-ffmpeg/ffmpeg: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=7ae4353b5de77b43b5f0aeabc1b2bcba028de078, for GNU/Linux 3.2.0, stripped

Install libraspberrypi0 armhf,
symlink not the directory /opt/vc to the directory /usr ??

Edit: read a third time, I think you meant the BINARIES that might have been in /opt/vc, but weren't, is that correct?
I might be able to figure it out from there..

I haven't tested this as I just keep the old /opt/vc folder in my install and vanilla debian doesn't offer the libraspberrypi0 package, but I think this is the symlink you would need to make, it was suggested by the Ubuntu wiki as a result of the move:

$ ln -s /usr /opt/vc
(makes the link)
$ ls -l /opt
lrwxrwxrwx 1 root root 4 Feb 21 10:29 vc -> /usr
(what you should see)

You're making a symlink at /opt/vc that points to the directory /usr. This is because then when ffmpeg looks for /opt/vc/lib/libopenmaxil.so (for example) it will be directed to /usr/lib/libopenmaxil.so, which is where libraspberrypi0 now installs those libraries (please verify - you might need to make /opt/vc a folder that contains a symlink /opt/vc/bin linking to /usr/bin and a symlink /opt/vc/lib that links to /usr/lib/arm-linux-gnueabihf? Depends where your libopenmaxil.so and libbcm_host.so etc got installed.).

An alternative "workaround" - if you are okay with only one transcode at a time and you're using a Pi 4, software transcoding can handle most h264 transcodes at roughly 1x speed. I have had good luck doing this with these static builds (get the git master build for arm64).

[Darkyere]

Did it eventually work to install librasperrypi0 armhf and make a symlink?
Or wasn't the pi or jellyfin able to use it ?

[Darkyere]

I believe a lot of RPI users like me and a friend is interested in this being added to jellyfin.

So I'm writing here to keep the post alive.

Best regards and hope the feature will be added.

Darkyere

[jacobalberty]

Can we maybe get the V4L option made a bit more generic with pixel format broken out as a dropdown option? It looks like that + unchecking 10-bit content would fix most of the issues

[slungaro]

Same playback error for me. However in my case the error only occurs with .avi files.
mp4 and mkv files work as expected with no cpu load and accelerated.

Can anyone confirm?

build_version | Linuxserver.io version:- 10.7.7-1-ls134 Build-date:- 2021-10-07T04:33:17+02:00

Webinterface:
Playback Error
This client isn't compatible with the media and the server isn't sending a compatible media format.

Logs:
[14:15:43] [ERR] [87] Jellyfin.Api.Helpers.TranscodingJobHelper: FFmpeg exited with code 1

[dfunkt]

Raspberry Pi OS Bullseye has ffmpeg 4.3.3 available, maybe V4L2 support is now stable enough for use in Jellyfin?

[nyanmisaka]

Raspberry Pi OS Bullseye has ffmpeg 4.3.3 available, maybe V4L2 support is now stable enough for use in Jellyfin?

The newly developed V4L2 H264/H265 decoders are stable enough in rpi-ffmpeg. But we are still lacking of scale_v4l2m2m and deinterlace_v4l2m2m filters to avoid software filtering otherwise the transcoding speed will be crushed by the CPU bottleneck and memory copying.

[tr1p0p]

Too bad the hw encoding are at this state on RPI 4 64bits OS. I think it like the only feature that's trully miss.

[Sn00zEZA]

ran into this issue with my rpi4 and jellyfin in docker trying to use h264_v4l2m2m

Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> h264 (h264_v4l2m2m))
Stream #0:1 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
[h264_v4l2m2m @ 0x55a1718540] Using device /dev/video11
[h264_v4l2m2m @ 0x55a1718540] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x55a1718540] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x55a1718540] Encoder requires yuv420p pixel format.
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
[aac @ 0x55a171db10] Qavg: 65536.000
[aac @ 0x55a171db10] 2 frames left in the queue on closing
Conversion failed!

/usr/lib/jellyfin-ffmpeg/ffmpeg -autorotate 0 -i file:"/data/movies/Bluray-1080p.mkv" -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 h264_v4l2m2m -pix_fmt nv21 -b:v 2552000 -maxrate 2552000 -bufsize 5104000 -level 41 -force_key_frames:0 "expr:gte(t,0+n_forced*3)" -g:v:0 72 -keyint_min:v:0 72 -sc_threshold:v:0 0 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale=trunc(min(max(iw\,ih*dar)\,1280)/64)*64:trunc(ow/dar/2)*2,format=yuv420p" -codec:a:0 aac -ac 6 -ab 448000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename "/transcode/3684e786242cae71a38699487a97b21e%d.ts" -hls_playlist_type vod -hls_list_size 0 -y "/transcode/3684e786242cae71a38699487a97b21e.m3u8"

[JohannesBe]

It seems that in jellyfin/jellyfin#7291 some progress on this issue has occurred.

However, as discussed over there:

I saw it last year but the lacking of scale_v4l2m2m filter will also kill the speed.
We still need an implementation of HWA scaling for having notable performance gain.

After some digging it seems like a patch for V4L2 scaling has been open for almost three years now: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20190903010230.96236-21-ffmpeg@tmm1.net/

If I understand it correctly, we thus have all components required to effectuate a full, transferless, in-GPU encoding pipeline, but the loose ends have just not been tied together. I have opened an issue over at rpi-ffmpeg in order to try to get this patch upstreamed (well, or rather, downstreamed, mhm 🤔) a little bit quicker. Hopefully this will once and for all make HWA on a RPI happen 👏