ffmpeg h264_omx doesn't work

[gogobd]

Hello, every 01!

ffmpeg lists h264_omx as a supported Decoder/Encoder, however if I try to use it it says

[h264_omx @ 0x5578fb7e10] libOMX_Core.so not found7014:32:22.77 bitrate= -0.0kbits/s speed=N/A
[h264_omx @ 0x5578fb7e10] libOmxCore.so not found

I tried to compile ffmpeg from source which worked except that it complains that it can't find the headers:

ERROR: OpenMAX IL headers from raspberrypi/firmware not found

I have installed libomxil-bellagio-dev but can't get hardware support for ffmpeg either way.

[lurch]

Possibly related to the MMAL changes? #10 & #86 (comment)

[gogobd]

It's really hard for me to tell what's missing, however the speed impact of losing OMX ist remarkable. I noticed a header file is present:

/usr/include/bellagio/omxcore.h

But including it didn't help. When I tried to compile ffmpeg from source I followed this tutorial "Compiling FFmpeg on your Raspberry Pi" and this is my configure step:

./configure --extra-cflags="-I/usr/local/include" --extra-ldflags="-L/usr/local/lib" --extra-libs="-lpthread -lm -latomic" --enable-gmp --enable-gpl --enable-libaom --enable-libass --enable-libdav1d --enable-libdrm --enable-libfdk-aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopus --enable-librtmp --enable-libsnappy --enable-libsoxr --enable-libssh --enable-libvorbis --enable-libvpx --enable-libzimg --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-nonfree --enable-omx --enable-omx-rpi --enable-version3 --target-os=linux --enable-pthreads --enable-openssl --enable-hardcoded-tables

[gogobd]

This post suggests that mmal doesn't affect omx, though:

https://stackoverflow.com/questions/40175644/ffmpeg-hardware-acceleration-on-raspberry-pi

--enable-mmal – Enable Broadcom Multi-Media Abstraction Layer (Raspberry Pi) via MMAL. For hardware decoding of H.264, VC-1, MPEG-2, MPEG-4. As a dependency you'll need the linux-raspberrypi-headers (Arch Linux) or linux-headers-*-raspi2 (Ubuntu) package which provides the required header file mmal.h.

--enable-omx-rpi – Enable OpenMAX IL code for Raspberry Pi. For hardware encoding of H.264 (encoder is named h264_omx) and MPEG-4 (mpeg4_omx). As a dependency you'll need the libomxil-bellagio (Arch Linux) or libomxil-bellagio-dev (Ubuntu) package which provides the required header file OMX_Core.h.

[6by9]

OpenMax IL will NEVER be supported on 64bit. The hell involved in mapping 64bit pointers to pass to the 32bit VPU just isn't worth it.

The V4L2 stateful codec API is the preferred approach for codecs, and is available in both 32 and 64bit OSes. In FFmpeg use h264_v4l2m2m instead.

[2bdkid]

@6by9 does h264_v4l2m2m use any hardware support? I'm not familiar with it.

[6by9]

does h264_v4l2m2m use any hardware support? I'm not familiar with it.

Yes, it's a V4L2 API wrapper on top of the MMAL components, but doing that allows us to add 64bit compatibility relatively easily.
Userspace MMAL usage is currently not compatible with 64bit, but will be fixed up when resource allows.

[2bdkid]

@6by9 So I'm curious now. On my RPI4 (64bit), ffmpeg has the h264_v4l2m2m encoder available - I take it it's not using omx underneath so then what is it actually doing? Some other software encoder through MMAL?

[lurch]

A quick search finds https://www.kwasi-ich.de/blog/2017/11/26/omx/ which seems to suggest that OpenMax is a portable-wrapper on top of the Broadcom-specific MMAL.
But https://www.raspberrypi.org/documentation/raspbian/applications/camera.md says "All the applications are driven from the command line, and written to take advantage of the MMAL API which runs over OpenMAX. The MMAL API provides an easier to use system than that presented by OpenMAX. Note that MMAL is a Broadcom-specific API used only on VideoCore 4 systems."
🤷‍♂️

[6by9]

h264_v4l2m2m uses the V4L2 stateful decoder API to then access the hardware codec (encode and decode) via the MMAL framework.

MMAL and OpenMAX IL both share the components at the lowest level. That was termed internally as Reduced-IL (or RIL) as it removes all the common state machine stuff from the individual components to give a reduced API.

MMAL and IL provide two different routes down to the RIL level, with MMAL being the more recent, and tries to get rid of a load of the bits of IL that caused grief (particularly all the asynchronous callback for completing state transitions and the like).

The MMAL framework code within the kernel has been adapted to handle 64bit kernels. Where the client needs to pass a 64bit pointer through a 32bit IPC field, it goes through an allocation and lookup table.
The MMAL userspace library did have the same done (raspberrypi/userland#586), but it has currently been reverted due to observed issues.

If you felt like picking IL apart to do the same, then a PR would be welcome.
It's far from straightforward though as for a start you have 6 userspace pointers stored in OMX_BUFFERHEADERTYPE to start with, each of which need to be picked apart, stored, and looked back up again on buffer return. All that is open source in https://github.com/raspberrypi/userland/blob/master/interface/vmcs_host/vcilcs.c (VideoCore IL Component Service) and the other counterpart files, but it's not the easiest code to follow.
It's bad enough that there are also non-natively packed structs in the default IL headers (the nTimeStamp field in OMX_BUFFERHEADERTYPE being the most annoying), so you'd need to repack that struct between 64bit and the IPC.

[lurch]

Thanks for the detailed info @6by9 ! 👍

[gogobd]

h264_v4l2m2m produces green video output - so I would consider it broken. I also wonder where this could be reported.

[6by9]

Green output but a visible image? The chroma planes are set to 0.
I will double check, but I'm 99.99% certain it's an FFmpeg bug that is resolved in their master but not in the Debian version.

Following normal Debian policies of not changing major version within a release you won't find FFmpeg being significantly bumped, but we can see if that fix can be simply backported.

[JooJooBee666]

I'm still waiting on this before I can even use the x64 version here. And it sucks as I'm doing live encoding of video stream from a queue of frames in RAM (doing surveillance with motion detection). The 2GB limit is killing me. But, this is a deal breaker. Is there just no hope this will ever be resolved?

[6by9]

As #98 (comment),

OpenMax IL will NEVER be supported on 64bit.

I thought h264_v4l2m2m had been fixed, but haven't double checked.

[JooJooBee666]

Hmm, I missed that. I just saw this issue was still open. I'll test out ffmpeg with h264_v4l2m2m instead. I guess if that is working, and knowing the history with OMS, maybe this should be closed with "won't fix" and a final reference to use the other hardware encoding option would be best?

[gogobd]

Please leave a comment how to get hardware support for decoding / encoding with ffmpeg if you find a solution! I'm still using older Pis which are much faster because they do support this.

[Andrei-Iosifescu123]

h264_v4l2m2m produces green video output - so I would consider it broken. I also wonder where this could be reported.

I have the same problem.

[popcornmix]

Make sure you are on an updated RpiOS bullseye image.
If you still have an issue then you can report it here.
Include enough detail to reproduce the problem.

[Andrei-Iosifescu123]

Make sure you are on an updated RpiOS bullseye image. If you still have an issue then you can report it here. Include enough detail to reproduce the problem.

I'm on Ubuntu 20.04

[ghollingworth]

I think you should report it to Ubuntu then

[gogobd]

I've seen this green output problem on other architectures, too. It could be a ffmpeg problem.

[JamesH65]

IIRC, an all-green screen is what you get when a YUV image is entirely 0s.

[Andrei-Iosifescu123]

I think you should report it to Ubuntu then

It works when compiling ffmpeg from source, but not when installing with apt get-install ffmpeg.

[popcornmix]

It works when compiling ffmpeg from source, but not when installing with apt get-install ffmpeg.

Which suggests Ubuntu are building the apt ffmpeg without support for this.
Either not using the source you are building from, or not configuring with the required options.

That's not something we can help you with here (on an RPiOS issue tracker).
You need to report this to Ubuntu.