Support zerocopy from MMAL camera to video encoding

[jasaw]

1. version [x.y.z, hash, other]: 9f4f0e9
2. installed as a package or compiled from sources [deb, rpm, git, other]: compiled from source
3. standalone or part of third party [motion, MotionEyeOS, other]: standalone
4. video stream source [V4L (card or USB), net cam (mjpeg, rtsp, other), mmal]: mmal
5. hardware [x86, ARM, other]: All Raspberry Pi models with Pi Camera
6. operating system [Linux (which), FreeBSD, other]: Linux

The idea is to take the frames from MMAL camera, process them in motion, and pass the processed frames to the encoder, all without copying the frame, i.e. zerocopy. Zerocopy will greatly improve the maximum resolution and framerate a system can handle, and will also reduce memory usage.

[Mr-Dave]

Is this like the pass through #124 and substream #123 enhancements that we already have listed but instead doing it for MMAL?

If there is a concern on the framerate and resolution, has the bcm module been evaluated to determine whether it provides better performance? e.g. sudo modprobe bcm2835-v4l2 creates a /dev/video device that Motion can capture images from. This modprobe method is going to be added into our documentation as the preferred method for using the pi camera since we now know we can not package the mmal feature.

[jasaw]

This is not pass-through recording nor sub-stream processing. It is also not simply MMAL providing better performance than bcm2835-v4l2.

This is how the zerocopy setup looks like:
MMAL raw frames (stored in GPU/ARM shared memory) ---> motion (motion detection & draw text & timestamp on the frame in shared memory) ---> h264_omx (encodes the edited frame without copying the frame into GPU memory).

I have been testing various setup that uses h264_omx encoder on all models of Raspberry Pis, and the bottleneck on all of them is the memory bandwidth (from copying frames everywhere). This zerocopy feature aims to eliminate the bottleneck, thus giving big performance improvement.

[Mr-Dave]

The concern on this would be that it would be very platform centric.

While there are many PI users, I don't believe that the majority of them would be able to use the feature since it couldn't be packaged. (See issue on Stretch and the MMAL camera). So the only users would be those that have the knowledge to build the application which isn't where I was wanting to move the application to. I was really trying to push the project to be a "packaged" application rather than a "build from source" application.

Now with that said, there is some room for negotiation. If the changes are benefiting all platforms, or are consolidated and isolated into a separate module, then we can evaluate it. However if the changes would be putting #IFs into a large number of modules/functions, then I'm not sure that it is really suited for the project.

[jasaw]

I had a quick look at the motion source code to see how much change is needed. Basically, these are the changes required:

• first of all, add the zerocopy option and only enable if zerocopy chain is available, i.e. MMAL source and h264_omx sink. I expect this to be a new function, well isolated.
• instead of motion doing buffer allocation to store frames, have the option of accepting pre-allocated buffer instead. To make it even more generic, and more similar to libav and OMX, motion allocates buffer headers, and optionally allocate buffers.
• add zerocopy option to struct ffmpeg and enable it accordingly.
• finally, hand over the frame to h264_omx encoder, which requires pull-request Extend ffmpeg_video_codec config to take optional preferred_codec #476, and pull-request Tune h264_omx encode video quality #477.

The main motivation of zerocopy feature is to benefit MotionEyeOS users. Having said all that, I haven't proven that this zerocopy chain will work. I was having issues with zerocopy, reported as issue #433 and raspberrypi/firmware#851.

[jasaw]

This is way too difficult to get working properly. It requires a lot of code changes in motion to handle the buffers. After all that, it still locks up with certain versions of kernel. When it doesn't lock up, it works great, but encoding pauses for 2 seconds after every 20 seconds.

[dfgweb]

Hi, currently working a little to use RPI HW encoder but instead of using the h264_omx, I'm going to use MMAL directly (like raspivid does).

Using the prefered codec you've added, I think the following changes will work:

• Create a mmalcodec.[ch] based on mmalcam.[ch] with a all required functions to initialise MMAL codec and feed frame to codec (raspvid.c is a good starting point).
• Change ffmpeg.c a little to use 'copy' codec if prefered codec is something like movie_codec = mkv:h264_mmal
• Update call mmalcodec_put_image instead of ffmpeg_put_image and call ffmpeg_put_image in 'mmalcodec_buffer_callback`.

This result in the following buffer pipe:

MMAL raw frames (stored in GPU/ARM shared memory) ---> motion (motion detection & draw text & timestamp on the frame in shared memory) ---> h264_mmal (encodes the edited frame without copying the frame into GPU memory) -> ffmpeg_put_image (using copy codec)

Change will be limited in ffmpeg.c.

What do you think of it?

[dfgweb]

Sorry, this is a very late comment, but the bugs is still present in RPI for h264_omx prefered_codec (I've just checked and still deadlock).

[jasaw]

@dfgweb I have implemented the zero copy version some time ago, and I had to make significant changes to motion for zero copy to work. From memory, these are the changes I did:

• Change mmalcam.c to enable zero copy. Basically set MMAL_PARAMETER_ZERO_COPY, call mmal_port_pool_create to create the image buffer pool with size of (pre_capture + minimum_motion_frames).
• Add a mmal_buffer_header_release call in mmalcam.c to release the image buffer back to MMAL.
• Replace all memory manipulation calls (memcpy, malloc, free...) in entire motion code base with proper handling of image buffer, i.e. keep track of which image buffer is still in use, return the image buffer to MMAL pool when no one is referencing it anymore, ...

Essentially, motion has to receive this image buffer from mmalcam, pass it along to motion's image ring buffer (used for motion detection & pre-capture), and pass it to ffmpeg(h264_omx) for encoding, and when ffmpeg is done with the image, release the image buffer back to MMAL pool.

You can replace ffmpeg part by calling MMAL directly to encode, but you still need to do all the proper image buffer handling.

I did a quick hack as a proof of concept, and it was more trouble than it's worth for various reasons:

• Significant changes to motion code means it's very unlikely to be merged into motion.
• Not sure whether zerocopy OMX_EmptyThisBuffer hanging issue has been fixed.

I gave up at that point because it wasn't worth it. If you want to look at my hack, you can find it here: https://github.com/jasaw/motion/tree/support-mmal-zerocopy
I don't know what state it's in, and I'm sure there will be things that are horribly broken.