✨ [$4.000 BOUNTY 💰] Improve VideoPipeline (lower-overhead, no OpenGL, all pixel-formats, auto-scaling)

[mrousavy]

What

$4.000 bounty to anyone who can solve this problem once and for all!!!!

One of VisionCamera's strong suits is the flexibility of it. You can configure the Camera for photo capture, video recording, frame processor, or even all at once.

This is roughly how VisionCamera's Camera Session is set up:

graph TD;
Camera-->Preview
Camera-->Photo
Camera-->VideoPipeline
VideoPipeline-->FP[Frame Processing]
VideoPipeline-->REC[Video Recording]

Loading

For the VideoPipeline, we have 6 requirements:

1. ✅ It needs to be as efficient as possible. If no Frame Processor is attached, we need fast recording (native pixel format). If no recording is attached, we need fast Frame Processing (no roundtrips or render-passes). And both should work at the same time.
2. ✅ It needs to support 3 pixel-formats: yuv, rgb and native (most efficient platform format)
3. ✅ Video Recording should support h.264 and h.265 and should be hardware-accelerated.
4. ✅ Video Recording should support Camera flipping (back <-> front) while recording and should auto-scale buffers to fit the size of the video.
5. ✅ Video Recording should be synchronous so that we can theoretically detect faces in MLKit, then draw a dog mask over them for the resulting recording.
6. ✅ Frame Processing should receive CPU/GPU buffers of the pixel data (see point 2. for formats). Ideally, this should be the android.media.Image type, but if that's not possible we can also use some other form of buffer (ByteBuffer/AHardwareBuffer)

iOS

On iOS, this was relatively easy to implement:

graph TD;
  Camera-->AVCaptureVideoDataOutput
  AVCaptureVideoDataOutput-->|"captureOutput(_:didOutput:from:)"| CMSampleBuffer
  CMSampleBuffer-->CVPixelBuffer-->FP["Frame Processor (e.g. MLKit)"]
  CMSampleBuffer-->AVAssetWriter

Loading

The CMSampleBuffer type is amazing, it exposes the GPU-backed buffer to the CPU (IOSurface), but can also fully stay on the GPU without any CPU copies if we don't use MLKit and only record to a file.

For the 6 main requirements:

1. ✅ It is as efficient as possible, it uses the most efficient native format (e.g. kCVPixelFormat_YCbCr4208BiFullRange) and doesn't involve any render passes or CPU copies.
2. ✅ We can fully configure the pixel formats (yuv, rgb or native) and the AVAssetWriter understands all of them.
3. ✅ AVAssetWriter can record either in h.264 or h.265 HEVC.
4. ✅ We can easily switch the Camera device (e.g. front <-> back) while recording, and the AVAssetWriter will handle scaling automatically if the buffers are a different size than what we originally configured it to.
5. ✅ The Video Pipeline is synchronous, we can theoretically first detect faces in ML then draw a dog mask above them for the recording.
6. ✅ We have access to a CMSampleBuffer, which is a GPU buffer. We can easily get CPU access on that using AVF APIs.

Android

On Android however, it seems like this approach is simply not possible. There is a new type in API 26 called HardwareBuffer which seems similar to CMSampleBuffer, but it is still not widely exposed in APIs and all of the Camera/android.media APIs expect surfaces.

Here's a few things I tried:

1. Separate outputs

Use separate Camera outputs, MediaRecorder and ImageReader - Does not work because the Camera only allows 3 outputs. We already have 3 (preview, photo, video). Also:

1. ✅ Is as efficient as possible, no client code involved if MediaRecorder is attached directly to the Camera output
2. ✅ We can configure pixel format for the ImageReader
3. ✅ We can configure h.264 or h.265
4. ❌ We cannot switch the Camera Devices while recording as the MediaRecorder's surface will be destroyed when I re-open the Camera
5. ❌ Our VideoPipeline is not synchronous.
6. ✅ We would receive an android.media.Image in the ImageReader's callback

2. ImageReader/ImageWriter

See #1789 + #1799 + #1834 for code

I tried to create an ImageReader/ImageWriter setup that just receives Images, then passes them through to the output Surface (MediaRecorder/MediaCodec):

graph TD;
  Camera-->ImageReader["VideoPipeline (ImageReader)"]
  ImageReader-->MLKit[MLKit Image Processing]
  ImageReader-->ImageWriter
  ImageWriter-->REC[MediaRecorder/MediaCodec]

Loading

This feels like closest to what we have on iOS and it seems like ImageReader/ImageWriter are really efficient as they are just moving buffers around. This is my code:

// TODO: Do I need those flags? Or no?
val flags = HardwareBuffer.USAGE_VIDEO_ENCODE or HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE
val readerFormat = ImageFormat.YUV_420_888 // (or PRIVATE or RGBA_8888)
imageReader = ImageReader.newInstance(width, height, readerFormat, MAX_IMAGES, flags) // <-- API 29+
// ...
val mediaRecorder = ...
mediaRecorder.prepare()
// TODO: Does this need to be ImageFormat.PRIVATE instead?
val writerFormat = readerFormat
imageWriter = ImageWriter.newInstance(mediaRecorder.surface, MAX_IMAGES, writerFormat) // <-- API 29+

imageReader.setOnImageAvailableListener({ reader ->
  val image = reader.acquireNextImage()
  imageWriter.queueInputImage(image)
}, handler)

...but I couldn't really get this to work. A really smart guy on StackOverflow said that it is not guaranteed that MediaRecorder/MediaCodec can be fed with Images from an ImageWriter, so sometimes it just silently crashes 🤦‍♂️

Also, I'm not sure what format the MediaRecorder/MediaCodec expects - so maybe this requires an additional conversion step:

graph LR;

R1["ImageReader (YUV)"]-->W1["ImageWriter (YUV)"]-->R2["ImageReader (PRIVATE)"]-->W2["ImageWriter (PRIVATE)"]-->REC["MediaRecorder/MediaCodec"]
R1-->MLKit[MLKit Image Processing]

Loading

...which is just ridiculous.

For the 6 main requirements:

1. ✅ Is efficient as far as I know. No rendering, just moving Images around.
2. ❌ We cannot configure pixel format for the ImageReader as the MediaRecorder/MediaCodec requires a PRIVATE format. If we feed it RGB/YUV data, it crashes.
3. ✅ We can configure h.264 or h.265
4. ❌ We cannot switch the Camera Devices while recording as both the ImageReader and the MediaRecorder are configured with a specific width/height, there's no scaling step involved here.
5. ✅ Our VideoPipeline would be synchronous starting from the point we receive an Image from the ImageReader.
6. ✅ We would receive an android.media.Image in the ImageReader's callback

3. Create a custom OpenGL Pipeline

See #1836 for code

I created a custom OpenGL pipeline that the Camera will render to, then we do a pass-through render pass to render the Frame to all the outputs:

graph TD;
Camera-->OpenGL["VideoPipeline (OpenGL)"]
OpenGL-->Pass[Pass-Through Shader]
Pass-->ImageReader-->MLKit[MLKit Image Processing]
Pass-->REC[MediaRecorder/MediaCodec]

Loading

But, this has four major drawbacks:

1. It's really really complex to build (I already built it, see this PR, so not a real problem tbh)
2. It seems like this is not as efficient as a ImageReader/ImageWriter approach, as we do an implicit RGB conversion and an actual render pass, whereas ImageReader/ImageWriter just moving Image Buffers around (at least as far as I understood this)
3. It only works in RGBA_8888, as OpenGL works in RGB. This means, our frame processor (MLKit) does not work if it is trained on YUV_420_888 data - this is a hard requirement.
4. It is not synchronous, the ImageReader gets called at a later point. We could not really use information from the Frame to decide what gets rendered later (e.g. to apply a face filter).

As for the 6 main requirements:

1. ❌ Is efficient, but it has quite a large overhead (creating the GL context and doing 2 render passes with the pass-through shaders). Compared to just moving Buffers around, this is quite a large overhead.
2. ❌ We cannot use whatever pixel-format we want - OpenGL always operates in RGBA_8888. I have no idea how to get a YUV_420_888 Image from there.
3. ✅ We can configure h.264 or h.265
4. ✅ We can switch the Camera device while recording as OpenGL automatically handles the scaling - nice!
5. ❌ Our VideoPipeline would not be synchronous, as we render to an ImageReader which calls the Frame Processor at some later point, whenever it has an Image available. This could be solved though by rendering to a HardwareBuffer wrapped as a Texture/FBO, which we can then wrap using our Frame type. But; we no longer have an Image.
6. ✅ We would receive an android.media.Image in the ImageReader's callback (unless we render to a HardwareBuffer; point 5.)

4. AHardwareBuffer

This is something I couldn't get working yet and I'm not sure if that's possible, but my theory is to receive HardwareBuffers (which should represent GPU memory afaik) instantly, then somehow pass them to a MediaRecorder/MediaCodec for encoding:

graph TD;
Camera-->|???|HW[HardwareBuffer]
HW-->FP["Frame Processing (e.g. MLKit)"]
HW-->REC[MediaRecorder/MediaCodec]

Loading

..but I have no idea how to get direct low-level access to such buffers from the Camera. Can the Camera only render to Surfaces? These Surface abstractions are really annoying.

5. FFmpeg

This is something I couldn't get working yet and I'm not sure if that's possible, but my theory is to use FFmpeg instead of MediaRecorder/MediaCodec to make the recording step simpler and more flexible:

graph TD;
Camera-->ImageReader
ImageReader-->FP["Frame Processing (e.g. MLKit)"]
ImageReader-->REC[FFmpeg]

Loading

..but I'm not sure if that would grant me any advantages. And also, I think ffmpeg is just using MediaRecorder/MediaCodec under the hood - so that's something I could build myself.

---

At this point I'm pretty clueless tbh. Is a synchronous video pipeline simply not possible at all in Android? I'd appreciate any pointers/help here, maybe I'm not aware of some great APIs.

Happy to pay $4.000 to anyone who comes up with a solution for this problem once and for all.

[mrousavy]

TL:DR; As of today, VisionCamera uses an OpenGL pipeline on Android. Downsides:

• 💥 It crashes when you try to use another format than RGB. E.g. if you pass pixelFormat="yuv" it will crash (YUV because your MLKit plugin might be trained on YUV). RGB also isn't the most efficient format (PRIVATE or YUV is).
• 🐌 This does introduce some additional overhead for recording videos as it converts from native Camera format to RGB, then renders it into the MediaRecorder. So it is theoretically slower than a native Camera2 app.
• 🧵 It is not synchronous, so the Frame Processor is called in parallel. If at some point I decide to add Skia to VisionCamera (e.g. V4), this architecture does not work so you can't detect faces first, then draw over the detected faces in the recording/preview because again, it runs in parallel, not synchronous.

Instead, I want to use an approach similar to how it works on iOS - just pass GPU buffers (AHardwareBuffer?) around. I tried to use ImageReader/ImageWriter, but those don't play well with the MediaRecorder/MediaCodec interfaces. So for now, I ran out of ideas. Pixel Format YUV just doesn't work in VisionCamera Android today. I just think Android doesn't have any APIs that support that, lmao.

If someone has any ideas, please comment here. $4.000 bounty if you can solve this problem.

Feel free to share this, e.g. if you know someone who works at Android/Google or someone with Camera2/android.media/OpenGL experience.

[mrousavy]

I found a temporary solution to the problem: #1874

Basically, I plug an ImageReader and an ImageWriter in-between the Camera and the OpenGL pipeline.

The problem here is that this does not support RGB, but at least it works in YUV and PRIVATE. I think most Frame Processors need YUV anyways.

As far as I can see, there is no real solution on Android right now.

Solution

I think this can be achieved if the Android OS adds two features:

1. ImageWriter should be capable of automatically resizing Images to the target surface's size (width + height)
2. MediaRecorder Surfaces should be capable of working in ImageFormat.YUV_420_888 and ImageFormat.PRIVATE formats.

I think 2. already works on most phones, but for some reason not on every phone. So 1. is the major point; if that works, we can scrap the entire OpenGL pipeline (a shit ton of C++ files).

I think I will create a feature request for that in the Android issue tracker, but they probably won't care about that tbh. iOS already supports this since iOS 12 😅

[mrousavy]

Lmao even ChatGPT says that in a perfect world Android would be able to handle frame resizing automatically in the GPU Encoder. (Same as iOS in AVAssetWriter)

It even says that this might be possible with custom SIMD instructions 😂

[mrousavy]

NEON (SIMD) only works on arm, and I'm definitely not gonna go down that rabbit hole 😂

[mrousavy]

CameraX has a stream sharing feature, this is kinda what I am aiming for. really cool stuff, no need for me to bring everything into OpenGL.

I hope it's as efficient as on iOS :)