🐛 Crashes on getPlanes() for YUV format (V3 RC5) #1714

Zuxelus · 2023-08-23T16:28:50Z

What were you trying to do?

Updating from 3.0.0-rc.4 to 3.0.0-rc.5

Reproduceable Code

const frameProcessor = useFrameProcessor(frame => {
    'worklet';
    examplePlugin(frame);
  }, []);

<Camera
  frameProcessor={frameProcessor}
  pixelFormat={'yuv'}

//Took original example plugin from code and added 2 lines to ExampleFrameProcessorPlugin.callback
Image image = frame.getImage();
Log.d("ExamplePlugin", Integer.toString(image.getPlanes().length));

What happened instead?

The app crashes on image.getPlanes(). The same when I use frame.toByteBuffer() instead of image.getPlanes().

Relevant log output

ExamplePlugin           com.test D  1280 x 960 Image with format #35. Logging 5 parameters:
com.test                com.test E  lockYCbCrImpl:341 failed with 'Error3::BAD_VALUE'
AndroidMediaUtils       com.test W  lockAsyncYCbCr failed with error 3 (format = 0x23)
com.test                com.test E  lockImpl:295 failed with 'Error3::BAD_VALUE'
Gralloc3                com.test W  lock(0x748046aa69e0, ...) failed: 3
AndroidMediaUtils       com.test E  Lock buffer failed!
AndroidMediaUtils       com.test E  lockImageFromBuffer: lock graphic buffer failed
System.err              com.test W  java.lang.RuntimeException: lock buffer failed for format 0x23
System.err              com.test W  	at android.media.ImageReader$SurfaceImage.nativeCreatePlanes(Native Method)
System.err              com.test W  	at android.media.ImageReader$SurfaceImage.getPlanes(ImageReader.java:1289)
System.err              com.test W  	at com.test.ExampleFrameProcessorPlugin.callback(ExampleFrameProcessorPlugin.java:52)
System.err              com.test W  	at com.mrousavy.camera.frameprocessor.FrameProcessor.call(Native Method)
System.err              com.test W  	at com.mrousavy.camera.CameraSession.onVideoFrameCaptured(CameraSession.kt:227)
System.err              com.test W  	at com.mrousavy.camera.utils.outputs.CameraOutputs._init_$lambda$4(CameraOutputs.kt:130)
System.err              com.test W  	at com.mrousavy.camera.utils.outputs.CameraOutputs.$r8$lambda$Rkf6S8U9W_w2j3C0-q1bBmy5uA0(Unknown Source:0)
System.err              com.test W  	at com.mrousavy.camera.utils.outputs.CameraOutputs$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:2)
System.err              com.test W  	at android.media.ImageReader$1.run(ImageReader.java:947)
System.err              com.test W  	at android.os.Handler.handleCallback(Handler.java:958)
System.err              com.test W  	at android.os.Handler.dispatchMessage(Handler.java:99)
System.err              com.test W  	at android.os.Looper.loopOnce(Looper.java:205)
System.err              com.test W  	at android.os.Looper.loop(Looper.java:294)
System.err              com.test W  	at android.os.HandlerThread.run(HandlerThread.java:67)

Device

Android

VisionCamera Version

3.0.0-rc.5

Additional information

I am using Expo
I have read the Troubleshooting Guide
I agree to follow this project's Code of Conduct
I searched for similar issues in this repository and found none.

moriax · 2023-08-25T22:35:21Z

I have the same Problem with "3.0.0-rc.8"
Maybe issue with Android Simulator ?

robinsoncol · 2023-08-26T05:42:48Z

@mrousavy Images passed into Frame seems to be of ImageFormat.PRIVATE (34) for some reason.

moriax · 2023-08-26T06:34:46Z

@robinsoncol
you need to pass pixelFormat='yuv' to the camera component, then it will be YUV_420_888

robinsoncol · 2023-08-26T08:17:55Z

@moriax Thanks, that actually solved an issue that I was having. Is that documented anywhere? Also, shouldn't that be the default pixelFormat value for the component?

Zuxelus · 2023-08-26T10:18:01Z

Yes. This error I have only in Android Simulator. I have pixelFormat='yuv' and I get correct format = 35.
But I think that my problem may be somewhere in my JS code. When I tried on the real device I've got different error
AndroidRuntime com.test E FATAL EXCEPTION: mrousavy/VisionCamera.video Process: com.test, PID: 22730 com.facebook.jni.CppException: Compiling JS failed: 1:1:invalid empty parentheses '( )' Buffer size 3 starts with: 280a29 at com.mrousavy.camera.frameprocessor.FrameProcessor.call(Native Method) at com.mrousavy.camera.CameraSession.onVideoFrameCaptured(CameraSession.kt:238) at com.mrousavy.camera.utils.outputs.CameraOutputs._init_$lambda$4(CameraOutputs.kt:130) at com.mrousavy.camera.utils.outputs.CameraOutputs.$r8$lambda$Rkf6S8U9W_w2j3C0-q1bBmy5uA0(Unknown Source:0) at com.mrousavy.camera.utils.outputs.CameraOutputs$$ExternalSyntheticLambda1.onImageAvailable(Unknown Source:2) at android.media.ImageReader$1.run(ImageReader.java:916) at android.os.Handler.handleCallback(Handler.java:942) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loopOnce(Looper.java:240) at android.os.Looper.loop(Looper.java:351) at android.os.HandlerThread.run(HandlerThread.java:67)

Zuxelus · 2023-08-27T10:07:01Z

Finally solved the error. In my case the problem was in react-native-reanimated. Versions 3.4.0 and 3.4.1 have the bug #4838 in code and break the FrameProcessor call. Bug can be closed.

mrousavy · 2023-08-28T10:26:21Z

Hey - yes the pixelFormat has to be specified - i don't know which pixel format you want. Per default, it is the most efficient platform native format, which is PRIVATE on Android, and YUV420 on iOS.

I'll add some docs on that for FPs :)

Also, rewriting the video pipeline now so the getPlanes and pixel formats will be greatly improved.

moriax · 2023-08-31T15:57:17Z

@mrousavy
finally I figured out the root cause of the reported error: java.lang.RuntimeException: lock buffer failed for format 0x23

I used your Stickman example app and just added a simple FrameProcessor to process the image with googles MLKit which triggered the same error reported as in this issue.
(Exception is thrown in .getPlanes() call or InputImage.fromMediaImage call from MLKit.

The reason for the error is, that you use always the following Flags in CameraOutput:
val flags = HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE or HardwareBuffer.USAGE_VIDEO_ENCODE
But in the Android Documentation is written that this flags should only be used for: ImageFormat.PRIVATE
If we use yuv format, we need to use other flag:
val flags = HardwareBuffer.USAGE_CPU_READ_RARELY;

https://developer.android.com/reference/android/media/ImageReader#newInstance(int,%20int,%20int,%20int,%20long)
so the simple fix is: set the flags based on you video.format variable

mrousavy · 2023-09-01T13:30:11Z

Gotcha! I updated the implementation of all of that to now use a default ImageReader, I think that would work better

Zuxelus · 2023-09-02T11:41:31Z

Tried version 3.0.0. Having new error.

ImageReader_JNI         com.test E  Producer output buffer format: 0x1, ImageReader configured format: 0x23
AndroidRuntime          com.test E  FATAL EXCEPTION: main
	Process: com.test, PID: 18726
	java.lang.UnsupportedOperationException: The producer output buffer format 0x1 doesn't match the ImageReader's configured buffer format 0x23.
		at android.media.ImageReader.nativeImageSetup(Native Method)
		at android.media.ImageReader.acquireNextSurfaceImage(ImageReader.java:597)
		at android.media.ImageReader.acquireNextImage(ImageReader.java:652)
		at android.media.ImageReader.acquireLatestImage(ImageReader.java:541)
		at com.mrousavy.camera.core.VideoPipeline.getImageReader$lambda$2(VideoPipeline.kt:99)
		at com.mrousavy.camera.core.VideoPipeline.$r8$lambda$RBt5oiVcrkwyTbFDf1xzJJfIdPg(Unknown Source:0)
		at com.mrousavy.camera.core.VideoPipeline$$ExternalSyntheticLambda0.onImageAvailable(Unknown Source:2)
		at android.media.ImageReader$1.run(ImageReader.java:947)
		at android.os.Handler.handleCallback(Handler.java:958)
		at android.os.Handler.dispatchMessage(Handler.java:99)
		at android.os.Looper.loopOnce(Looper.java:205)
		at android.os.Looper.loop(Looper.java:294)
		at android.app.ActivityThread.main(ActivityThread.java:8176)
		at java.lang.reflect.Method.invoke(Native Method)
		at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
		at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)

mrousavy · 2023-09-04T12:49:08Z

What ImageFormat/PixelFormat is 0x1 and 0x23? I think 34 is PRIVATE and 35 is RGB but not sure what the others are

Syar-tech · 2023-09-04T15:22:37Z

Same issue for me when using the 'yuv' pixelFormat. on 3.0.0 version
0x23 = 35 : it's the YUV_420_888 according to ImageFormat class.
No idea where the 0x1 came from...

mrousavy · 2023-09-05T17:01:01Z

'native' works?

Syar-tech · 2023-09-06T06:58:50Z

Yes, native and rgb both work,
but YUV is mandatory for the frame processors I am using.

mrousavy · 2023-09-06T08:55:57Z

I mean there is a wip PR to use HardwareBuffers instead, see this: https://github.com/mrousavy/react-native-vision-camera/pull/1711/files

This will work with every format and should be much faster, but I benchmarked it and for some reason it is much slower than getPlanes(). That's the only reason it isn't merged yet.

Fabianurrutia · 2023-09-06T18:20:16Z

Same error than Syar-tech, im trying to use ML Kit and need the JPEG format or YUV_420_888, but when using pixelformat="yuv" is giving the error The producer output buffer format 0x1 doesn't match the ImageReader's configured buffer format 0x23.
Trying to use the native and parse the PRIVATE format to JPEG or YUV, but I can't find a way to do the parsing.
add the comment to get an update of this thread

mrousavy · 2023-09-07T15:37:37Z

Just to confirm, are you all using 3.0.0 (the latest release)? And not any rc?

Fabianurrutia · 2023-09-07T16:11:21Z

Yeah @mrousavy the last release, not any rc

aliisajokinen · 2023-09-08T06:24:22Z

Same error here, I use release version 3.0.0

yBihma · 2023-09-08T10:24:22Z

Same, I wrote a test frame processor that only calls frame.getImage()

It gives the following error when pixelFormat = 'rgb':
RGBA override BLOB format buffer should have height == width
And this one if pixelFormat = 'yuv':
Producer output buffer format: 0x1, ImageReader configured format: 0x23

Also pretty much the same thing happens if I use InputImage.fromByteBuffer() from Google ML Kit - it yells at the buffer format being 0x1

Edit:
I tried to specify Camera's format on js side so that videoWidth == videoHeight, but the frame produced from such format is not a perfect square and instead 4:3

Edit2:
I wasn't just calling .frame.getImage(), was also passing it to InputImage.fromImage(), which actually threw the error above

rocket13011 · 2023-09-08T13:10:41Z

same issue on android pixel 3 / 3.0.0 release.
exactly same on @yBihma

Sekiro-kost · 2023-09-08T13:45:11Z

Same issue for me on my android pixel.

I also tried the solution of @moriax , but I'm still stuck

mrousavy · 2023-09-11T08:53:09Z

I think OpenGL only deals in RGB, my ImageReader is configured to take YUV, JPEG or PRIVATE images but OpenGL just always pushes RGB (JPEG). Not sure if OpenGL can also do other formats like YUV or PRIVATE

mrousavy · 2023-09-11T09:24:42Z

I experimented with using ImageWriter instead of the C++ OpenGL pipeline here: #1789

Apparently instead of rendering into OpenGL and converting to RGB, this just passes graphics buffers around. If that is true, then this is more efficient on battery, more performant (better frame rate), and it can do every format (including yuv, rgb and even private).

If anyone wants to test it, please do.

Zuxelus · 2023-09-12T13:48:54Z

I took current main branch and now I have next error.

ImageReader_JNI com.test W  Unable to acquire a buffer item, very likely client tried to acquire more than maxImages buffers
AndroidRuntime com.test E  FATAL EXCEPTION: mrousavy/VisionCamera.video
    Process: com.test, PID: 20905
    java.lang.IllegalStateException: maxImages (3) has already been acquired, call #close before acquiring more.
        at android.media.ImageReader.acquireNextImage(ImageReader.java:661)
        at android.media.ImageReader.acquireLatestImage(ImageReader.java:541)
        at com.mrousavy.camera.core.VideoPipeline.onImageAvailable(VideoPipeline.kt:69)
        at android.media.ImageReader$1.run(ImageReader.java:947)
        at android.os.Handler.handleCallback(Handler.java:958)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loopOnce(Looper.java:205)
        at android.os.Looper.loop(Looper.java:294)
        at android.os.HandlerThread.run(HandlerThread.java:67)

mrousavy · 2023-09-13T10:31:49Z

@Zuxelus ah, of course. That fixes it: #1799

mrousavy · 2023-09-13T10:51:46Z

Actually hmm, I think there's an issue when recording videos with the ImageWriter. Maybe the target Surface is not yet initialized. I'll try to take a look... not sure what's wrong here as it just crashes

W  onFrameAvailable: EOS is sent, ignoring frame
W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
E  FATAL EXCEPTION: main
    Process: com.mrousavy.camera.example, PID: 9320
    java.lang.IllegalStateException
	at android.media.MediaRecorder.stop(Native Method)
	at com.mrousavy.camera.core.RecordingSession.stop(RecordingSession.kt:100)
	at com.mrousavy.camera.core.RecordingSession._init_$lambda$0(RecordingSession.kt:72)
	at com.mrousavy.camera.core.RecordingSession.$r8$lambda$enBPf4dA0r-MTevnd6ChW75ZCDg(Unknown Source:0)
     	at com.mrousavy.camera.core.RecordingSession$$ExternalSyntheticLambda0.onError(Unknown Source:2)                         	    
        at android.media.MediaRecorder$EventHandler.handleMessage(MediaRecorder.java:1615)
 	at android.os.Handler.dispatchMessage(Handler.java:106)
 	at android.os.Looper.loopOnce(Looper.java:205)
 	at android.os.Looper.loop(Looper.java:294)
	at android.app.ActivityThread.main(ActivityThread.java:8177)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:552)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:971)

Zuxelus · 2023-09-13T13:22:48Z

Something is wrong in branch fix/image-writer-queueing. Will check again but it seems that the frameProcessor.call is incorrect.

Error: Regular javascript function 'anonymous' cannot be shared. Try decorating the function with the 'worklet' keyword to allow the javascript function to be used as a worklet.
   at fn (native)
   at anonymous (...\node_modules\react-native-vision-camera\src\hooks\useFrameProcessor.ts:7:20)

mrousavy · 2023-09-30T10:04:52Z

Closing as this is a stale issue - this might have been fixed with the full rewrite in VisionCamera V3 (🥳) - if not, please create a new issue.

If your issue has been fixed, consider sponsoring me on GitHub to say thanks 💖

JFYI; We now use the GPU HardwareBuffer instead of getPlanes :)

jeffasd · 2024-12-04T12:26:25Z

For the Camera you get to pick one of two output formats (NV21 or YV12), so pick YV12. That's your raw YUV data. For screen capture Or Opengl es render the output will always be RGB or RGBA, so you need to pick RGBA_8888 (format 0x1) for your ImageReader, rather than YUV_420_888 (format 0x23). If you need YUV for that, you will have to do the conversion yourself. The ImageReader gives you a series of Plane objects, not a byte[], so you will need to adapt to that.

change code to
ImageReader imageReader = ImageReader.newInstance(width, height, PixelFormat.RGBA_8888, 4);

Zuxelus added the 🐛 bug Something isn't working label Aug 23, 2023

Zuxelus changed the title 🐛 Crashes on getPlanes() for YUV format (V3 RC5)🐛 Aug 23, 2023

Zuxelus changed the title ~~Crashes on getPlanes() for YUV format (V3 RC5)🐛~~ 🐛 Crashes on getPlanes() for YUV format (V3 RC5) Aug 23, 2023

Zuxelus mentioned this issue Sep 9, 2023

🐛 Error when using frameProcessor on iOS (V3 - RN 0.72.4) #1776

Closed

6 tasks

rodgomesc mentioned this issue Sep 28, 2023

🔧 Example app crashed on ios release build #1880

Closed

5 tasks

mrousavy closed this as completed Sep 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Crashes on getPlanes() for YUV format (V3 RC5) #1714

🐛 Crashes on getPlanes() for YUV format (V3 RC5) #1714

Zuxelus commented Aug 23, 2023

moriax commented Aug 25, 2023

robinsoncol commented Aug 26, 2023

moriax commented Aug 26, 2023 •

edited

Loading

robinsoncol commented Aug 26, 2023 •

edited

Loading

Zuxelus commented Aug 26, 2023

Zuxelus commented Aug 27, 2023

mrousavy commented Aug 28, 2023

moriax commented Aug 31, 2023

mrousavy commented Sep 1, 2023

Zuxelus commented Sep 2, 2023

mrousavy commented Sep 4, 2023

Syar-tech commented Sep 4, 2023 •

edited

Loading

mrousavy commented Sep 5, 2023

Syar-tech commented Sep 6, 2023

mrousavy commented Sep 6, 2023

Fabianurrutia commented Sep 6, 2023

mrousavy commented Sep 7, 2023

Fabianurrutia commented Sep 7, 2023

aliisajokinen commented Sep 8, 2023

yBihma commented Sep 8, 2023 •

edited

Loading

rocket13011 commented Sep 8, 2023

Sekiro-kost commented Sep 8, 2023

mrousavy commented Sep 11, 2023

mrousavy commented Sep 11, 2023

Zuxelus commented Sep 12, 2023 •

edited

Loading

mrousavy commented Sep 13, 2023

mrousavy commented Sep 13, 2023

Zuxelus commented Sep 13, 2023

mrousavy commented Sep 30, 2023

jeffasd commented Dec 4, 2024

🐛 Crashes on getPlanes() for YUV format (V3 RC5) #1714

🐛 Crashes on getPlanes() for YUV format (V3 RC5) #1714

Comments

Zuxelus commented Aug 23, 2023

What were you trying to do?

Reproduceable Code

What happened instead?

Relevant log output

Device

VisionCamera Version

Additional information

moriax commented Aug 25, 2023

robinsoncol commented Aug 26, 2023

moriax commented Aug 26, 2023 • edited Loading

robinsoncol commented Aug 26, 2023 • edited Loading

Zuxelus commented Aug 26, 2023

Zuxelus commented Aug 27, 2023

mrousavy commented Aug 28, 2023

moriax commented Aug 31, 2023

mrousavy commented Sep 1, 2023

Zuxelus commented Sep 2, 2023

mrousavy commented Sep 4, 2023

Syar-tech commented Sep 4, 2023 • edited Loading

mrousavy commented Sep 5, 2023

Syar-tech commented Sep 6, 2023

mrousavy commented Sep 6, 2023

Fabianurrutia commented Sep 6, 2023

mrousavy commented Sep 7, 2023

Fabianurrutia commented Sep 7, 2023

aliisajokinen commented Sep 8, 2023

yBihma commented Sep 8, 2023 • edited Loading

rocket13011 commented Sep 8, 2023

Sekiro-kost commented Sep 8, 2023

mrousavy commented Sep 11, 2023

mrousavy commented Sep 11, 2023

Zuxelus commented Sep 12, 2023 • edited Loading

mrousavy commented Sep 13, 2023

mrousavy commented Sep 13, 2023

Zuxelus commented Sep 13, 2023

mrousavy commented Sep 30, 2023

jeffasd commented Dec 4, 2024

moriax commented Aug 26, 2023 •

edited

Loading

robinsoncol commented Aug 26, 2023 •

edited

Loading

Syar-tech commented Sep 4, 2023 •

edited

Loading

yBihma commented Sep 8, 2023 •

edited

Loading

Zuxelus commented Sep 12, 2023 •

edited

Loading