🐛 iOS barcode scanning is very slow (unusable in some cases) #2060

lc-mm · 2023-10-20T18:47:41Z

What's happening?

On iOS the native API's are performing very poorly (all devices that I have tested on). The time to scan a regular barcode is so slow and inaccurate compared to Android that uses MLKit. The QR scanning on iOS is fine, but the other types of barcodes (vertical lines, ie: upc/code/etc) I have tested have been almost unusable, what takes android 2 seconds to scan/verify will take iOS almost 10 times as long sometimes.

Some additional context - when I say scan/"verify" we are waiting to see the same code 4 times before we count that as a successful scan.

The framerate on iOS is good, but the time to scan is not adequate.

We were previously using a fork of this library https://github.com/rodgomesc/vision-camera-code-scanner which also uses MLKit on iOS and it was significantly better and matched the Android implementation/performance.

Could iOS be changed to use MLKit instead of the native API's?

Reproduceable Code

const codeScanner = useCodeScanner({
    codeTypes: [
      'code-128',
      'code-39',
      'code-93',
      'codabar',
      'ean-13',
      'ean-8',
      'itf',
      'upc-e',
      'qr',
      'pdf-417',
      'aztec',
      'data-matrix',
    ],
    onCodeScanned: (codes) => {
// ...
},
  });

Relevant log output

NA

Camera Device

NA

Device

iPhone 13, iPad Mini

VisionCamera Version

3.5.1

Can you reproduce this issue in the VisionCamera Example app?

Yes, I can reproduce the same issue in the Example app here

Additional information

I am using Expo
I have enabled Frame Processors (react-native-worklets-core)
I have read the Troubleshooting Guide
I agree to follow this project's Code of Conduct
I searched for similar issues in this repository and found none.

celik75 · 2023-10-20T19:48:55Z

I noticed this too, after some scans it looks like its needs more and more time to scan.
On android it works fine.
for ios fix i use temporary external barcode scanner "@mgcrea/vision-camera-barcode-scanner" but only works with react-native-vision-camera 3.4.1

lc-mm · 2023-10-21T00:13:55Z

I just want to make it clear that the issue I am describing is not a performance issue as I scan more and more codes. The issue is that on iOS the detection of non-qr codes is very poor. 10x slower than android

lc-mm · 2023-10-21T06:49:10Z

I just tested @mgcrea/vision-camera-barcode-scanner and can confirm that the implementation there that uses the Vision APIs is much better at detecting barcodes than AVFoundation implementation

Is using either the Vision APIs or MLKit on iOS an option? In the current state it feels as if the iOS implementation is only good for QR codes

chinieer · 2023-10-23T08:36:37Z

I just tested @mgcrea/vision-camera-barcode-scanner and can confirm that the implementation there that uses the Vision APIs is much better at detecting barcodes than AVFoundation implementation

Is using either the Vision APIs or MLKit on iOS an option? In the current state it feels as if the iOS implementation is only good for QR codes

MLKit on iOS is much better than iOS innner scan speed

mrousavy · 2023-10-24T10:56:03Z

Hey!

Great to hear y'all are using my library and it's working fine so far.

On iOS the native API's are performing very poorly

I am using the recommended AVCaptureMetadataOutput output, this is what Apple recommends for quick and easy Barcode/QR-code detection.

The QR scanning on iOS is fine, ...

nice!

..., but the other types of barcodes (vertical lines, ie: upc/code/etc) I have tested have been almost unusable

This honestly sounds like an iOS 17/Apple bug. Did you try to run this on the latest iOS 17 beta, or on iOS 16 or older?

Could iOS be changed to use MLKit instead of the native API's?

No, because on iOS this is a separate output that uses a private hardware channel. Using MLKit on iOS would be a significant amount of extra work, and would fall under the Video Pipeline, which means a lot of nested code/ifs that merge to the same codepath. Right now it's separated very clean.

Again, I believe this is an Apple bug since QR code scanning is fine. Maybe they'll fix it in an upcoming update (or already fixed it), but try searching for apple bug reports to verify.

Is using either the Vision APIs or MLKit on iOS an option?

Again, not really as this would fall under the Video Pipeline (and requires a significant amount of extra work and a lot of code complicating the setup of the camera pipeline).

celik75 · 2023-10-24T12:41:37Z

Maybe this can help for fixing the slow IOS barcode scan.
https://stackoverflow.com/a/55995043

mrousavy · 2023-10-24T12:46:07Z

@celik75 as per the Apple docs;

mrousavy · 2023-10-24T12:47:29Z

You can choose a custom format, this will also affect the CodeScanner. If your format has a very low video resolution, so will the codescanner. If it has a very high video resolution, so will the codescanner. See Camera Formats

Maybe try passing enableBufferCompression={false} to see what happens?

iliapnmrv · 2023-10-24T17:01:56Z

You can choose a custom format, this will also affect the CodeScanner. If your format has a very low video resolution, so will the codescanner. If it has a very high video resolution, so will the codescanner. See Camera Formats

Maybe try passing enableBufferCompression={false} to see what happens?

This does not resolve issue for me. Tried passing max video resolution and enableBufferCompression={false}

lc-mm · 2023-10-24T18:50:10Z

The behaviour has been the same on iOS 16 and 17, very poor performance when scanning anything that isn't a QR code.

zzz08900 · 2023-10-24T19:26:33Z

Maybe try place the barcode close the center of camera view and put them in parallel with camera view's top border(don't rotate bar codes) and see if things gets better?

Or you can build a MLKit based frame processor to handle the task. Guess I'll be building one when I upgrade to v3 (not happening anything soon though due to, reasons)

seanpcoyle · 2023-10-26T05:38:51Z

Just wanted to chime in here and say that I too am seeing this issue. Non-QR barcodes take a considerable amount of time to be recognized (in the neighborhood of 8-10 seconds). I've tinkered with enableBufferCompression and format as @mrousavy recommended, but with no luck thus far.

And to add something to the discussion, rather than just throwing in a useless "+1":

Did you try to run this on the latest iOS 17 beta, or on iOS 16 or older?

I've tried iOS 16.4, iOS 17.0, and iOS 17.1 (released today).

seanpcoyle · 2023-10-26T18:19:45Z

An update - I'm seeing barcodes that fail be recognized when scanned horizontally are consistently recognized when the barcode is turned vertically.

The phone orientation is vertical during all of this testing, for what it's worth.

mrousavy · 2023-10-31T11:46:37Z

I mean you guys can test to create a new bare native iOS app and use the Barcode Scanner there (AVCaptureMetadataOutput) and see if that is also slow. If it is, there's nothing I can do about that since it's an Apple issue.

zjkuang · 2023-10-31T16:40:33Z

An update - I'm seeing barcodes that fail be recognized when scanned horizontally are consistently recognized when the barcode is turned vertically.

The phone orientation is vertical during all of this testing, for what it's worth.

I have the same experience. When I use vision-camera-v2 in our app to scan a barcode, it takes less than 1/100 second. But with vision-camera-v3 (3.6.4) on the same device I have to rotate the phone to fit the barcode vertically. (I guess thus it makes the barcode image larger.)

vision-camera-v2:

v2.mov

vision-camera-v3:

v3.mov

@mrousavy

lc-mm · 2023-10-31T16:40:58Z

I just tested @mgcrea/vision-camera-barcode-scanner and can confirm that the implementation there that uses the Vision APIs is much better at detecting barcodes than AVFoundation implementation

Is using either the Vision APIs or MLKit on iOS an option? In the current state it feels as if the iOS implementation is only good for QR codes

@mrousavy when I started this thread I said that it was an apple issue and presented alternatives. Is this ever something you would change so that the performance of this feature is acceptable for barcodes on iOS? Currently it is not for anything other than QR codes.

mrousavy · 2023-11-08T10:08:12Z

@lc-mm to be honest - no.

As mentioned previously, the Code Scanner API is available under an AVCaptureMetadataOutput channel on iOS. This API is really straight forward, that's all the code needed to scan QR codes on iOS:

react-native-vision-camera/package/ios/Core/CameraSession+Configuration.swift

Lines 130 to 149 in 8f986a4

    
           // 1. Add 
        
           guard captureSession.canAddOutput(codeScannerOutput) else { 
        
             throw CameraError.codeScanner(.notCompatibleWithOutputs) 
        
           } 
        
           captureSession.addOutput(codeScannerOutput) 
        
           // 2. Configure 
        
           let options = codeScanner.options 
        
           codeScannerOutput.setMetadataObjectsDelegate(self, queue: CameraQueues.codeScannerQueue) 
        
           try codeScanner.options.codeTypes.forEach { type in 
        
             // CodeScanner::availableMetadataObjectTypes depends on the connection to the 
        
             // AVCaptureSession, so this list is only available after we add the output to the session. 
        
             if !codeScannerOutput.availableMetadataObjectTypes.contains(type) { 
        
               throw CameraError.codeScanner(.codeTypeNotSupported(codeType: type.descriptor)) 
        
             } 
        
           } 
        
           codeScannerOutput.metadataObjectTypes = options.codeTypes 
        
           if let rectOfInterest = options.regionOfInterest { 
        
             codeScannerOutput.rectOfInterest = rectOfInterest 
        
           }

If I wanted to use MLKit for this, I would have to:

Add a Video Channel output:

react-native-vision-camera/package/ios/Core/CameraSession+Configuration.swift

Lines 107 to 120 in 8f986a4

    
           // 1. Add 
        
           let videoOutput = AVCaptureVideoDataOutput() 
        
           guard captureSession.canAddOutput(videoOutput) else { 
        
             throw CameraError.parameter(.unsupportedOutput(outputDescriptor: "video-output")) 
        
           } 
        
           captureSession.addOutput(videoOutput) 
        
           // 2. Configure 
        
           videoOutput.setSampleBufferDelegate(self, queue: CameraQueues.videoQueue) 
        
           videoOutput.alwaysDiscardsLateVideoFrames = true 
        
           let pixelFormatType = try video.getPixelFormat(for: videoOutput) 
        
           videoOutput.videoSettings = [ 
        
             String(kCVPixelBufferPixelFormatTypeKey): pixelFormatType, 
        
           ]

Since we now have a third variable depending on the Video Channel (video + frameProcessor + codeScanner), I need to smartly decide which resolution I am going to pick and if there even needs to be a video channel.
I need to add the MLKit dependency (I think that's 2.4 MB added to your app/VisionCamera, no matter if you use QR codes or not!)

I need to initialize the MLKit Barcode Detector:

let format = .all
let barcodeOptions = BarcodeScannerOptions(formats: format)
let barcodeScanner = BarcodeScanner.barcodeScanner(options: barcodeOptions)

I need to call it in the Video Capture Output delegate, if a code scanner is added:

barcodeScanner.process(visionImage) { features, error in
  guard error == nil, let features = features, !features.isEmpty else {
    // Error handling
    return
  }
  // Recognized barcodes
}

I need to call the callback there then

Here's the downsides of the video channel approach compared to the AVCaptureMetadataOutput approach:

The separation of concern is no longer here - it's all nested in a video channel.
The video channel is running with high resolution frames:
If you don't use a format, it will default to the highest resolution (4k frames) - which is going to be very slow in terms of MLKit Code Scanning performance
If you use a format, it will use the format's width and height for the MLKit Code Scanner. This means, you can't really do high quality video recordings/frame processing and codescanning in one - you have to either decide for code scanning (low resolution video) or video recordings (very slow code scanning) since again, it's all in one queue.
We have an added dependency in pods
It might go out of sync and we need to upgrade it
It might have additional security vulnerabilities
It increases the app size by 2.4MB - no matter if you use QR codes or not!
We all know that Google pods (e.g. Firebase) sometimes just won't build. That sucks, especially if you're not even using the QR code scanner
We have a bit more complex code - it's all in the video channel instead of simply separating it to a separate AVCaptureMetadataOutput channel.
We cannot benefit from platform-level optimizations. So far, I am not sure what optimizations the native iOS platform exactly does, but I believe that their AVCaptureMetadataOutput API is very battery/energy efficient. It might use pure GPU algorithms, benefit from raw YUV buffers, etc. In theory, it should be fast and lightweight.
Yes, I know it is apparently not fast for Barcodes. But this is a bug in apple's code which will be fixed.
We can separate the video output (video + frameProcessor) from the QR code scanner (codeScanner), meaning it can detect Barcodes without a hickup while you're recording a video or processing Frames. It is running on a separate output channel. That's simply not possible with the video approach

So in short; Yes I know it absolutely sucks that the code scanner is only fast on QR codes for iOS, this is most definitely a bug since I don't think I'm doing anything terribly wrong here. I can try to spend some more time to investigate this in a native app (no React Native) thanks to @ticketscloud for sponsoring me/this! ❤️

ecaii · 2023-12-21T15:02:51Z

I can't seem to be able to scan a code 128 (or anything barcode) anymore on iOS.

I'm on 3.6.4, any workaround ? Even if I fit the barcode vertically it doesn't work.

trikstudio · 2023-12-21T16:24:09Z

@ecaii We had this issue as well and we've decided to use a frameprocessor plugin: https://github.com/mgcrea/vision-camera-barcode-scanner for the iOS part which is still faster & more reliable

ecaii · 2023-12-22T13:53:51Z

**trikstudio ** commented Dec 21, 2023 •

Indeed it's a good solution ! Thank you, it's indeed fast, whether it be on Android or iOS.

ErAmanDhiman · 2023-12-26T06:06:55Z

An update - I'm seeing barcodes that fail be recognized when scanned horizontally are consistently recognized when the barcode is turned vertically.
The phone orientation is vertical during all of this testing, for what it's worth.

I have the same experience. When I use vision-camera-v2 in our app to scan a barcode, it takes less than 1/100 second. But with vision-camera-v3 (3.6.4) on the same device I have to rotate the phone to fit the barcode vertically. (I guess thus it makes the barcode image larger.)

vision-camera-v2:

v2.mov

vision-camera-v3:

v3.mov
@mrousavy

Have you found any solution , because it is still not working in latest updates @zjkuang

mrousavy · 2024-01-15T13:25:31Z

can you try rotating the Camera to see if that changes anything? Maybe the orientation on the scanner is wrong (e.g. landscape left by default)

Regina-v · 2024-04-26T09:12:53Z

For future visitors of this issue: I had the same issue with very slow barcode scanning in iOS with vision-camera v3. I have upgraded yesterday to v4.0.1 and this solved the issue for me, scanning is now really fast. Thank you @mrousavy great job!

ErAmanDhiman · 2024-04-26T09:14:33Z

Great If it works ☑️

…

On Fri, Apr 26, 2024 at 2:43 PM Regina ***@***.***> wrote: For future visitors of this issue: I had the same issue with very slow barcode scanning in iOS with vision-camera v3. I have upgraded yesterday to v4.0.1 and this solved the issue for me, scanning is now really fast. Thank you @mrousavy <https://github.com/mrousavy> great job! — Reply to this email directly, view it on GitHub <#2060 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUNDHHYWFVV7TN3XIJ4GKADY7ILCZAVCNFSM6AAAAAA6JN64H2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZYHE3TKMBWGI> . You are receiving this because you commented.Message ID: ***@***.***>

lc-mm added the 🐛 bug Something isn't working label Oct 20, 2023

iliapnmrv mentioned this issue Oct 23, 2023

🐛 iOS itf barcode not working #2064

Closed

5 tasks

mrousavy added the hacktoberfest Hacktoberfest 2023 label Oct 24, 2023

mrousavy closed this as completed Jan 15, 2024

klabusta mentioned this issue Jan 25, 2024

🐛 App crash using QR scanner (useCodeScanner) #2341

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 iOS barcode scanning is very slow (unusable in some cases) #2060

🐛 iOS barcode scanning is very slow (unusable in some cases) #2060

lc-mm commented Oct 20, 2023

celik75 commented Oct 20, 2023

lc-mm commented Oct 21, 2023

lc-mm commented Oct 21, 2023

chinieer commented Oct 23, 2023

mrousavy commented Oct 24, 2023

celik75 commented Oct 24, 2023

mrousavy commented Oct 24, 2023

mrousavy commented Oct 24, 2023 •

edited

Loading

iliapnmrv commented Oct 24, 2023

lc-mm commented Oct 24, 2023

zzz08900 commented Oct 24, 2023

seanpcoyle commented Oct 26, 2023

seanpcoyle commented Oct 26, 2023

mrousavy commented Oct 31, 2023

zjkuang commented Oct 31, 2023 •

edited

Loading

lc-mm commented Oct 31, 2023

mrousavy commented Nov 8, 2023 •

edited

Loading

ecaii commented Dec 21, 2023

trikstudio commented Dec 21, 2023 •

edited

Loading

ecaii commented Dec 22, 2023

ErAmanDhiman commented Dec 26, 2023

vision-camera-v2:

vision-camera-v3:

mrousavy commented Jan 15, 2024

Regina-v commented Apr 26, 2024

ErAmanDhiman commented Apr 26, 2024 via email

🐛 iOS barcode scanning is very slow (unusable in some cases) #2060

🐛 iOS barcode scanning is very slow (unusable in some cases) #2060

Comments

lc-mm commented Oct 20, 2023

What's happening?

Reproduceable Code

Relevant log output

Camera Device

Device

VisionCamera Version

Can you reproduce this issue in the VisionCamera Example app?

Additional information

celik75 commented Oct 20, 2023

lc-mm commented Oct 21, 2023

lc-mm commented Oct 21, 2023

chinieer commented Oct 23, 2023

mrousavy commented Oct 24, 2023

celik75 commented Oct 24, 2023

mrousavy commented Oct 24, 2023

mrousavy commented Oct 24, 2023 • edited Loading

iliapnmrv commented Oct 24, 2023

lc-mm commented Oct 24, 2023

zzz08900 commented Oct 24, 2023

seanpcoyle commented Oct 26, 2023

seanpcoyle commented Oct 26, 2023

mrousavy commented Oct 31, 2023

zjkuang commented Oct 31, 2023 • edited Loading

vision-camera-v2:

vision-camera-v3:

lc-mm commented Oct 31, 2023

mrousavy commented Nov 8, 2023 • edited Loading

ecaii commented Dec 21, 2023

trikstudio commented Dec 21, 2023 • edited Loading

ecaii commented Dec 22, 2023

ErAmanDhiman commented Dec 26, 2023

vision-camera-v2:

vision-camera-v3:

mrousavy commented Jan 15, 2024

Regina-v commented Apr 26, 2024

ErAmanDhiman commented Apr 26, 2024 via email

mrousavy commented Oct 24, 2023 •

edited

Loading

zjkuang commented Oct 31, 2023 •

edited

Loading

mrousavy commented Nov 8, 2023 •

edited

Loading

trikstudio commented Dec 21, 2023 •

edited

Loading