Boost quality of scan

[folin03]

What is the maximum picture Quality possible? with capturedQuality set to 1 I do have issue reading scanned documents. Quality of a picture taken just with my phone camera is higher that the quality from the scanner. Can I boost the scanner picture quality?

[humphreyja]

Is this on Android?

Couple of things:

1. The capturedQuality setting is a percentage of quality in the encoding of the jpeg from the captured image.
2. On android devices, there is 2 settings for image resolution. The preview resolution (the video feed quality), and the captured resolution. These can be different which can cause some problems. The package looks for a preview size that matches the aspect ratio of your device's screen. Then it tries to find a captured resolution size that matches the aspect ratio of the preview size. This is so that the rectangle that is detected will also line up in the captured image.

What I suspect is that for your device, it doesn't have a matching high resolution capture size, only a lower resolution. This is device specific which causes slightly different results for different devices.

So with that said, there isn't much that can be done about that. There is a potential solution by removing the "ratio matching" part of the captured resolution size, and effectively just getting the highest quality image possible. This can be done by changing this line to just get the largest resolution from getPictureResolutionList(). In theory, this should not effect the ability to crop the image to the detected rectangle because this function in the code responsible for cropping the image will first crop the captured image to the same aspect ratio of the preview size in which the rectangle was detected in. I say "In Theory" because I have not tested this but it was built with that intention.

If you find that is successfully works for you and on a few different devices, please create a PR! This would be a great addition!

[folin03]

You have pointed my the right direction. This is how I have forced the camera use the highest possible resolution.
Camera.Size maxRes = getOptimalResolution(previewRatio, getPictureResolutionList());
if (maxRes != null) {
int max = 0;
int index = 0;
List<Camera.Size> resolutionList = getPictureResolutionList();

    for (int i = 0; i < resolutionList.size(); i++){
      Camera.Size s = resolutionList.get(i);
      int sizeResolution = s.height * s.width;
        if (sizeResolution > max) {
          index = i;
          max = sizeResolution ;
        }
    }
    param.setPictureSize(resolutionList.get(index).width, resolutionList.get(index).height);

// param.setPictureSize(maxRes.width, maxRes.height);
// Log.d(TAG, "max supported picture resolution: " + maxRes.width + "x" + maxRes.height);
}

It did created another issue. With max possible resolution the camera takes too long to take the picture. The Issue is that while taking picture with flash, the flash goes out too soon and the picture is super dark. Do you know how can be extended the time for how long the flash stays on while taking a picture?

[humphreyja]

That's rather interesting. I turn the flash off in the after the image is taken here. So I'm not sure how that could happen. You could maybe try moving that function call to after converting the captured data into an image here.

As for how long the image takes to capture, that's actually why I include the capturedQuality because having that at the highest quality means the image is much larger and takes longer to save to disk (which is the slowest operation of the entire process). So having a higher resolution but a lower capturedQuality can still result in a fairly decent image.

Let me know if that works for you!

[folin03]

I have played with the captureQuality while forcing the camera take the highest resolution picture and it seams not to have any effect on the speed of capturing the picture, interestingly enough the picture seams brighter when taken with flash. All my test since beginning I was performing on Android using Asus Zenphone. Today I dis some test with Samsung S6, this device doesn't have any of issues mentioned above. So I'm going with the result than Asus Zenphone doesn't have a good camera. Perhaps good to mention then forcing the highest camera resolution and camputeQuality is down to 0.1 the picture is still brilliant. I'm thinking about making a function to force the highest resolution on press of a button. But I have never written app in java and not sure about the procedure I should follow. Any recommendation on that?

[humphreyja]

Hmm that's interesting. The first phone I tested this on was an LG G3 (good to know it works on a 6 year old phone!) and noticed the camera preview was really dark. The final photo was fine though so it could be related to the phone's hardware.

It might not be a bad idea to just default to highest resolution?

If you wanted to take a crack at it, I find the best way to develop (and honestly, I'm not sure if there is another decent way to do this) is just to make the changes in your node_modules folder.

So basically, fork this repository. Then make the changes in the node_modules, which will then show up in your app you are developing, then copy the code over to the forked repository.

I've really wanted to add tests and stuff for this but I haven't found a decent sample project showing how to do that.

[folin03]

Thank you for the advice. My friend has tested for me the app on Samsung S10E (Android10). The detection seams to be way off. By his description he did not manage to detect the notebook on the table at all. Do you have any idea? I have been trying to find the rectangle detection algorithm but I didn't find it and if I did it din't recognize it. Can you point in the file where the rectangle is detected so I can try find out how is detected and find out what happened when it was tested on S10E.
Screen_Recording_20200810-133306.zip

[humphreyja]

Whoah that is quite far off. I wonder if this has something to do with the different resolution/ratio sizes. I have some functions to assist with that but like I said, forcing the highest resolution might have some unintended side affects.

Check out detectRectangleInFrame. It takes an image, finds the edges, and finds the rectangle from those edges. It should at least find something close. I don't know why you are getting those results unless that resolution/ratio change you made is causing that issue.

You could check the rotation of the image as well. For some reason, the preview image output might be rotated incorrectly (I have a function at the bottom of that file that is meant to fix that as well).

[folin03]

The whole thing has been done with out forcing the max resolution. I have been using fresh install with no changes in the Java files. We have run some more test and find the potential problem.
this are logs from the phone from the video.

[Wed Aug 12 2020 10:43:25.905] LOG Window dimensions: {"fontScale": 1, "height": 673.3333333333334, "scale": 3, "width": 360}
[Wed Aug 12 2020 10:43:25.907] LOG Preview size: {"height": 0.631578947368421, "marginLeft": 0, "marginTop": 124.03508771929826, "width": 1}

this is what is send to onDeviceSetup:
LOG onDeviceSetup - deviceDetails: {"flashIsAvailable": true, "hasCamera": true, "permissionToUseCamera": true, "previewHeightPercent": 0.631578947368421, "previewWidthPercent": 1}

We have tried and force height to 1 in preview size, which stretched the view over whole screen (see the zip file). Note that the object is more of a square than rectangle.

It reams that the aspect ratio of the S10E is not matching up with preview size returned by the rectangle-scanner.

The same logs from my phone:
[Wed Aug 12 2020 15:47:37.229] LOG Window dimensions: {"fontScale": 1, "height": 640, "scale": 3, "width": 360}
[Wed Aug 12 2020 15:47:37.233] LOG Preview size: {"height": 1, "marginLeft": 0, "marginTop": 0, "width": 1}
LOG onDeviceSetup - deviceDetails: {"flashIsAvailable": true, "hasCamera": true, "permissionToUseCamera": true, "previewHeightPercent": 1, "previewWidthPercent": 1}

Do You have any thoughts?
I have looked into

react-native-rectangle-scanner/android/openCVLibrary310/src/main/java/org/opencv/android/Camera2Renderer.java

Line 275 in 07c0678

protected void setCameraPreviewSize(int width, int height) {

and

react-native-rectangle-scanner/android/src/main/java/com/rectanglescanner/views/CameraDeviceController.java

Line 365 in d36f6fe

public void setDevicePreviewSize(float previewRatio) {

But could not see anything obvious.

Screen_Recording_20200812-141537_TrackerSnap.zip

[humphreyja]

Yeah that would be a result from trying to find a matching screen and preview ratio. The reason I have those previewHeightPercent and previewWidthPercent is to correct the image for the screen's ratio. (I tried to do this entirely on the android side but for some reason I couldn't get the view to change size. So I apologize for this little hack haha).

Take a look at the Quadrilateral.java file. It is responsible for converting the rectangle detection to match the ratio and resolution of the captured image. (crops the captured image to the same ratio, then scales the rectangle to match the resolution).

Otherwise, what I suspect might be happening here is that the height and width percents are styled on the component rather than on a view that includes the detected rectangle component. Here is the code I use in my production app, maybe this will help:

const previewSize = this.getPreviewSize();
let rectangleOverlay = null;
if (!this.state.loadingCamera && !this.state.processingImage) {
  rectangleOverlay = (
    <RectangleOverlay
      detectedRectangle={this.state.detectedRectangle}
      previewRatio={previewSize}
      onDetectedCapture={this.capture}
      allowDetection
    />
  );
}
return (
  <View style={{ backgroundColor: 'rgba(0, 0, 0, 0)', position: 'relative', marginTop: previewSize.marginTop, marginLeft: previewSize.marginLeft, height: `${previewSize.height * 100}%`, width: `${previewSize.width * 100}%` }}>
    <Scanner
      onPictureTaken={this.onPictureTaken}
      onPictureProcessed={this.onPictureProcessed}
      onErrorProcessingImage={this.onErrorProcessingImage}
      enableTorch={this.state.flashEnabled}
      filterId={this.state.filterId}
      ref={this.camera}
      capturedQuality={0.7}
      onRectangleDetected={({ detectedRectangle }) => this.setState({ detectedRectangle })}
      onDeviceSetup={this.onDeviceSetup}
      onTorchChanged={({ enabled }) => this.setState({ flashEnabled: enabled })}
      style={{ flex: 1 }}
   />
   {rectangleOverlay}
   <FlashAnimation overlayFlashOpacity={this.state.overlayFlashOpacity} />
   {this.renderCameraOverlay()}
</View>
);

And the getPreviewSize function for reference

getPreviewSize() {
  const dimensions = Dimensions.get('window');
  const heightMargin = (1 - this.state.device.previewHeightPercent) * dimensions.height / 2;
  const widthMargin = (1 - this.state.device.previewWidthPercent) * dimensions.width / 2;
  if (dimensions.height > dimensions.width) {
    // Portrait
    return {
      height: this.state.device.previewHeightPercent,
      width: this.state.device.previewWidthPercent,
      marginTop: heightMargin,
      marginLeft: widthMargin,
    };
  }

  // Landscape
  return {
    width: this.state.device.previewHeightPercent,
    height: this.state.device.previewWidthPercent,
    marginTop: widthMargin,
    marginLeft: heightMargin,
  };
}

[folin03]

Hi mate. The rectangle is on right spot now, I missed previewRatio={previewSize} in RectangleOverlay. Still having Issues with the preview size. It seams that S10E gives incorrect height of the screen in hardware request. If I use the same dimensions as S10E has, your code work great an I can see that the basic height is different. I'm doing some test and will keep you updated on that one.
for iOS devices I have to use simulator but as you say in the description for simulators, The screen is black and it does not find a camera. Is there a way to force the app that it believes it has camera, for example to take picture of a black screen or something, so I might be able to test what it does with the picture after it has been taken and stored in cache?

[humphreyja]

Hmm interesting. Are the const dimensions = Dimensions.get('window'); incorrect as well? Glad to hear we got part of it working! I almost wonder if I could make the API simpler by using context and having the rectangle overlay be a child.... just a random thought I might experiment with haha.

Unfortunately I don't have a great solution for that iOS problem. This is one instance that the Android simulator really shines. I originally tried to swap out the camera feed for an image but the APIs are all so different that it just didn't work very well. I'm definitely open to a PR on this one as it would make developing for iOS waaaay easier.

[folin03]

yes, the real S10E gives the "height": 673.3333333333334 and emulator with the same dimensions gives "height": 712 and from there everything is affected.