Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 Camera Glitchy on Initial Switch From Front to Back #2525

Closed
4 of 5 tasks
griffinbaker12 opened this issue Feb 7, 2024 · 11 comments · Fixed by #2537
Closed
4 of 5 tasks

🐛 Camera Glitchy on Initial Switch From Front to Back #2525

griffinbaker12 opened this issue Feb 7, 2024 · 11 comments · Fixed by #2537
Labels
🐛 bug Something isn't working

Comments

@griffinbaker12
Copy link

griffinbaker12 commented Feb 7, 2024

What's happening?

This is a continuation of an issue that was closed (somewhat) prematurely, but when attempting to switch between the camera using different zooms and the properly initialized device array I observe:

"It seems that the glitchy behavior from before happens the first time in a given session that I open the camera and attempt to switch from front to back. If I close the camera and re-open within the same session, the switch looks as it does in the video when I switch from front to back the second time in a given session. Is this normal behavior, or can it be improved in some way?

Thank you!"

RN.Vision.Cam.Updated.MP4

Reproduceable Code

useLayoutEffect(() => { if (cameraPos === "front") { zoom.value = withTiming(device?.minZoom ?? 1, { duration: 300 }); } else { if (includeUltraWide) { zoom.value = withTiming(device?.minZoom ?? 1, { duration: 300 }); } else { zoom.value = withTiming(device?.neutralZoom ?? 2, { duration: 300 }); } } }, [cameraPos, includeUltraWide]);

I initialized the device array accordingly as well:

const physicalDevices: PhysicalCameraDeviceType[] = ['wide-angle-camera', 'ultra-wide-angle-camera'];

Relevant log output

...

Camera Device

{
  "name": "Back Camera",
  "hasFlash": true,
  "position": "back",
  "hardwareLevel": "full",
  "supportsLowLightBoost": false,
  "sensorOrientation": "landscape-right",
  "isMultiCam": false,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:0",
  "formats": [],
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "supportsFocus": true,
  "supportsRawCapture": false,
  "hasTorch": true,
  "maxZoom": 123.75,
  "minZoom": 1
}
 LOG  {
  "sensorOrientation": "landscape-right",
  "hasTorch": false,
  "isMultiCam": false,
  "minZoom": 1,
  "id": "com.apple.avfoundation.avcapturedevice.built-in_video:1",
  "supportsFocus": true,
  "supportsRawCapture": false,
  "neutralZoom": 1,
  "physicalDevices": [
    "wide-angle-camera"
  ],
  "hardwareLevel": "full",
  "position": "front",
  "supportsLowLightBoost": false,
  "name": "Front Camera",
  "hasFlash": true,
  "maxZoom": 128.875,
  "formats": []
}

Device

iPhone 15 Pro

VisionCamera Version

3.6.8

Can you reproduce this issue in the VisionCamera Example app?

No, I cannot reproduce the issue in the Example app

Additional information

@griffinbaker12 griffinbaker12 added the 🐛 bug Something isn't working label Feb 7, 2024
@stevengoldberg
Copy link

I think that's what it looks like when the camera is initializing — most apps (like the native iOS camera) provide a static loading view or a blurred view of the last active frame until the camera is ready. Handling this better is on my to-do list for my own app, but I just checked and it looks like onInitialized gets called when the camera is fully ready — have you tried using that to set a loading state?

@mrousavy
Copy link
Owner

mrousavy commented Feb 8, 2024

I have also seen that in ShadowLens.

Could you please add logs though, as without logs I will need to close the issue.

@stevengoldberg
Copy link

Ok, not sure if this is helpful, but: I spent some time on this today and attempted to use the onStarted, onStopped, and onInitialized callbacks to show a BlurView until the camera is ready, similar to the native photos app or instagram. I was able to do this somewhat successfully, and found that it depends on the camera format.

My app allows users to choose from 4 photo quality settings; the first 3 affect the value of qualityPrioritization when calling Camera.takePhoto, and the highest setting changes the format filter to also use the max iso:

const formatFilter = [
        {
            photoAspectRatio: 4 / 3,
        },
        { photoResolution: 'max' },
        { videoResolution: { width: 1024, height: 768 } },
        { photoHdr: true },
    ]
    if (photoQualitySetting === 3) {
        formatFilter.unshift({ iso: 'max' })
    }

In this video (recorded on an iPhone 14 pro), I'm switching to and from the camera screen and flipping the device orientation first in the "quality" setting, and then in the "max" setting (max iso). The former works as expected, but the latter causes the camera view to visibly resize when it first starts up and each time it flips from front to back.

@mrousavy
Copy link
Owner

mrousavy commented Feb 9, 2024

Looks much better already with the BlurView!

I'd still say there is a little jump here, maybe this is a bug in VisionCamera, I'm not sure.

That's what I need the logs for, otherwise I can't help.

@griffinbaker12
Copy link
Author

griffinbaker12 commented Feb 12, 2024

@mrousavy What logs would be helpful for you to see?

Also, @stevengoldberg thank you for posting that video. That is the same behavior that I am observing as well. I would probably take the approach of showing a blurred view of the last frame because you can still see the glitchy behavior underneath, but I appreciate the demonstration. How are you controlling when to show it? I know you mention those 3 callbacks, but a larger example would be awesome.

The issues that @stevengoldberg refer to in his video are the same that I am experiencing as well.

@stevengoldberg
Copy link

Sure @griffinbaker12 — I've found that navigating to a screen with the Camera component on it causes onInitialized and then onStarted to be called the first time it happens, and onStopped is called when navigating away. Navigating back to the camera subsequently just calls onStarted, and switching between front and back cameras just calls onInitialized.

Therefore I have it set up basically like this:

const [isCameraReady, setIsCameraReady] = useState(false)
const [cameraPosition, setCameraPosition] = useState('back')
const isCameraFlipping = useRef(false)

// pass this to a button somewhere
const onFlipCameraPressed = useCallback(() => {
    isCameraFlipping.current = true
    setIsCameraReady(false)
    setCameraPosition((p) => (p === 'back' ? 'front' : 'back'))        
}, [])

return (
    <>
        <Camera
            onStarted={() => setIsCameraReady(true)}
            onStopped={() => setIsCameraReady(false)}
            onInitialized={() => {
                if (isCameraFlipping.current) {
                    setIsCameraReady(true)
                    isCameraFlipping.current = false
                }
             }}
            // etc
        />
        {!isCameraReady && <BlurView />}
    </>
)

@mrousavy here are some of the camera logs from my iPhone 14 pro when I switch from front to back camera, as in the video I posted:

error	16:00:09.754394-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 15
error	16:00:09.754416-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 15 complete (res=0x00000000)
error	16:00:09.764570-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	16:00:09.764658-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	16:00:09.818445-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 15
error	16:00:09.818470-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 15 complete (res=0x00000000)
error	16:00:09.822952-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	16:00:09.823246-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	16:00:09.824237-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 10
error	16:00:09.824260-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 10 complete (res=0x00000000)
error	16:00:09.830796-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10
error	16:00:09.831013-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10 complete (res=0x00000000)
error	16:00:09.833564-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10
error	16:00:09.833839-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10 complete (res=0x00000000)
error	16:00:09.836628-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10
error	16:00:09.836836-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10 complete (res=0x00000000)
error	16:00:09.840286-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	16:00:09.840334-0500	appleh13camerad	Jasper calibration
default	16:00:09.840590-0500	appleh13camerad		Looking for local cache
default	16:00:09.842593-0500	appleh13camerad		Found
error	16:00:09.842618-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	16:00:09.855195-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	16:00:09.855220-0500	appleh13camerad	Jasper calibration
default	16:00:09.855242-0500	appleh13camerad		Looking for local cache
default	16:00:09.855264-0500	appleh13camerad		Found
error	16:00:09.855287-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	16:00:10.325464-0500	appleh13camerad	New connection: pid <private>
error	16:00:10.325488-0500	appleh13camerad	Total number of connections: 2
default	16:00:10.325593-0500	appleh13camerad	[0xa5aa1d080] activating connection: mach=false listener=false peer=true name=com.apple.appleh13camerad.peer.0xa5aa1d080
error	16:00:10.325619-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 103
error	16:00:10.325644-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 103 complete (res=0x00000000)
error	16:00:10.325695-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 104
error	16:00:10.325717-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 104 complete (res=0x00000000)
default	16:00:10.328440-0500	appleh13camerad	[0xa5aa1d080] invalidated after getting a no-senders notification - client is gone
error	16:00:10.328707-0500	appleh13camerad	Client disconnecting (pid <private>)
error	16:00:10.328732-0500	appleh13camerad	Active client pid = 28973
error	16:00:10.329201-0500	appleh13camerad	Removing client: pid <private>
error	16:00:10.329261-0500	appleh13camerad	Total number of connections: 1
default	16:00:10.329315-0500	appleh13camerad	Active client pid <private> deviceID <private>
error	16:00:10.329363-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	16:00:10.329418-0500	appleh13camerad	Jasper calibration
default	16:00:10.329441-0500	appleh13camerad		Looking for local cache
default	16:00:10.329463-0500	appleh13camerad		Found
error	16:00:10.329515-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	16:00:10.400016-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	16:00:10.400048-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	16:00:10.647667-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	16:00:10.647710-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	16:00:10.733912-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	16:00:10.734000-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	16:00:10.995667-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	16:00:10.995717-0500	appleh13camerad	Jasper calibration
default	16:00:10.995851-0500	appleh13camerad		Looking for local cache
default	16:00:10.996088-0500	appleh13camerad		Found
error	16:00:10.996114-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	16:00:11.008822-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	16:00:11.008848-0500	appleh13camerad	Jasper calibration
default	16:00:11.008872-0500	appleh13camerad		Looking for local cache
default	16:00:11.009520-0500	appleh13camerad		Found
error	16:00:11.009553-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	16:00:11.010862-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 9
error	16:00:11.012016-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 9 complete (res=0x00000000)
error	16:00:11.062035-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	16:00:11.062067-0500	appleh13camerad	Jasper calibration
default	16:00:11.062526-0500	appleh13camerad		Looking for local cache
default	16:00:11.062583-0500	appleh13camerad		Found
error	16:00:11.062722-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	16:00:11.064081-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 9
error	16:00:11.064105-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 9 complete (res=0x00000000)
error	16:00:11.172883-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	16:00:11.172908-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	16:00:11.190185-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 14
error	16:00:11.190210-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 14 complete (res=0x00000000)
error	16:00:11.215864-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 14
error	16:00:11.215893-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 14 complete (res=0x00000000)

@griffinbaker12
Copy link
Author

@stevengoldberg makes sense, thx for the help on this.

@mrousavy
Copy link
Owner

Found a solution! 💪😄 #2537

@stevengoldberg
Copy link

I just tested this with 3.9.0 and the resizing is still visible when I use the maxISO format.

Video here: https://imgur.com/a/3ZPeG38

Here are the logs from the camera taken while switching from front to rear:

error	17:26:11.781728-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 15
error	17:26:11.781750-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 15 complete (res=0x00000000)
error	17:26:11.787401-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	17:26:11.787425-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	17:26:11.868330-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 15
error	17:26:11.868397-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 15 complete (res=0x00000000)
error	17:26:11.877711-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	17:26:11.877736-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	17:26:11.886612-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10
error	17:26:11.886763-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10 complete (res=0x00000000)
error	17:26:11.889919-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10
error	17:26:11.889944-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10 complete (res=0x00000000)
error	17:26:11.893184-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10
error	17:26:11.893246-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 10 complete (res=0x00000000)
error	17:26:11.896070-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	17:26:11.896104-0500	appleh13camerad	Jasper calibration
default	17:26:11.896142-0500	appleh13camerad		Looking for local cache
default	17:26:11.896445-0500	appleh13camerad		Found
error	17:26:11.896562-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	17:26:11.907597-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	17:26:11.907622-0500	appleh13camerad	Jasper calibration
default	17:26:11.907644-0500	appleh13camerad		Looking for local cache
default	17:26:11.907881-0500	appleh13camerad		Found
error	17:26:11.908042-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	17:26:12.570200-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	17:26:12.570260-0500	appleh13camerad	Jasper calibration
default	17:26:12.570295-0500	appleh13camerad		Looking for local cache
default	17:26:12.570353-0500	appleh13camerad		Found
error	17:26:12.570410-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	17:26:12.570630-0500	appleh13camerad	New connection: pid <private>
error	17:26:12.570808-0500	appleh13camerad	Total number of connections: 2
default	17:26:12.570843-0500	appleh13camerad	[0xe00d3eb10] activating connection: mach=false listener=false peer=true name=com.apple.appleh13camerad.peer.0xe00d3eb10
error	17:26:12.570874-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 103
error	17:26:12.570904-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 103 complete (res=0x00000000)
error	17:26:12.571954-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 104
error	17:26:12.572072-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 104 complete (res=0x00000000)
default	17:26:12.572201-0500	appleh13camerad	[0xe00d3eb10] invalidated after getting a no-senders notification - client is gone
error	17:26:12.572274-0500	appleh13camerad	Client disconnecting (pid <private>)
error	17:26:12.572352-0500	appleh13camerad	Active client pid = 1008
error	17:26:12.572459-0500	appleh13camerad	Removing client: pid <private>
error	17:26:12.572530-0500	appleh13camerad	Total number of connections: 1
default	17:26:12.572601-0500	appleh13camerad	Active client pid <private> deviceID <private>
error	17:26:12.628129-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	17:26:12.628180-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	17:26:12.874217-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	17:26:12.874309-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	17:26:12.966578-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	17:26:12.966764-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	17:26:13.264549-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	17:26:13.264574-0500	appleh13camerad	Jasper calibration
default	17:26:13.264857-0500	appleh13camerad		Looking for local cache
default	17:26:13.265196-0500	appleh13camerad		Found
error	17:26:13.265530-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	17:26:13.279571-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	17:26:13.279599-0500	appleh13camerad	Jasper calibration
default	17:26:13.279623-0500	appleh13camerad		Looking for local cache
default	17:26:13.279989-0500	appleh13camerad		Found
error	17:26:13.280169-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	17:26:13.281960-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 9
error	17:26:13.282388-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 9 complete (res=0x00000000)
error	17:26:13.330198-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7
default	17:26:13.330248-0500	appleh13camerad	Jasper calibration
default	17:26:13.330482-0500	appleh13camerad		Looking for local cache
default	17:26:13.331202-0500	appleh13camerad		Found
error	17:26:13.331402-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 7 complete (res=0x00000000)
error	17:26:13.332298-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 9
error	17:26:13.332323-0500	appleh13camerad	H13ISPServicesAssistant: getProperty 9 complete (res=0x00000000)
error	17:26:13.446706-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 2
error	17:26:13.446731-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 2 complete (res=0x00000000)
error	17:26:13.463929-0500	appleh13camerad	H13ISPServicesAssistant: setProperty 14
error	17:26:13.463989-0500	appleh13camerad	H10ISPServicesAssistant: setProperty 14 complete (res=0x00000000)

@mrousavy
Copy link
Owner

Those aren't logs from VisionCamera/your app. Can you run your app through Xcode and share the logs in that console please? I need to see if the session is being configured more than once

@stevengoldberg
Copy link

Sorry about that — here are those logs (taken when switching from front to back camera):

VisionCamera.didSetProps(_:): Updating 2 props: [cameraId, format]
VisionCamera.configure(_:): configure { ... }: Waiting for lock...
VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: true, outputsChanged: true, videoStabilizationChanged: true, orientationChanged: true, formatChanged: true, sidePropsChanged: true, torchChanged: true, zoomChanged: true, exposureChanged: true, audioSessionChanged: false)
VisionCamera.configureDevice(configuration:): Configuring Input Device...
VisionCamera.configureDevice(configuration:): Configuring Camera com.apple.avfoundation.avcapturedevice.built-in_video:7...
VisionCamera.configureDevice(configuration:): Successfully configured Input Device!
VisionCamera.configureOutputs(configuration:): Configuring Outputs...
VisionCamera.configureOutputs(configuration:): Adding Photo output...
VisionCamera.configureOutputs(configuration:): Successfully configured all outputs!
VisionCamera.configureFormat(configuration:device:): Configuring Format (4032x3024 | 4032x3024@30.0 (ISO: 57.0..12768.0, Pixel Formats: [VisionCamera.PixelFormat.yuv, VisionCamera.PixelFormat.yuv, VisionCamera.PixelFormat.rgb, VisionCamera.PixelFormat.unknown, VisionCamera.PixelFormat.yuv, VisionCamera.PixelFormat.unknown, VisionCamera.PixelFormat.yuv, VisionCamera.PixelFormat.unknown, VisionCamera.PixelFormat.rgb]))...
VisionCamera.configureFormat(configuration:device:): Already selected active format, no need to configure.
VisionCamera.didSetProps(_:): Updating 2 props: [exposure, zoom]
VisionCamera.configure(_:): configure { ... }: Waiting for lock...
VisionCamera.onSessionInitialized(): Camera initialized!
VisionCamera.configure(_:): configure { ... }: Updating CameraSession Configuration... Difference(inputChanged: false, outputsChanged: false, videoStabilizationChanged: false, orientationChanged: false, formatChanged: false, sidePropsChanged: false, torchChanged: false, zoomChanged: true, exposureChanged: false, audioSessionChanged: false)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants