Frame timeout

[Evgeniy0805]

Hello!
I have one problem with using poll_frame(): at first everything worked as I needed, but at one point (always in different ways) an exception occurred:
RuntimeError: frame timeout. Verify timeout exceeds exposure time. If applicable, check external trigger source
Tell me, what could be the cause of this problem? I will be grateful for any help.

[lwalder]

Hello, in order to investigate this problem we may need to more details:

• What was the exposure time?
• Was any timeout specified in the poll_frame function?
• Was the camera running in internal or external trigger?
• Please provide the camera model and interface used (if applicable)
• What operating system was used?
• Is the same issue reproducible in the PVCamTest application that is installed with PVCAM?
  Thank you

[Evgeniy0805]

• This problem may occur with different exposure time. It can also happen a few hours after you start using the camera.
• No, I used the default value (WAIT_FOREVER).
• In internal trigger.
• Prime BSI Express
• Windows 10
• No, I haven't seen this error in PVCamTest.
  Example of a method that throws an error in the poll_frame function:

    def show_live_thread(self):
        try:
            self.cam.start_live(exp_time=self.exp_time)
            self.live_counter = 0
            while self.isLive:
                try:
                    frame, fps, frame_count = self.cam.poll_frame()
                    frame['pixel_data'] = cv2.resize(frame['pixel_data'], self.dim, interpolation = cv2.INTER_AREA)
                    frame['pixel_data'] = cv2.convertScaleAbs(frame['pixel_data'], alpha=255/frame['pixel_data'].max())

                    self.update_image(frame['pixel_data']) 
                    if self.rec:
                        if self.live_counter % 2 != 0:
                            writer = cv2.VideoWriter(f"{ui.camera_path_to_save.text()}/{ui.file_name.text()}.avi", cv2.VideoWriter_fourcc(*'XVID'), ui.recording_framerate.value(), self.dim, isColor = False)
                            self.live_counter = self.live_counter + 1
                        writer.write(frame['pixel_data'].astype('uint8'))
                    if self.live_counter % 2 != 0 and writer.isOpened():
                        writer.release()
                        self.live_counter = self.live_counter + 1
                except Error:
                    print(Error)
            self.cam.finish()
            threading.exit()
        except AttributeError:
            print('Camera is not connected')

[svierne]

Hello, we're having a similar issue since last week. :(

When acquiring frames with start_seq() followed by poll_frame(), the camera gets stuck after some time, resulting in a frame timeout error. Calling check_frame_status() returns EXPOSURE_IN_PROGRESS even after a few minutes of waiting.

• Exposure Time: 3 ms
• ROI Size: 256x256
• Frame Count: 3320, error occurs randomly after 3315-3319 frames have been polled successfully, or does not occur at all
• Frame Timeout: 10 s
• Trigger Mode: Trigger first
• Camera Model: Prime 95B PCIE
• Operating System: Windows 10
• PVCamTest: Cannot reproduce so far

Is my understanding correct that in our situation the camera suddenly stops delivering data to the host? What are possible causes for this?

Thanks a lot in advance for your support.

[lwalder]

Thank you for your comments, we will be looking into this issue and try to reproduce this on our side.

@svierne Was there any change in the system, PVCAM version upgrade or anything that could possibly trigger this issue? (your comments sound like it might have been working fine for you in the past). In your case you use the start_seq call, assuming in a loop. I'd suspect it could be some threading, synchronization issue in the PyVCAM code, since it works fine in PVCamTest (which is a native application). You can try to comment out the 'pvc.finish_seq(self.__handle)' from the finish routine, because this call is not absolutely necessary with recent PVCAM versions.

Also please state what PVCAM runtime version you both use, the current one is PVCAM 3.9.10.5.

[svierne]

@lwalder Yes, it has been working in the past. No changes to the system except for code changes in our software, and of course Windows system updates. If you need the exact calls being made to PyVCAM, I can mock the python module in our software and record the calls for you. PVCAM runtime version is 3.9.10, we will most likely upgrade to the latest version in the following days.

Another idea I came up with is to patch pvc_check_frame_status in pvcmodule.cpp to log bytes_arrived to the console and see if this value changes. Would that be helpful or can we just assume that it's not changing?

Unfortunately I don't completely understand your suggestion of commenting out the pvc.finish_seq(self.__handle) from the finish routine. Is it called somewhere internally? Because otherwise in our software it would only be called after the loop containing poll_frame.

Thank you again for your quick response last week. :-)

[lwalder]

@svierne printing the bytes_arrived and status may help diagnose the issue. You should see the number increasing up to the size of the sequence buffer. My suspicion is that there might be some synchronization issue. If you see the hang somewhere by the end of the long sequence it would suggest that either the python or native part is not properly counting the incoming frames and it skips some (e.g. you acquire 3000 frames, camera actually acquires 3000 frames but python "misses a poll" and counts 2998 only for example). This would especially happen with fast frame rates (short exposures) or when CPU is loaded during the acquisition. On the other hand, if the pl_exp_check_status properly completes and fills the entire buffer, the issue will most likely be up the stream (native PVCAM, USB/PCIe subsystem, or simply the triggers are coming too fast - if triggering externally).

The pvc.finish_seq is not relevant then, I was under impression that you acquire single snaps repeatedly, not longer sequences.

We have an engineer looking into the issue but so far we were unable to reproduce either of those issues. We'll keep digging.

[ZhengqiHao313]

We tested both sequence mode and Live mode capture to reproduce this two issue.
So far. can not reproduced yet.
With CPU loading with our loading tool to 50%. The PC CPU has one thread always running occupy CPU resource to 50% at background. Tested both USB3 and PCIE interface on Prime 95B and Prime BSI camera.
1: Live capture 3 hours to simulate #1 issue.
2: Sequence capture 256x256 ROI for 3300 frames (3ms exposure). loop of " start--capture-->stop after 3300 frames done". tested 40 times of loop.

There is no any TIME_OUT detected. All success finished.

Here we post the segment of python code for testing sequence 3300 frames:

num_frames = 3300
cnt = 0
try:
    cam.start_seq(exp_time=3, num_frames=num_frames)
    while cnt < num_frames:
        frame, fps, frame_count = cam.poll_frame()

        low = np.amin(frame['pixel_data'])
        high = np.amax(frame['pixel_data'])
        average = np.average(frame['pixel_data'])

        print('Min:{}\tMax:{}\tAverage:{:.0f}\tFrame Rate: {:.1f}\tFrame Count: {:.0f}\n'.format(low, high, average, fps, 
                        frame_count))
        cnt += 1
except Error:
    print(Error)        

cam.finish()
cam.close()
pvc.uninit_pvcam()

[svierne]

Oh, that's unfortunate to hear. :( Thank you very much for investigating. I'll report back when I have something more substantial to share. I suspect that I will also have to do several test runs. Depending on our lab schedule, that might take a while. Currently, our workaround is to call µManager, but if we can get PyVCAM working, that would be much less overhead.

[ZhengqiHao313]

We tested external trigger "Trigger first" mode on PCIE 95B camera, 3330 sequence frames of Roi 256x256
It always working fine and no timeout

Exposure Time: 3 ms
ROI Size: 256x256
Frame Count: 3330,
Trigger Mode: Trigger first
Camera Model: Prime 95B PCIE
Operating System: Windows 10