obs-ffmpeg: Encoder timeout hybridmp4 dataloss

[Naumanbo]

Description

This pull request refines the error-handling logic in signal_failure() within obs-ffmpeg-mux.c. The updates ensure that after an encoder timeout or other muxing-related failures occur, the function now properly resets internal state variables and concludes the data capture session. This change addresses scenarios where subsequent recordings could fail silently after an encoder timeout, improving the overall reliability of hybrid MP4 muxing.

Motivation and Context

Previously, when an encoder timed out, certain state flags and resources within signal_failure() were not fully reset. As a result, starting a new recording session after a failure could lead to silent hangs and zero-byte files, forcing users to restart OBS. By fully cleaning up and ending the data capture process, these issues are prevented, ensuring that subsequent recordings can start without residual state causing problems.
In response to following issue: #11329

How Has This Been Tested?

Test PC:
OS: Windows 10 Pro 22H2
CPU: AMD Ryzen 7 5800x3d
GPU: NVIDIA GeForce RTX 4070 Super
RAM: 32 GB 3600 MHZ

Steps taken:

1. Started a recording using the hybrid MP4 container and highly demanding encoder settings to induce a timeout.
2. Observed the “Recording error” dialog appearing due to the timeout.
3. Initiated a second recording session without restarting OBS.
4. Verified that the second session now functions correctly, records frames as intended, and cleanly stops.
5. Ran automated tests using ctest to ensure no regression or loss of functionality in the program.

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[derrod]

This can't fix the linked issue because it's a change in completely unrelated code.

[derrod]

This doesn't fix anythiing.