mac-videotoolbox: Add Spatial AQ option (macOS 15)

[dsaedtler]

Description

Adds support for the new spatial aq property introduced in macOS 15: https://developer.apple.com/documentation/videotoolbox/kvtcompressionpropertykey_spatialadaptiveqplevel?language=objc

Motivation and Context

Disabling spatial aq seems to improve video quality of screen content (such as games) by a decent bit.

How Has This Been Tested?

M1 Pro with FFmpeg patch applied for direct frame-accurate comparison of AOM video game compression test corpus.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[dsaedtler]

Apparently we need Xcdoe 16 for this to work, which is getting removed from the macos-14 images soon (actions/runner-images#10703) so we'd have to update to macos-15. Let's see what breaks?

[Lain-B]

I'm not a macOS specialist so might want @PatTheMav to give this a look just as a double-check, but seems trivial/simple enough so I threw an approval on it.

[PatTheMav]

If this does indeed add a requirement to use Xcode 16 (which is fine in my book), then the lower version boundaries should be changed here as well:

obs-studio/cmake/macos/compilerconfig.cmake

Lines 24 to 25 in 52a0486

	set(_obs_macos_minimum_sdk 14.2) # Keep in sync with Xcode
	set(_obs_macos_minimum_xcode 15.1) # Keep in sync with SDK

[dsaedtler]

Updated the PR with the following changes:

• Checkbox was replaced by combobox with "Auto", "Enabled", and "Disabled" settings
• Auto will disable spatial AQ for CBR/ABR, but keep it enabled for CRF

---

The choice to disable it by default for ABR/CBR was made based on an extensive analysis involving VMAF scoring and subjective checks on a large number of test clips¹. In all cases the encoder produced significantly better results if spatial AQ was disabled.
In addition, having spatial AQ enabled would also result in bitrate overshoot in complex scenes, whereas keeping it disabled would result in the encoder sticking closer to the target bitrate.

One of the most noticeable issues when spatial AQ is enabled is that text often gets mangled:

---

The choice to keep it enabled for CRF was made in order to not change anything for users using target quality modes, as having spatial AQ enabled bumps the output bitrate by 50-100%, which provides a corresponding quality increase. If we were to disable it by default for CRF we would also have to bump the target quality values a bit (7-8 points) to make up for it.

Footnotes

1. The test clips included both game footage as well as live action/"IRL" video filmed on a variety of devices. ↩

[MysticalOS]

I wonder if this is what made apples H264 utterly wreck wow content all the time (so much text). i basically am forced to use software encoder cause apples mangles it to hell and back. nice find either way :o