figure out the correct dimensions masks for codecs

[totaam]

The lazy approach we use is to clamp the dimensions using:

width_mask=0xFFFE
height_mask=0xFFFE

But some csc modules and video encoders may well be able to handle odd dimensions correctly, especially when handling RGB input directly.

[totaam]

The fix above was another lazy fix
Instead, I should have figured out if it was the jpeg encoder or the libyuv csc module that was causing the corrupted image I was seeing when downscaling or subsampling was used with odd input image dimensions.
Modifying the csc modules is the easy part, ie for swscale:

-            self.out_height[i] = (int) (dst_height * dst.height_mult[i])
-            self.out_stride[i] = roundup((int) (dst_width * dst.width_mult[i]), 16)
+            self.out_height[i] = ceil(dst_height * dst.height_mult[i])
+            self.out_stride[i] = roundup(ceil(dst_width * dst.width_mult[i]), 16)

[totaam]

As per the commit above, x264 can handle odd heights in all cases and odd widths with non-subsampled input (ie: BGRX or YUV444 - not tested YUV422P yet, probably won't work as it is sumbsampling 2x1).

Still to test:

• nvenc (swap GPU)
• ffmpeg encoder with and without libva (might be safer to just leave it as it is?)
• x265

[totaam]

This may have caused the crashes in #3706 (comment)