Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

video/out/vulkan: add MoltenVK context #7857

Closed
wants to merge 1 commit into from
Closed

video/out/vulkan: add MoltenVK context #7857

wants to merge 1 commit into from

Conversation

tmm1
Copy link
Contributor

@tmm1 tmm1 commented Jun 24, 2020

Similar to #7482, but without any of the macOS specific bits. Instead, the underlying CAMetalLayer* can be passed in via --wid=

I tested this briefly on iOS and it worked.

cc @Akemi @qiudaomao

@tmm1
video/out/vulkan: add MoltenVK context
3ada048 
Signed-off-by: Aman Gupta <aman@tmm1.net>
@tmm1 tmm1 mentioned this pull request Jun 24, 2020
Merged
haasn
haasn reviewed Jun 26, 2020
View reviewed changes
DOCS/man/options.rst
@@ -3175,6 +3175,10 @@ Window
``--hwdec=mediacodec`` for direct rendering using MediaCodec, or with
``--vo=gpu --gpu-context=android`` (with or without ``--hwdec=mediacodec-copy``).

If compiled with MoltenVK on iOS/tvOS/macOS, the ID is interpreted as
``CAMetalLayer *``. Pass it as a value cast to ``intptr_t``. Use with
``--vo=gpu --gpu-api=vulkan --gpu-context=moltenvk``
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifying --gpu-api=vulkan here is redundant.

@qiudaomao
Copy link

@tmm1 I want try your PR on iOS/tvOS, could you share me commands how to cross compiling libplacebo..

@tmm1
Copy link
Contributor Author

tmm1 commented Jun 27, 2020 •
edited
Loading

I used the idea from this comment: haasn/libplacebo#44 (comment)

In MoltenVK once fetch deps and build, there will be libglslang.a which libplacebo can use. I had to disable other dependencies since everything needed is already included: tmm1/libplacebo@5450560

For linking vulkan, use a similar pc file like #7482

@qiudaomao
Copy link

I tried on macOS first and meet black screen, was I missing some configurations?

    self->wrapper.wantsLayer = YES;
    self->wrapper.layer = [CAMetalLayer layer];
    int64_t wid = (intptr_t) self->wrapper.layer;
    mpv_set_option(mpv, "wid", MPV_FORMAT_INT64, &wid);
    mpv_set_option_string(mpv, "vo", "gpu");
    mpv_set_option_string(mpv, "gpu-api", "vulkan");
    mpv_set_option_string(mpv, "gpu-context", "moltenvk");
    mpv_initialize(mpv);
    mpv_command(mpv, {"loadfile", "some videos", NULL});
MPVMoltenVK[52574:1936915] Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2)
detail log [cplayer] info: (+) Video --vid=1 (*) (h264 1920x1080 25.000fps) event: log-message [cplayer] info: (+) Audio --aid=1 (*) (aac 2ch 48000Hz) event: log-message [mvk-info] MoltenVK version 1.0.44. Vulkan version 1.0.141. The following 53 Vulkan extensions are supported: VK_KHR_16bit_storage v1 VK_KHR_8bit_storage v1 VK_KHR_bind_memory2 v1 VK_KHR_dedicated_allocation v3 VK_KHR_descriptor_update_template v1 VK_KHR_device_group v4 VK_KHR_device_group_creation v1 VK_KHR_external_memory v1 VK_KHR_external_memory_capabilities v1 VK_KHR_get_memory_requirements2 v1 VK_KHR_get_physical_device_properties2 v2 VK_KHR_get_surface_capabilities2 v1 VK_KHR_image_format_list v1 VK_KHR_maintenance1 v2 VK_KHR_maintenance2 v1 VK_KHR_maintenance3 v1 VK_KHR_push_descriptor v2 VK_KHR_relaxed_block_layout v1 VK_KHR_sampler_mirror_clamp_to_edge v3 VK_KHR_sampler_ycbcr_conversion v14 VK_KHR_shader_draw_parameters v1 VK_KHR_shader_float16_int8 v1 VK_KHR_storage_buffer_storage_class v1 VK_KHR_surface v25 VK_KHR_swapchain v70 VK_KHR_swapchain_mutable_format v1 VK_KHR_uniform_buffer_standard_layout v1 VK_KHR_variable_pointers v1 VK_EXT_debug_marker v4 VK_EXT_debug_report v9 VK_EXT_debug_utils v2 VK_EXT_fragment_shader_interlock v1 VK_EXT_hdr_metadata v2 VK_EXT_host_query_reset v1 VK_EXT_inline_uniform_block v1 VK_EXT_memory_budget v1 VK_EXT_metal_surface v1 VK_EXT_robustness2 v1 VK_EXT_scalar_block_layout v1 VK_EXT_shader_stencil_export v1 VK_EXT_shader_viewport_index_layer v1 VK_EXT_swapchain_colorspace v4 VK_EXT_texel_buffer_alignment v1 VK_EXT_vertex_attribute_divisor v3 VK_EXTX_portability_subset v1 VK_MVK_macos_surface v2 VK_MVK_moltenvk v26 VK_AMD_gpu_shader_half_float v2 VK_AMD_negative_viewport_height v1 VK_AMD_shader_trinary_minmax v1 VK_INTEL_shader_integer_functions2 v1 VK_GOOGLE_display_timing v1 VK_NV_glsl_shader v1 [mvk-info] GPU device: model: Intel(R) Iris(TM) Plus Graphics 655 type: Integrated vendorID: 0x8086 deviceID: 0x3ea5 pipelineCacheUUID: 0000273C-0300-07D2-A644-84F62B55D2DE supports the following Metal Versions, GPU's and Feature Sets: Metal Shading Language 2.2 GPU Family Mac 2 GPU Family Mac 1 GPU Family Common 3 GPU Family Common 2 GPU Family Common 1 macOS GPU Family 2 v1 macOS GPU Family 1 v4 macOS GPU Family 1 v3 macOS GPU Family 1 v2 macOS GPU Family 1 v1 [mvk-info] Created VkInstance with the following 5 Vulkan extensions enabled: VK_KHR_external_memory_capabilities v1 VK_KHR_get_physical_device_properties2 v2 VK_KHR_surface v25 VK_EXT_metal_surface v1 VK_EXT_swapchain_colorspace v4 [mvk-info] Using MTLFence for Vulkan semaphores. [mvk-info] Created VkDevice to run on GPU Intel(R) Iris(TM) Plus Graphics 655 with the following 5 Vulkan extensions enabled: VK_KHR_external_memory v1 VK_KHR_push_descriptor v2 VK_KHR_swapchain v70 VK_EXT_hdr_metadata v2 VK_EXT_host_query_reset v1 event: audio-reconfig event: audio-reconfig event: file-loaded 2020-06-27 15:53:55.262357+0800 MPVMoltenVK[52574:1936913] [plugin] AddInstanceForFactory: No factory registered for id F8BB1C28-BAE8-11D6-9C31-00039315CD46 [cplayer] info: AO: [coreaudio] 48000Hz stereo 2ch floatp event: log-message event: audio-reconfig event: video-reconfig [cplayer] info: VO: [gpu] 1920x1080 yuv420p event: log-message event: video-reconfig [mvk-info] Created 3 swapchain images with initial size (480, 320). event: playback-restart 2020-06-27 15:54:41.474301+0800 MPVMoltenVK[52574:1936915] Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2) 2020-06-27 15:54:41.474493+0800 MPVMoltenVK[52574:1936915] Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2) 2020-06-27 15:54:41.475849+0800 MPVMoltenVK[52574:1936915] Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2) 2020-06-27 15:54:41.475960+0800 MPVMoltenVK[52574:1936915] Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2) 2020-06-27 15:54:41.476092+0800 MPVMoltenVK[52574:1936915] Execution of the command buffer was aborted due to an error during execution. Caused GPU Timeout Error (IOAF code 2)

@Akemi
Copy link
Member

Akemi commented Jun 27, 2020

i am not sure how to handle this. on the old cocoa backend we embedded an NSView in our own windowing code and the user didn't have to do anything besides providing that NSView. here the user has to do their own windowing and provide a CAMetalLayer where we render into. it's a lot less complex for us, but more for the user. i am not sure what the user expects here (since different behaviour with same option) and if we should change that? though it seems on other platform it also behaves different. the big pro though, is that it is (Apple) platform independent.

lets say we want to add a moltenvk/vulkan macOS backend that supports wid embedding like the old cocoa backend, eg we provide the window stuff and embed the provided CAMetalLayer into it. in that case we need two separate moltenvk/vulkan (macOS) contexts, since we can't distinguish between those two usages solely with the wid option?

@tmm1
Copy link
Contributor Author

tmm1 commented Jun 27, 2020

I tried on macOS first and meet black screen, was I missing some configurations?

Was it with HDR video or normal? For HDR, you need to add target-prim=bt.2020 and target-trc=pq. Otherwise it tries to tone map and the shader is too slow and is aborted.

@tmm1
Copy link
Contributor Author

tmm1 commented Jun 27, 2020

info: VO: [gpu] 1920x1080 yuv420p

Looks non-HDR. Strange it doesn't work.

On macOS do you have to use makeBackingLayer?

@tmm1
Copy link
Contributor Author

tmm1 commented Jun 27, 2020

@Akemi I guess if it's important to keep the same type for --wid, then we could keep passing a view and the context could retrieve the layer from the view. But you still have NSView vs UIView so I'm not sure how much better it is? We can also check types inside the context and handle different types of inputs.

I have also been working on another vo, which accepts --wid=(AVSampleBufferDisplayLayer *). Personally I feel it's not important that the type of wid be the same always, and it seems reasonable that the object can change depending on the vo, gpu and context options used.

@Akemi
Copy link
Member

Akemi commented Jun 27, 2020

yes, i agree on the type for wid point.

qiudaomao
qiudaomao reviewed Dec 13, 2020
View reviewed changes
video/out/vulkan/context_moltenvk.m
{
struct priv *p = ctx->priv;
CGSize s = p->layer.drawableSize;
ra_vk_ctx_resize(ctx, s.height, s.height);
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

//should pass width here not height
ra_vk_ctx_resize(ctx, s.width, s.height);

@skrew skrew mentioned this pull request Jun 11, 2021
Open
@alexiscn
Copy link

alexiscn commented Mar 29, 2023 •
edited
Loading

Hi, @tmm1 . I want to use your PR in iOS 16 for HDR rendering with CAMetalLayer new property wantsExtendedDynamicRangeContent and edrMetadata . But I could not get the correct HDR color.

The test video link comes from lg-new-york-hdr-uhd-4k-demo.ts .

with mediainfo, here is the video information :

Video_Format_List : HEVC
Video_Format_WithHint_List : HEVC
Codecs Video : HEVC
Format : MPEG-TS
Format/Extensions usually used : ts m2t m2s m4t m4s tmf ts tp trp ty
Overall bit rate mode : CBR
Frame rate : 25.000 FPS

Inform : 3840*2160 (16:9), at 25.000 FPS, HEVC (Main 10@L5@High), HDR10
HDR format : SMPTE ST 2086, HDR10 compatible
Width : 3840
Height : 2160
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 10
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant

Here is how CAMetalLayer created. (I also tried pixelFormat=.bgr10a2Unorm and colorspace=itur_2020)

metalLayer.frame = bounds
metalLayer.contentsScale = UIScreen.main.nativeScale
metalLayer.wantsExtendedDynamicRangeContent = true
metalLayer.pixelFormat = .rgba16Float
metalLayer.isOpaque = true
metalLayer.colorspace = CGColorSpace(name: CGColorSpace.extendedLinearITUR_2020)
metalLayer.edrMetadata = .hdr10(minLuminance: 0.5, maxLuminance: 1000, opticalOutputScale: 100)
layer.addSublayer(metalLayer)

The mpv side, built with --enable-libplacebo --enable-vulkan --enable-moltenvk --disable-shaderc.

wid=&renderView.metalLayer
vo=gpu
gpu-api=vulkan
gpu-context=moltenvk
hwdec=auto

with target-prim=bt.2020 and target-trc=pq, the picture washed out. Like following
IMG_4047

without target-prim=bt.2020 and target-trc=pq. The picture was just like OpenGLES renders.

IMG_4042

The correct color should be like this:
IMG_4043

The full log

mpv event (unhandled): idle
mpv log: [cplayer] v: mpv 0.35.0-UNKNOWN Copyright © 2000-2023 mpv/MPlayer/mplayer2 projects

mpv log: [cplayer] v: built on Tue Mar 28 11:22:37 2023

mpv log: [cplayer] v: libplacebo version: v5.264.0-88-g97d008b6

mpv log: [cplayer] v: FFmpeg version: 6.0

mpv log: [cplayer] v: FFmpeg library versions:

mpv log: [cplayer] v: libavutil 58.2.100

mpv log: [cplayer] v: libavcodec 60.3.100

mpv log: [cplayer] v: libavformat 60.3.100

mpv log: [cplayer] v: libswscale 7.1.100

mpv log: [cplayer] v: libavfilter 9.3.100

mpv log: [cplayer] v: libswresample 4.10.100

mpv log: [cplayer] v:

mpv log: [cplayer] v: Configuration: ./waf configure --prefix=/Users/alex/github/MetalLayer/BuildMPV/scratch-ios/arm64 --exec-prefix=/Users/alex/github/MetalLayer/BuildMPV/scratch-ios/arm64 --disable-cplayer --disable-lcms2 --disable-lua --disable-rubberband --disable-zimg --enable-libmpv-static --enable-ios-gl --enable-gl --enable-avfoundation --enable-libplacebo --enable-vulkan --enable-moltenvk --disable-shaderc --disable-javascript --disable-libbluray --disable-vapoursynth --enable-uchardet --enable-lgpl

mpv log: [cplayer] v: List of enabled features: asm audiounit av-channel-layout avfoundation bsd-fstatfs build-date cplugins debug-build ffmpeg fork gl glob glob-posix iconv ios ios-gl jpegxl lgpl libass libavdevice libdl libm libmpv-static libplacebo moltenvk optimize osx-thread-name plain-gl posix posix-or-mingw posix-shm pthreads spirv-cross spirv-cross-shared stdatomic swift uchardet vector videotoolbox-hwaccel vulkan zlib

mpv log: [cplayer] v: Set property: target-trc="pq" -> 1

mpv log: [cplayer] v: Set property: target-prim="bt.2020" -> 1

mpv log: [file] v: Opening /var/mobile/Containers/Data/Application/279EDC26-94E4-4CEA-AC42-C585598CABAD/Documents/HDR.ts

mpv log: [demux] v: Trying demuxers for level=normal.

mpv log: [lavf] v: Found 'mpegts' at score=50 size=2048.

mpv log: [ffmpeg/demuxer] warn: mpegts: start time for stream 1 is not set in estimate_timings_from_pts

mpv log: [file] v: stream level seek from 5570560 to 469607120

mpv log: [file] v: stream level seek from 469857120 to 469357120

mpv log: [file] v: stream level seek from 469857120 to 468857120

mpv log: [file] v: stream level seek from 469857120 to 467857120

mpv log: [file] v: stream level seek from 469857120 to 465857120

mpv log: [file] v: stream level seek from 469857120 to 461857120

mpv log: [file] v: stream level seek from 469857120 to 453857120

mpv log: [ffmpeg/demuxer] warn: mpegts: stream 1 : no TS found at start of file, duration not set

mpv log: [file] v: stream level seek from 462245728 to 0

mpv log: [ffmpeg/demuxer] warn: mpegts: Could not find codec parameters for stream 1 (Audio: aac ([15][0][0][0] / 0x000F), 0 channels): unspecified sample format

mpv log: [ffmpeg/demuxer] warn: Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options

mpv log: [lavf] v: avformat_find_stream_info() finished after 0 bytes.

mpv log: [demux] v: Detected file format: mpegts (libavformat)

mpv log: [cplayer] v: Opening done: /var/mobile/Containers/Data/Application/279EDC26-94E4-4CEA-AC42-C585598CABAD/Documents/HDR.ts

mpv log: [find_files] v: Loading external files in /var/mobile/Containers/Data/Application/279EDC26-94E4-4CEA-AC42-C585598CABAD/Documents/

mpv log: [lavf] v: select track 0

mpv log: [lavf] v: select track 1

mpv log: [cplayer] info: (+) Video --vid=1 (hevc 3840x2160 25.000fps)

mpv log: [cplayer] info: (+) Audio --aid=1 (aac)

mpv log: [vo/gpu/vulkan] v: Initializing GPU context 'moltenvk'

mpv log: [vo/gpu/libplacebo] v: Initialized libplacebo v5.264.0-88-g97d008b6 (API v266)

mpv log: [vo/gpu/libplacebo] v: Creating vulkan instance with extensions:

mpv log: [vo/gpu/libplacebo] v: VK_KHR_get_physical_device_properties2

mpv log: [vo/gpu/libplacebo] v: VK_KHR_surface

mpv log: [vo/gpu/libplacebo] v: VK_EXT_swapchain_colorspace

mpv log: [vo/gpu/libplacebo] v: VK_KHR_external_memory_capabilities

mpv log: [vo/gpu/libplacebo] v: VK_KHR_external_semaphore_capabilities

mpv log: [vo/gpu/libplacebo] v: VK_KHR_get_surface_capabilities2

mpv log: [vo/gpu/libplacebo] v: VK_KHR_surface

mpv log: [vo/gpu/libplacebo] v: VK_EXT_metal_surface

[mvk-info] MoltenVK version 1.2.3, supporting Vulkan version 1.2.239.
The following 88 Vulkan extensions are supported:
VK_KHR_16bit_storage v1
VK_KHR_8bit_storage v1
VK_KHR_bind_memory2 v1
VK_KHR_buffer_device_address v1
VK_KHR_copy_commands2 v1
VK_KHR_create_renderpass2 v1
VK_KHR_dedicated_allocation v3
VK_KHR_depth_stencil_resolve v1
VK_KHR_descriptor_update_template v1
VK_KHR_device_group v4
VK_KHR_device_group_creation v1
VK_KHR_driver_properties v1
VK_KHR_dynamic_rendering v1
VK_KHR_external_fence v1
VK_KHR_external_fence_capabilities v1
VK_KHR_external_memory v1
VK_KHR_external_memory_capabilities v1
VK_KHR_external_semaphore v1
VK_KHR_external_semaphore_capabilities v1
VK_KHR_fragment_shader_barycentric v1
VK_KHR_get_memory_requirements2 v1
VK_KHR_get_physical_device_properties2 v2
VK_KHR_get_surface_capabilities2 v1
VK_KHR_imageless_framebuffer v1
VK_KHR_image_format_list v1
VK_KHR_maintenance1 v2
VK_KHR_maintenance2 v1
VK_KHR_maintenance3 v1
VK_KHR_multiview v1
VK_KHR_portability_subset v1
VK_KHR_push_descriptor v2
VK_KHR_relaxed_block_layout v1
VK_KHR_sampler_mirror_clamp_to_edge v3
VK_KHR_sampler_ycbcr_conversion v14
VK_KHR_separate_depth_stencil_layouts v1
VK_KHR_shader_draw_parameters v1
VK_KHR_shader_float_controls v4
VK_KHR_shader_float16_int8 v1
VK_KHR_shader_subgroup_extended_types v1
VK_KHR_spirv_1_4 v1
VK_KHR_storage_buffer_storage_class v1
VK_KHR_surface v25
VK_KHR_swapchain v70
VK_KHR_swapchain_mutable_format v1
VK_KHR_timeline_semaphore v2
VK_KHR_uniform_buffer_standard_layout v1
VK_KHR_variable_pointers v1
VK_EXT_buffer_device_address v2
VK_EXT_debug_marker v4
VK_EXT_debug_report v10
VK_EXT_debug_utils v2
VK_EXT_descriptor_indexing v2
VK_EXT_external_memory_host v1
VK_EXT_fragment_shader_interlock v1
VK_EXT_host_query_reset v1
VK_EXT_image_robustness v1
VK_EXT_inline_uniform_block v1
VK_EXT_memory_budget v1
VK_EXT_metal_objects v1
VK_EXT_metal_surface v1
VK_EXT_pipeline_creation_cache_control v3
VK_EXT_post_depth_coverage v1
VK_EXT_private_data v1
VK_EXT_robustness2 v1
VK_EXT_sample_locations v1
VK_EXT_scalar_block_layout v1
VK_EXT_separate_stencil_usage v1
VK_EXT_shader_atomic_float v1
VK_EXT_shader_stencil_export v1
VK_EXT_shader_viewport_index_layer v1
VK_EXT_subgroup_size_control v2
VK_EXT_surface_maintenance1 v1
VK_EXT_swapchain_colorspace v4
VK_EXT_swapchain_maintenance1 v1
VK_EXT_texel_buffer_alignment v1
VK_EXT_texture_compression_astc_hdr v1
VK_EXT_vertex_attribute_divisor v3
VK_AMD_gpu_shader_half_float v2
VK_AMD_negative_viewport_height v1
VK_AMD_shader_image_load_store_lod v1
VK_AMD_shader_trinary_minmax v1
VK_IMG_format_pvrtc v1
VK_INTEL_shader_integer_functions2 v1
VK_GOOGLE_display_timing v1
VK_MVK_ios_surface v3
VK_MVK_moltenvk v37
VK_NV_fragment_shader_barycentric v1
VK_NV_glsl_shader v1
[mvk-info] GPU device:
model: Apple A14 GPU
type: Integrated
vendorID: 0x106b
deviceID: 0x100403ef
pipelineCacheUUID: DD5FF2A9-1004-03EF-0000-000000000000
supports the following Metal Versions, GPU's and Feature Sets:
Metal Shading Language 3.0
GPU Family Apple 7
GPU Family Apple 6
GPU Family Apple 5
GPU Family Apple 4
GPU Family Apple 3
GPU Family Apple 2
GPU Family Apple 1
GPU Family Common 3
GPU Family Common 2
GPU Family Common 1
iOS GPU Family 5 v1
iOS GPU Family 4 v2
iOS GPU Family 4 v1
iOS GPU Family 3 v4
iOS GPU Family 3 v3
iOS GPU Family 3 v2
iOS GPU Family 3 v1
iOS GPU Family 2 v5
iOS GPU Family 2 v4
iOS GPU Family 2 v3
iOS GPU Family 2 v2
iOS GPU Family 2 v1
iOS GPU Family 1 v5
iOS GPU Family 1 v4
iOS GPU Family 1 v3
iOS GPU Family 1 v2
iOS GPU Family 1 v1
[mvk-info] Created VkInstance for Vulkan version 1.2.239, as requested by app, with the following 7 Vulkan extensions enabled:
VK_KHR_external_memory_capabilities v1
VK_KHR_external_semaphore_capabilities v1
VK_KHR_get_physical_device_properties2 v2
VK_KHR_get_surface_capabilities2 v1
VK_KHR_surface v25
VK_EXT_metal_surface v1
VK_EXT_swapchain_colorspace v4
mpv log: [vo/gpu/libplacebo] v: Probing for vulkan devices:

mpv log: [vo/gpu/libplacebo] v: GPU 0: Apple A14 GPU (integrated)

mpv log: [vo/gpu/libplacebo] v: uuid: 00:00:10:6B:10:04:03:EF:00:00:00:01:00:00:05:65

mpv log: [vo/gpu/libplacebo] v: Vulkan device properties:

mpv log: [vo/gpu/libplacebo] v: Device Name: Apple A14 GPU

mpv log: [vo/gpu/libplacebo] v: Device ID: 106b:100403ef

mpv log: [vo/gpu/libplacebo] v: Device UUID: 00:00:10:6B:10:04:03:EF:00:00:00:01:00:00:05:65

mpv log: [vo/gpu/libplacebo] v: Driver version: 27db

mpv log: [vo/gpu/libplacebo] v: API version: 1.2.239

[mvk-info] Vulkan semaphores using MTLEvent.
mpv log: [vo/gpu/libplacebo] v: Creating vulkan device with extensions:

mpv log: [vo/gpu/libplacebo] v: VK_KHR_swapchain

mpv log: [vo/gpu/libplacebo] v: VK_KHR_swapchain

mpv log: [vo/gpu/libplacebo] v: VK_KHR_push_descriptor

mpv log: [vo/gpu/libplacebo] v: VK_EXT_external_memory_host

mpv log: [vo/gpu/libplacebo] v: VK_KHR_portability_subset

mpv log: [vo/gpu/libplacebo] v: VK_EXT_metal_objects

[mvk-info] Created VkDevice to run on GPU Apple A14 GPU with the following 5 Vulkan extensions enabled:
VK_KHR_portability_subset v1
VK_KHR_push_descriptor v2
VK_KHR_swapchain v70
VK_EXT_external_memory_host v1
VK_EXT_metal_objects v1
mpv log: [vo/gpu/libplacebo] v: Memory heaps supported by device:

mpv log: [vo/gpu/libplacebo] v: 0: flags 0x1 size 5702M

mpv log: [vo/gpu/libplacebo] v: Memory summary: 0 used 0 res 0 alloc, efficiency 100.00%, utilization 100.00%

mpv log: [vo/gpu/libplacebo] v: glslang version: 12.0.0

mpv log: [vo/gpu/libplacebo] v: Initialized SPIR-V compiler 'glslang'

mpv log: [vo/gpu/libplacebo] v: GPU information:

mpv log: [vo/gpu/libplacebo] v: GLSL version: 450 (vulkan)

mpv log: [vo/gpu/libplacebo] v: max_shmem_size: 32768

mpv log: [vo/gpu/libplacebo] v: max_group_threads: 1024

mpv log: [vo/gpu/libplacebo] v: max_group_size[0]: 1024

mpv log: [vo/gpu/libplacebo] v: max_group_size[1]: 1024

mpv log: [vo/gpu/libplacebo] v: max_group_size[2]: 1024

mpv log: [vo/gpu/libplacebo] v: subgroup_size: 32

mpv log: [vo/gpu/libplacebo] v: min_gather_offset: -8

mpv log: [vo/gpu/libplacebo] v: max_gather_offset: 7

mpv log: [vo/gpu/libplacebo] v: Limits:

mpv log: [vo/gpu/libplacebo] v: thread_safe: 1

mpv log: [vo/gpu/libplacebo] v: callbacks: 1

mpv log: [vo/gpu/libplacebo] v: max_buf_size: 5979881472

mpv log: [vo/gpu/libplacebo] v: max_ubo_size: 1494958080

mpv log: [vo/gpu/libplacebo] v: max_ssbo_size: 1494958080

mpv log: [vo/gpu/libplacebo] v: max_vbo_size: 5979881472

mpv log: [vo/gpu/libplacebo] v: max_mapped_size: 5979881472

mpv log: [vo/gpu/libplacebo] v: max_buffer_texels: 67108864

mpv log: [vo/gpu/libplacebo] v: align_host_ptr: 16384

mpv log: [vo/gpu/libplacebo] v: host_cached: 1

mpv log: [vo/gpu/libplacebo] v: max_tex_1d_dim: 16384

mpv log: [vo/gpu/libplacebo] v: max_tex_2d_dim: 16384

mpv log: [vo/gpu/libplacebo] v: max_tex_3d_dim: 2048

mpv log: [vo/gpu/libplacebo] v: blittable_1d_3d: 1

mpv log: [vo/gpu/libplacebo] v: buf_transfer: 1

mpv log: [vo/gpu/libplacebo] v: align_tex_xfer_pitch: 1

mpv log: [vo/gpu/libplacebo] v: align_tex_xfer_offset: 16

mpv log: [vo/gpu/libplacebo] v: max_variable_comps: 0

mpv log: [vo/gpu/libplacebo] v: max_constants: 18446744073709551615

mpv log: [vo/gpu/libplacebo] v: max_pushc_size: 4096

mpv log: [vo/gpu/libplacebo] v: align_vertex_stride: 4

mpv log: [vo/gpu/libplacebo] v: max_dispatch[0]: 1073741824

mpv log: [vo/gpu/libplacebo] v: max_dispatch[1]: 1073741824

mpv log: [vo/gpu/libplacebo] v: max_dispatch[2]: 1073741824

mpv log: [vo/gpu/libplacebo] v: fragment_queues: 1

mpv log: [vo/gpu/libplacebo] v: compute_queues: 1

mpv log: [vo/gpu/libplacebo] v: External API interop:

mpv log: [vo/gpu/libplacebo] v: UUID: 00:00:10:6B:10:04:03:EF:00:00:00:01:00:00:05:65

mpv log: [vo/gpu/libplacebo] v: PCI: 0000:00:00:0

mpv log: [vo/gpu/libplacebo] v: buf export caps: 0x0

mpv log: [vo/gpu/libplacebo] v: buf import caps: 0x10

mpv log: [vo/gpu/libplacebo] v: tex export caps: 0x0

mpv log: [vo/gpu/libplacebo] v: tex import caps: 0x70

mpv log: [vo/gpu/libplacebo] v: sync export caps: 0x0

mpv log: [vo/gpu/libplacebo] v: sync import caps: 0x0

mpv log: [vo/gpu/libplacebo] v: Available surface configurations:

mpv log: [vo/gpu/libplacebo] v: 0: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_SRGB_NONLINEAR_KHR

mpv log: [vo/gpu/libplacebo] v: 1: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_SRGB_NONLINEAR_KHR

mpv log: [vo/gpu/libplacebo] v: 2: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_SRGB_NONLINEAR_KHR

mpv log: [vo/gpu/libplacebo] v: 3: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_DISPLAY_P3_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 4: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_DISPLAY_P3_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 5: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_DISPLAY_P3_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 6: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_DCI_P3_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 7: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_DCI_P3_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 8: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_DCI_P3_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 9: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_BT709_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 10: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_BT709_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 11: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_BT709_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 12: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_ADOBERGB_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 13: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_ADOBERGB_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 14: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_ADOBERGB_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 15: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_PASS_THROUGH_EXT

mpv log: [vo/gpu/libplacebo] v: 16: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_PASS_THROUGH_EXT

mpv log: [vo/gpu/libplacebo] v: 17: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_PASS_THROUGH_EXT

mpv log: [vo/gpu/libplacebo] v: 18: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 19: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 20: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_EXTENDED_SRGB_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 21: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_EXTENDED_SRGB_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 22: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_EXTENDED_SRGB_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 23: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_EXTENDED_SRGB_NONLINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 24: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_DISPLAY_P3_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 25: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_DISPLAY_P3_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 26: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_DISPLAY_P3_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 27: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_BT2020_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 28: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_BT2020_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 29: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_BT2020_LINEAR_EXT

mpv log: [vo/gpu/libplacebo] v: 30: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_HDR10_HLG_EXT

mpv log: [vo/gpu/libplacebo] v: 31: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_HDR10_HLG_EXT

mpv log: [vo/gpu/libplacebo] v: 32: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_HDR10_HLG_EXT

mpv log: [vo/gpu/libplacebo] v: 33: VK_FORMAT_B8G8R8A8_UNORM VK_COLOR_SPACE_HDR10_ST2084_EXT

mpv log: [vo/gpu/libplacebo] v: 34: VK_FORMAT_B8G8R8A8_SRGB VK_COLOR_SPACE_HDR10_ST2084_EXT

mpv log: [vo/gpu/libplacebo] v: 35: VK_FORMAT_R16G16B16A16_SFLOAT VK_COLOR_SPACE_HDR10_ST2084_EXT

mpv log: [vo/gpu/libplacebo] v: Picked surface configuration 15: VK_FORMAT_B8G8R8A8_UNORM + VK_COLOR_SPACE_PASS_THROUGH_EXT

mpv log: [vo/gpu] v: Testing FBO format rgba16f

mpv log: [vo/gpu] v: Using FBO format rgba16f.

mpv log: [vo/gpu] v: No advanced processing required. Enabling dumb mode.

mpv log: [vd] v: Container reported FPS: 25.000000

mpv log: [vd] v: Codec list:

mpv log: [vd] v: hevc - HEVC (High Efficiency Video Coding)

mpv log: [vd] v: Opening decoder hevc

mpv log: [vd] v: Looking at hwdec hevc-videotoolbox...

mpv log: [vo/gpu] v: Loading hwdec drivers for format: 'videotoolbox'

mpv log: [vo/gpu] v: Loading hwdec driver 'videotoolbox'

mpv log: [vo/gpu] v: Loading failed.

mpv log: [vd] v: Could not create device.

mpv log: [vd] v: Looking at hwdec hevc-videotoolbox-copy...

mpv log: [vd] v: Trying hardware decoding via hevc-videotoolbox-copy.

mpv log: [vd] v: Selected codec: hevc (HEVC (High Efficiency Video Coding))

mpv log: [vf] v: User filter list:

mpv log: [vf] v: (empty)

mpv log: [ad] v: Codec list:

mpv log: [ad] v: aac - AAC (Advanced Audio Coding)

mpv log: [ad] v: aac_fixed (aac) - AAC (Advanced Audio Coding)

mpv log: [ad] v: aac_at (aac) - aac (AudioToolbox)

mpv log: [ad] v: Opening decoder aac

mpv log: [ad] v: Requesting 1 threads for decoding.

/var/mobile/Containers/Data/Application/279EDC26-94E4-4CEA-AC42-C585598CABAD/Documents/HDR.ts
mpv log: [ad] v: Selected codec: aac (AAC (Advanced Audio Coding))

mpv log: [af] v: User filter list:

mpv log: [af] v: (empty)

mpv log: [cplayer] v: Starting playback...

mpv log: [cplayer] v: Set property: pause=true -> 1

mpv log: [cplayer] v: Set property: pause=false -> 1

mpv log: [vd] v: Pixel formats supported by decoder: videotoolbox_vld yuv420p10le

mpv log: [vd] v: Codec profile: Main 10 (0x2)

mpv log: [vd] v: Requesting pixfmt 'videotoolbox_vld' from decoder.

mpv log: [vd] info: Using hardware decoding (videotoolbox-copy).

mpv log: [vd] v: Decoder format: 3840x2160 p010 bt.2020-ncl/bt.2020/pq/limited/auto SP=5.911330 CL=mpeg2/4/h264

mpv log: [vf] v: [in] 3840x2160 p010 bt.2020-ncl/bt.2020/pq/limited/display SP=5.911330 CL=mpeg2/4/h264

mpv log: [vf] v: [userdeint] 3840x2160 p010 bt.2020-ncl/bt.2020/pq/limited/display SP=5.911330 CL=mpeg2/4/h264

mpv log: [vf] v: [userdeint] (disabled)

mpv log: [vf] v: [autorotate] 3840x2160 p010 bt.2020-ncl/bt.2020/pq/limited/display SP=5.911330 CL=mpeg2/4/h264

mpv log: [vf] v: [autorotate] (disabled)

mpv log: [vf] v: [convert] 3840x2160 p010 bt.2020-ncl/bt.2020/pq/limited/display SP=5.911330 CL=mpeg2/4/h264

mpv log: [vf] v: [convert] (disabled)

mpv log: [vf] v: [out] 3840x2160 p010 bt.2020-ncl/bt.2020/pq/limited/display SP=5.911330 CL=mpeg2/4/h264

mpv log: [af] v: [in] 48000Hz stereo 2ch floatp

mpv log: [af] v: [userspeed] 48000Hz stereo 2ch floatp

mpv log: [af] v: [userspeed] (disabled)

mpv log: [af] v: [convert] 48000Hz stereo 2ch floatp

mpv log: [ao] v: Trying audio driver 'audiounit'

mpv log: [ao/audiounit] v: requested format: 48000 Hz, stereo channels, floatp

mpv log: [ao/audiounit] v: max channels: 2, requested: 2

mpv log: [ao/audiounit] v: setting audio output format...

mpv log: [ao/audiounit] v: AU channel layout tag: 0 (0)

mpv log: [ao/audiounit] v: channel map: 0: 1

mpv log: [ao/audiounit] v: channel map: 1: 2

mpv log: [ao/audiounit] v: using stereo output

mpv log: [ao/audiounit] v: using soft-buffer of 9600 samples.

mpv log: [cplayer] info: AO: [audiounit] 48000Hz stereo 2ch floatp

mpv log: [cplayer] v: AO: Description: AudioUnit (iOS)

mpv log: [af] v: [convert] (disabled)

mpv log: [af] v: [out] 48000Hz stereo 2ch floatp

mpv log: [cplayer] info: VO: [gpu] 3840x2160 p010

mpv log: [cplayer] v: VO: Description: Shader-based GPU Renderer

mpv log: [vo/gpu] v: reconfig to 3840x2160 p010 bt.2020-ncl/bt.2020/pq/limited/display SP=5.911330 CL=mpeg2/4/h264

[mvk-info] Created 3 swapchain images with initial size (1170, 2532) and contents scale 3.0 for screen Main Screen.
mpv log: [vo/gpu] v: Resize: 1170x2532

mpv log: [vo/gpu] v: Window size: 1170x2532 (Borders: l=0 t=0 r=0 b=0)

mpv log: [vo/gpu] v: Video source: 3840x2160 (1:1)

mpv log: [vo/gpu] v: Video display: (0, 0) 3840x2160 -> (0, 937) 1170x658

mpv log: [vo/gpu] v: Video scale: 0.304688/0.304630

mpv log: [vo/gpu] v: OSD borders: l=0 t=937 r=0 b=937

mpv log: [vo/gpu] v: Video borders: l=0 t=937 r=0 b=937

mpv log: [vo/gpu] v: Texture for plane 0: 3840x2160

mpv log: [vo/gpu] v: Texture for plane 1: 1920x1080

mpv log: [vo/gpu] v: Testing FBO format rgba16f

mpv log: [vo/gpu] v: Using FBO format rgba16f.

mpv log: [vo/gpu] v: No advanced processing required. Enabling dumb mode.

mpv log: [vo/gpu/libplacebo] v: Spent 52.782 ms translating SPIR-V

mpv log: [cplayer] v: first video frame after restart shown

MPV_EVENT_PLAYBACK_RESTART
mpv log: [cplayer] v: audio ready

mpv log: [cplayer] v: delaying audio start 0.000000 vs. 0.000000, diff=0.000000

mpv log: [cplayer] v: playback restart complete @ 0.000000, audio=ready, video=playing

mpv log: [cplayer] v: starting audio playback

mpv log: [ao/audiounit] v: set latency: 0.010271 + 0.021333 -> 0.010271

mpv log: [cplayer] v: EOF code: 4

MPV_EVENT_END_FILE
mpv log: [cplayer] v: finished playback, success (reason 2)

My question is that

  1. Is that possible using gpu-context=moltenvk and iOS 16 new API to render HDR video ?
  2. If possible, how could I get the correct HDR color using gpu-context=moltenvk.

Thanks in advance.

**update at 2023-03-30: **

If I hard coded best_id to 33 in function pick_surf_format of libplacebo/src/vulkan/swapchain.c. The HDR color seems correct. And debug output picked correct surface configuration which mapped the video.

mpv log: [vo/gpu/libplacebo] v: Picked surface configuration 33: VK_FORMAT_B8G8R8A8_UNORM + VK_COLOR_SPACE_HDR10_ST2084_EXT

And here how VK_COLOR_SPACE_HDR10_ST2084_EXT configured.

    case VK_COLOR_SPACE_HDR10_ST2084_EXT:
        *out = (struct pl_color_space) {
            .primaries = PL_COLOR_PRIM_BT_2020,
            .transfer  = PL_COLOR_TRC_PQ,
        };

Still confused why libplacebo picked the wrong surface.

@tmm1
Copy link
Contributor Author

tmm1 commented Mar 29, 2023

Sorry I'm not sure how to fix it. Maybe @rcombs has some idea.

@m154k1 m154k1 mentioned this pull request Sep 26, 2023
Closed
@sfan5 sfan5 closed this Nov 20, 2023
@andrefmsilva andrefmsilva mentioned this pull request Dec 11, 2023
Merged
@karelrooted karelrooted mentioned this pull request Dec 14, 2023
Closed
@Akemi Akemi mentioned this pull request Feb 19, 2024
Closed
@Akemi Akemi mentioned this pull request May 10, 2024
Closed
@Akemi Akemi mentioned this pull request Nov 1, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qiudaomao qiudaomao qiudaomao left review comments

@haasn haasn haasn left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@tmm1 @qiudaomao @Akemi @alexiscn @haasn @sfan5