The MediaFoundation thread

[h1cks]

I've started this thread to basically discuss and share the experiences with this rather troublesome area of DX.

There are some additional pieces that need to be moved I believe before this library is fully functional, I'm also looking at the Media Engine as well, but after spending some fruitless hours a few weeks ago, I realised why no one really should port it. It's a #include all type scenario. Basically seems to pull in aspects from different libraries (as I reviewed the sharpdx version also) which in turn requires additional code and mappings to be added to other libraries.

@amerkoleci is that your understanding of the media engine component from sharpdx, it seems to be a bit of a hydra.

[h1cks]

@amerkoleci I've upped my sponsorship btw. Keep up the good work.

[amerkoleci]

Oh didn't know that, thank you, truly appreciate!

[amerkoleci]

Did you had time to try recent media foundation changes? I've improved some things recently

[h1cks]

@amerkoleci not of late, but definitely can go and have a look as the code base is there. If it works, I will move to Vortice permanently. All my tooling is still in Sharpdx, this is the only piece left (Video playback).

[amerkoleci]

Lets try and port it to vortice, I can add the missing bits, just tell me what changes you need

[h1cks]

I've gone and at least made the challenge easier by removing the need for MediaEX, which was completely unneccessary. Hoping this will allow for an easier implementation now. Will try to convert the media component when I have some time, bit busy of late.

[h1cks]

I dont know why i used mediaex except that I was being a bit lazy. The changes to playback in my current use of mediaengine are a lot simpler now, will have a play when I can to get this converted. By using mediaengineex, i also cut over playback from Windows 7 based machines. Now that i'm using a more basic version, I hope that improves the capability.

[h1cks]

@amerkoleci looks like we might be missing some interfaces. I have linked to code that I basically have almost a photocopy of in c# (I didnt copy, this guy just coded it like i did, and im freaked).

https://github.com/microsoft/Xbox-ATG-Samples/blob/main/UWPSamples/Graphics/VideoTextureUWP/MediaEnginePlayer.cpp

Basically there might be some interfaces I cant seem to fine in Vortice atm, and they might be there.

• DXGIDeviceManager -> Created by MFCreateDXGIDeviceManager
• Playback on MediaEngine seems to be readonly , I cant set a callback.

Will let you know as I find more.

[amerkoleci]

1. this is the class: https://github.com/amerkoleci/Vortice.Windows/blob/main/src/Vortice.MediaFoundation/IMFDXGIDeviceManager.cs
2. https://github.com/amerkoleci/Vortice.Windows/blob/main/src/samples/MediaFoundation/AudioPlay/Program.cs example on how to setup callback

[h1cks]

I tried it out and have it working...sort of. its not playing at all frames but time pacing is correct, but audio is playing correctly and I seem to be getting about 1/5th of the frames I should be getting. All events are as expected. The code is almost the same as the Sharpdx versioin no error. Will play later and see how it goes. I am getting video and audio though.

[h1cks]

I checked the MediaEngineNotifyDelegate and noticed you are using a NUINT rather than a long. is this intentional?

[h1cks]

Im still digging, its not an issue at the moment for me to look into it.

[h1cks]

Not sure what to do atm, there isnt too much to do to get media running, and it works...to a degree.

[h1cks]

I tried other videos just in case, same issue.

[amerkoleci]

But I need something to test, otherwise:)

[h1cks]

will drop you my videoplayer class in the coming day or so

[h1cks]

using Vortice;
using Vortice.DXGI;
using Strafe.GameControllers;
using System;
using System.IO;
using System.Runtime.InteropServices;
using System.Security;
using D3D11 = Vortice.Direct3D11;
using Vortice.Direct3D11;
using Vortice.Mathematics;
using Vortice.MediaFoundation;
using System.Drawing;
using Vortice.Multimedia;
using SharpGen.Runtime;
using System.Diagnostics;

namespace Insane3D
{
    public class VideoPlayer : IDisposable
    {
        private IMFDXGIDeviceManager _dxgiDeviceManager;
        private IMFMediaEngine _mediaEngine;

        protected bool _initialised;
        protected bool _isEndOfStream;
        protected bool _isVideoStopped;
        protected bool _isPlaying;
        protected bool _frameReady;

        protected IDXGISurface _renderSurface;

        protected bool _fileLoaded;
        MFByteStream _byteStream;

        private readonly object lockObject = new object();
        private Color3 _backgroundColor = new Color3(0);

        protected ID3D11Texture2D _outputVideoTexture;
        protected ID3D11Device1 _parentDevice;
        protected DirectXManager _directManager;
        protected bool _noFMV;

        public string Url { get; set; }

        //==========================================================================================================================//

        public VideoPlayer(DirectXManager directXManager)
        {
            _directManager = directXManager;

            _initialised = false;
            _isEndOfStream = false;
            _isPlaying = false;

            _fileLoaded = false;

            _noFMV = false;
        }

        //==========================================================================================================================//

        public void Initialise(ID3D11Device1 device)
        {
            try
            {
                _parentDevice = device;

                lock (lockObject)
                {
                    MediaFactory.MFStartup();
                }
            }
            catch (Exception ex)
            {
                ErrorHandler.DoErrorHandling(ex, ErrorHandler.GetCurrentMethod(ex));
            }

        }

        //==========================================================================================================================//

        bool Restart()
        {
            lock (lockObject)
            {
                lock (lockObject)
                {
                    _renderSurface?.Dispose();
                    _outputVideoTexture?.Dispose();
                    _mediaEngine?.Dispose();
                    _dxgiDeviceManager?.Dispose();

                    var multithread = _parentDevice.QueryInterface<ID3D11Multithread>();
                    multithread.SetMultithreadProtected(true);

                    try
                    {
                        using (var factory = new IMFMediaEngineClassFactory())
                        {
                            // Create a DXGI Device Managezr

                            _dxgiDeviceManager = MediaFactory.MFCreateDXGIDeviceManager();
                            _dxgiDeviceManager.ResetDevice(_parentDevice);

                            Debug.Assert(_dxgiDeviceManager != null);

                            IMFAttributes attributes = MediaFactory.MFCreateAttributes(1);

                            attributes.VideoOutputFormat = Format.R8G8B8A8_UNorm;
                            attributes.DxgiManager = _dxgiDeviceManager;

                            _mediaEngine = factory.CreateInstance(MediaEngineCreateFlags.WaitForStableState, attributes, OnMediaEngineEvent);

                        }
                    }
                    catch (Exception ex)
                    {
                        return false;
                        //ErrorHandler.DoErrorHandling(ex, ErrorHandler.GetCurrentMethod(ex));
                    }
                }


            }
            D3D11.Texture2DDescription desc = new D3D11.Texture2DDescription()
            {
                MiscFlags = D3D11.ResourceOptionFlags.Shared,
                CPUAccessFlags = D3D11.CpuAccessFlags.None,
                Format = Format.R8G8B8A8_UNorm,
                Width = _directManager.ModeDescription.Width,
                Height = _directManager.ModeDescription.Height,
                ArraySize = 1,
                Usage = ResourceUsage.Default,
                MipLevels = 1,
                SampleDescription = new Vortice.DXGI.SampleDescription() { Count = 1, Quality = 0 },
                BindFlags = D3D11.BindFlags.RenderTarget
            };


            _outputVideoTexture = _parentDevice.CreateTexture2D(desc);

            _outputVideoTexture.DebugName = "output video texture";

            _renderSurface = _outputVideoTexture.QueryInterface<IDXGISurface>();

            _initialised = true;

            return true;
        }


        //==========================================================================================================================//

        public bool Execute(ID3D11DeviceContext1 deviceContext, ID3D11Texture2D targetSurface, AbstractInputDevice inputdevice)
        {
            if (_noFMV)
                return false;

            try
            {
                if (!_initialised)
                {
                    if (!Restart())
                    {
                        return false;
                    }

                    SetBytestream();

                    _isVideoStopped = false;
                }


                if (_fileLoaded)
                {
                    if (inputdevice.AnyButtonPressed(false))
                    {
                        return false;
                    }
                    else
                    {
                        OnRender(deviceContext, targetSurface);
                    }

                    return !_isVideoStopped;
                }

                return !_isEndOfStream;
            }
            catch
            {
                _noFMV = true;
            }

            return false;
        }

        //==========================================================================================================================//

        private void StopVideo()
        {
            _isVideoStopped = true;
            _isPlaying = false;
            _fileLoaded = false;

            _mediaEngine.Shutdown();


            var multithread = _parentDevice.QueryInterface<ID3D11Multithread>();
            multithread.SetMultithreadProtected(false);
        }

        //==========================================================================================================================//

        void Play()
        {
            if (_mediaEngine != null)
            {
                if (_mediaEngine.HasVideo() && _isVideoStopped)
                    _isVideoStopped = false;

                if (_isEndOfStream)
                {
                    PlaybackPosition = 0;
                    _isPlaying = true;
                }
                else
                {
                    _mediaEngine.Play();
                }

                _isEndOfStream = false;
            }


        }


        public double PlaybackPosition
        {
            get
            {
                if (_mediaEngine != null)
                    return _mediaEngine.CurrentTime;
                return 0.0;
            }
            set
            {
                if (_mediaEngine != null)
                    _mediaEngine.CurrentTime = value;
            }
        }


        //==========================================================================================================================//

        public virtual void OnRender(ID3D11DeviceContext1 deviceContext, ID3D11Texture2D targetSurface)
        {
            if (_isVideoStopped)
                return;

            if (_mediaEngine != null)
            {
                if (_mediaEngine.OnVideoStreamTick(out long value_))
                {
                    if (value_ > 0)
                    {
                        if (_outputVideoTexture != null)
                        {
                            var desc = _outputVideoTexture.Description;
                            RawRect region = new RawRect(0, 0, desc.Width, desc.Height);

                            VideoNormalizedRect rect_ = new VideoNormalizedRect()
                            {
                                Bottom = 1,
                                Left = 0,
                                Right = 1,
                                Top = 0,
                            };

                            _mediaEngine.TransferVideoFrame(_renderSurface, rect_, region, new Color4(0, 0, 0, 1));

                            deviceContext.ResolveSubresource(targetSurface, 0, _outputVideoTexture, 0, DirectXManager.SupportedBackbufferFormat);
                        }
                    }
                }
            }
        }

        //==========================================================================================================================//

        public void Shutdown()
        {
            lock (lockObject)
            {
                _initialised = false;

                StopVideo();

                _renderSurface?.Dispose();
                _outputVideoTexture?.Dispose();

                _renderSurface = null;
                _outputVideoTexture = null;
                _renderSurface = null;
            }
        }

        //==========================================================================================================================//

        public void SetBytestream()
        {
            FileStream streamHandle = new FileStream(Url, FileMode.Open);

            var absoluteurl_ = new Uri(streamHandle.Name, UriKind.RelativeOrAbsolute);

            _mediaEngine.SetSource(absoluteurl_.AbsoluteUri);

            _fileLoaded = false;
            _isEndOfStream = false;
        }

        ////==========================================================================================================================//

        int _width;
        int _height;

        protected virtual void OnMediaEngineEvent(MediaEngineEvent mediaEvent, nuint param1, int param2)
        {
            switch (mediaEvent)
            {
                case MediaEngineEvent.NotifyStableState:
                    SafeNativeMethods.SetEvent(new IntPtr((long)param1));
                    break;
                case MediaEngineEvent.LoadedMetadata:
                    _isEndOfStream = false;
                    break;

                case MediaEngineEvent.LoadedData:
                    _fileLoaded = true;

                    _mediaEngine.GetNativeVideoSize(out _width, out _height);
                    break;

                case MediaEngineEvent.CanPlay:

                    // Start the Playback
                    if (!_isPlaying)
                        if (_initialised)
                            Play();
                    break;
                case MediaEngineEvent.Play:
                    _isPlaying = true;
                    break;
                case MediaEngineEvent.Pause:
                    _isPlaying = false;
                    break;
                case MediaEngineEvent.Ended:
                    if (_mediaEngine.HasVideo())
                    {
                        StopVideo();
                    }
                    _isEndOfStream = true;
                    break;
                case MediaEngineEvent.TimeUpdate:
                    _frameReady = true;
                    break;
                case MediaEngineEvent.Error:
                    break;

            }
        }

        [SuppressUnmanagedCodeSecurityAttribute]
        internal static class SafeNativeMethods
        {
            [DllImport("kernel32.dll", EntryPoint = "SetEvent")]
            internal static extern bool SetEvent(IntPtr hEvent);
        }

        #region IDisposable Support
        private bool disposedValue; // To detect redundant calls

        protected virtual void Dispose(bool disposing)
        {
            if (!disposedValue)
            {
                if (disposing)
                {
                    if (_renderSurface != null)
                        _renderSurface.Dispose();
                    if (_outputVideoTexture != null)
                        _outputVideoTexture.Dispose();

                }

                disposedValue = true;
            }
        }

        // This code added to correctly implement the disposable pattern.
        public void Dispose()
        {
            // Do not change this code. Put cleanup code in Dispose(bool disposing) above.
            Dispose(true);
            // TODO: uncomment the following line if the finalizer is overridden above.
            // GC.SuppressFinalize(this);
        }
        #endregion
    }



}

[h1cks]

@amerkoleci my hacked up video player class atm

I was playing mp4s.

[amerkoleci]

Here you have an example on how to play video using D3D11:

https://github.com/amerkoleci/Vortice.Windows.Samples/blob/main/src/MediaFoundation/MediaEnginePlayVideo/Program.cs

I've tried with some video from here: (Bunny) and it works, both video and audio plays correctly.
https://gist.github.com/jsturgis/3b19447b304616f18657

[h1cks]

I found the issue and it was by updating to the new nuget,.

You updated the function definition to return the bool which before only returned the frametime in => _mediaEngine.OnVideoStreamTick()

to

mediaEngine.OnVideoStreamTick(out long value)

I could then wait for the frame to be ready.

btw. You dont need mediaengineex to play these videos. In fact, If i recall, media engine ex isnt available for use in Windows 7 FMV. I need a windows 7 machine to retest, but I had tested it previously.

[h1cks]

I've updated my videoplayer code for anyone to use if they like.