Skip to content

Guide for Video CODEC Encoder App Developers

Jump to bottom
Huts, Roman edited this page Jan 9, 2024 · 3 revisions

Advanced Micro Devices

Advanced Media Framework – Practical Guide for Video CODEC Encoder App Developers

Usage Guide

Disclaimer

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies.

Windows™, Visual Studio and DirectX are trademark of Microsoft Corp.

© 2024 Advanced Micro Devices, Inc. All rights reserved.

Copyright Notice

© 2024 Advanced Micro Devices, Inc. All rights reserved

Notice Regarding Standards. AMD does not provide a license or sublicense to any Intellectual Property Rights relating to any standards, including but not limited to any audio and/or video codec technologies such as MPEG-2, MPEG-4; AVC/H.264; HEVC/H.265; AV1; AAC decode/FFMPEG; AAC encode/FFMPEG; VC-1; and MP3 (collectively, the “Media Technologies”). For clarity, you will pay any royalties due for such third party technologies, which may include the Media Technologies that are owed as a result of AMD providing the Software to you.

MIT license

Copyright (c) 2024 Advanced Micro Devices, Inc. All rights reserved.

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Contents

1 Before reading this document, you need to know

  1. Who is the target audience of this document?'

    Software developers who wish to utilize AMD's hardware GPU or APU for video encoding or decoding.

  2. What does this document contain?

    This document contains a basic introduction to AMF and code samples for a simple video encoder developed based on AMF.

  3. What is AMF?

    AMF is a multimedia application development framework released by AMD, which includes a combination of data structures and interfaces. AMF serves as a bridge between application developers and AMD hardware, allowing developers to utilize the capabilities of AMD hardware through AMF.

  4. What specific capabilities does AMF include?

    AMF includes video encoders, video decoders, screen capturing, color space conversion, image scaling, and more.

  5. Where can AMF be found?

    The latest AMF package is released on https://github.com/GPUOpen-LibrariesAndSDKs/AMF .

  6. Where can application code samples be found?

    On https://github.com/GPUOpen-LibrariesAndSDKs/AMF/tree/master/amf/public/samples, there are various application code samples provided by AMF. Application developers can refer to these code samples to quickly develop their own applications.

2 Introduction

The Advanced Media Framework (AMF) is a multimedia processing framework developed by AMD (Advanced Micro Devices). It provides a set of components and interfaces for developing and optimizing multimedia applications, particularly those involving graphics processing units (GPUs) and accelerated processing units (APUs).

AMF features cross-platform and cross-API capabilities, allowing integration with various technologies and graphics APIs such as DirectX, OpenGL, OpenCL and Vulkan. It offers powerful functionalities including video encoding and decoding, audio processing, image processing, data transformation, and more.

The goal of AMF is to provide a lightweight and efficient media processing solution, enabling independent software vendor (ISV) developers to leverage the performance advantages of AMD hardware for developing and optimizing a wide range of multimedia applications. It is a powerful and flexible tool widely used in game development, video editing, video archive, real-time streaming, and other domains that require high-performance media processing by hardware.

3 AMF in ecosystem

The following diagram shows the position of AMF in full multimedia software stack ecosystem.

AMF in ecosystem

The AMF is a software component that plays a central role in a graphics stack by abstracting and unifying various graphics APIs, including DirectX, Vulkan, OpenGL, OpenCL, and the underlying layers. Its purpose is to provide a consistent interface for applications to access multimedia functionality across different platforms and APIs.

AMF doesn’t require an application to use the AMF framework for the entire application, unlike some other frameworks. Instead, it should be considered as a library where the application can obtain only what it needs. Applications can interact with the AMF functionality in multiple ways. The primary method is through the AMF Native API, which offers direct access to the capabilities provided by the AMF runtime. Additionally, other software layers and frameworks, such as Media Foundation, game engine plugins, and FFmpeg, can utilize the AMF functionality as well. This approach ensures that all multimedia functionalities are accessible through a single path, regardless of the specific API or framework being used.

As for the thread perspective, the AMF framework leaves threading to the application, although it supports both single-threaded and multi-threaded approaches.

4 Inside AMF

AMF is a single API accessible from C++ or from plain C language. This allows developers to use the framework with their preferred programming language and integrate it into their applications accordingly.

To support developers in using AMF in C++ code, AMF includes classes and interfaces that encapsulate the framework's functionality. This is designed to make it easier for developers to interact with AMD AMF within a C++ codebase.

On the other hand, to support developers in using AMF in plain C code, AMF includes functions and data structures that can be used with plain C programming. This allows developers who are working with C-based projects or prefer the simplicity of the C language to integrate and use AMD AMF functionality.

In the AMF code samples, we have implemented an AMF video encoder using C++ and plain C separately. Developers can find the differences between using the AMF API in C++ and C by going through the code in two encoders.

The following content will introduce the architecture of AMF. For the sake of simplicity, we will not intentionally distinguish the access from C++ or from plain C language.

You can find detailed information about AMF API on https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_API_Reference.md.

4.1 AMF Foundation

4.1.1 Elementary data types

AMF provides a standardized way to represent elementary data types like numbers and booleans. It offers some level of platform independence, it is not specifically designed to abstract elementary data types to be platform or compiler-independent. Its focus is on providing a compact and efficient serialization format for exchanging data across different systems.

Elementary data types are defined to offer some level of platform independence and make code potentially portable to other OSs. Detailed list of Elementary Data types is available in public/include/core/Platform.h.

Here are some examples of atomic data types.

typedef     int64_t             amf_int64;
typedef     int32_t             amf_int32;
typedef     int16_t             amf_int16;
typedef     int8_t              amf_int8;
typedef     uint64_t            amf_uint64;

4.1.2 Complex data types

These data types are complex data types defined as structures and constructed based on elementary data types, used to represent various aspects related to video encoding. Examples include types for representing points, representing rectangular regions, representing frame rates, and so on. The following are some complex data types.

Detailed list of complex data types is available in public/include/core/Platform.h

  • AMFRect
  • AMFSize
  • AMFPoint
  • AMFRate
  • AMFRatio
  • AMFColor
  • AMFGuid

4.1.3 AMFVariantStruct

AMFVariantStruct is a universal data type which can represent any type of data or can be used to store values of different types. It is a type that allows for flexible data handling and is used in code where dynamic typing or polymorphism is employed.

In its specific implementation, AMFVariantStruct is realized using a C union.

Variant Types

    typedef enum AMF_VARIANT_TYPE
    {
        AMF_VARIANT_EMPTY           = 0,

        AMF_VARIANT_BOOL            = 1,
        AMF_VARIANT_INT64           = 2,
        AMF_VARIANT_DOUBLE          = 3,

        AMF_VARIANT_RECT            = 4,
        AMF_VARIANT_SIZE            = 5,
        AMF_VARIANT_POINT           = 6,
        AMF_VARIANT_RATE            = 7,
        AMF_VARIANT_RATIO           = 8,
        AMF_VARIANT_COLOR           = 9,

        AMF_VARIANT_STRING          = 10,  // value is char*
        AMF_VARIANT_WSTRING         = 11,  // value is wchar_t*
        AMF_VARIANT_INTERFACE       = 12,  // value is AMFInterface*
        AMF_VARIANT_FLOAT           = 13,

        AMF_VARIANT_FLOAT_SIZE      = 14,
        AMF_VARIANT_FLOAT_POINT2D   = 15,
        AMF_VARIANT_FLOAT_POINT3D   = 16,
        AMF_VARIANT_FLOAT_VECTOR4D  = 17
    } AMF_VARIANT_TYPE;

Variant Structure

    typedef struct AMFVariantStruct
    {
        AMF_VARIANT_TYPE            type;
        union
        {
            amf_bool                boolValue;
            amf_int64               int64Value;
            amf_double              doubleValue;
            struct AMFRect          rectValue;
            struct AMFSize          sizeValue;
        ……
        };
    } AMFVariantStruct;

Variant Accessors

    static AMF_VARIANT_TYPE AMFVariantGetType(const AMFVariantStruct* _variant) { return (_variant)->type; }
    static amf_bool AMFVariantGetBool(const AMFVariantStruct* _variant) { return (_variant)->boolValue; }
    static amf_int64 AMFVariantGetInt64(const AMFVariantStruct* _variant) { return (_variant)->int64Value; }
    static amf_double AMFVariantGetDouble(const AMFVariantStruct* _variant) { return (_variant)->doubleValue; }
    static amf_float AMFVariantGetFloat(const AMFVariantStruct* _variant) { return (_variant)->floatValue; }
    static const AMFRect AMFVariantGetRect (const AMFVariantStruct* _variant) { return (_variant)->rectValue; }
    static const AMFSize AMF_STD_CALL AMFVariantGetSize (const AMFVariantStruct* _variant) { return (_variant)->sizeValue; }
……

AMFVariant

AMFVariantStruct has a helper class called AMFVariant. AMFVariant is a C++ class that inherits from AMFVariantStruct. AMFVariant provides some object-oriented features, such as copy constructor, assignment constructor, operator overloading, and so on. Users can use AMFVariant just like they would use primitive data types.

4.2 AMF Interfaces

In AMF, one of the most fundamental concepts is the use of interfaces. All functionality of the AMF runtime is accessible through interfaces, rather than exporting individual functions and class methods from the AMF runtime DLL. This design approach allows applications to be compiled with different compilers, linked with different C runtimes, and ensures compatibility regardless of these differences. The concept of AMF interfaces is similar to COM interfaces in Windows and follows similar rules. Notably, all AMF interfaces are reference-counted, simplifying memory management significantly.

All new objects and components in AMF are implemented in the form of AMF Interfaces. These interfaces are implemented in the form of abstract C++ classes. Most AMF interfaces will be derived from the AMFInterface basic interface. The AMFInterface defines three methods: Acquire, Release, and QueryInterface.

  • Acquire and Release: These methods are used to increment and decrement the reference counter of the object associated with the interface. When the reference count reaches 0, the object will destroy itself. It is important for applications to use these methods instead of directly calling delete() or free() on a pointer to an interface.
  • QueryInterface: This method allows for querying and obtaining other interfaces supported by the object. It is similar in concept to the QueryInterface method in COM.

By utilizing the reference counting mechanism and adhering to the Acquire and Release methods, memory management and object lifetime are simplified and controlled within the AMF framework.

The default implementation of AMFInterface is thread-safe.

    struct AMFInterface
    {
        virtual amf_long    AMF_STD_CALL Acquire()=0;
        virtual amf_long    AMF_STD_CALL Release()=0;
        virtual AMF_RESULT  AMF_STD_CALL QueryInterface(AMFInterfaceIDType interfaceID, void**ppInterface)=0; 
    };

AMF also defines a smart pointer C++ template, which calls Acquire() in its assignment operator and copy constructor and Release() in the destructor. Smart pointers simplify memory management and dramatically increase application reliability, eliminating a potential for memory leaks. Do use smart pointers when working with AMF objects instead of calling Acquire() and Release() manually.

All AMF interfaces have a corresponding smart pointer defined – following the pattern of adding the “Ptr” suffix to the interface name, as shown in the example here.

   AMFInterfacePtr_T<>
   struct MyInterface : public AMFInterface
   {
   AMF_DECLARE_IID(0x3bc312de, 0xc9b4, 0x48ed, 0x92, 0xe8, 0xd2, 0x42, 0xe6, 0x2d, 0xa8, 0xfa)

   AMF_RESULT AMF_STD_CALL MyMethod() = 0;
   };
   typedef AMFInterfacePtr_T<MyInterface> MyInterfacePtr;

   MyInterfacePtr pObject;

This diagram illustrates the interface hierarchy of AMF interfaces.

interface hierarchy of AMF interfaces

Please note that this is not an exhaustive list of all AMF interfaces, as there are many more. However, the interfaces depicted here are the most significant and fundamental ones that would be utilized for any task involving AMF. These interfaces can be classified into several categories, including data representation (such as buffers, surfaces, and textures), property storage, components, and contexts. We will explore each group in more detail in the following part.

4.3 AMF Property Storage

In AMF, property storage is implemented as a dictionary system that represents a name-value map. It is managed through two closely related interfaces: AMFPropertyStorage and AMFPropertyStorageEx.

The AMFPropertyStorage interface provides a flexible mechanism for attaching custom parameters or properties to data object. These properties can be used to store additional metadata, configuration settings, or any other relevant information associated with the specific data object.

The AMFPropertyStorageEx inherits from AMFPropertyStorage, while AMFPropertyStorageEx requires all properties stored in it to be pre-declared and performs value validation against type and range specified in the declaration.

The AMFPropertyStorage interface provides two crucial methods for working with properties:

  1. SetProperty: This method is used to write or set a property within the storage. It takes the name of the property as a parameter along with its corresponding value. By calling SetProperty, you can store or update the value of a specific property in the AMFPropertyStorage.

  2. GetProperty: This method retrieves the value of a property from the storage based on its name. By specifying the name of the property as a parameter, you can retrieve the corresponding value stored in the AMFPropertyStorage using the GetProperty method.

In C++ template functions, AMFPropertyStorage simplifies SetProperty/GetProperty and allows bypassing the creation of AMFVariantStruct directly.

    class AMFPropertyStorage : public AMFInterface
    {
    public:
        virtual AMF_RESULT SetProperty(const wchar_t *pName, 
                                AMFVariantStruct value)=0;
        virtual AMF_RESULT GetProperty(const wchar_t *pName, 
                                AMFVariantStruct* pValue)=0;
        ……
    };

4.4 AMF Memory

In AMF, the processing of video frames and audio data, both compressed and uncompressed, is a common scenario. This data can be stored either in an ordered form within buffers or as 2D images in different color formats. However, different graphics APIs like DirectX, OpenGL, Vulkan, etc., have their own specific structures and handles to represent this data.

To abstract these variations, AMF provides two main interfaces: AMFBuffer and AMFSurface, both inheriting from the AMFData interface.

  1. AMFBuffer: This interface is used to handle unordered memory, such as buffers. It allows for the storage and manipulation of data in the GPU or host memory.

  2. AMFSurface: This interface is specifically designed for representing 2D images in various color formats. It provides a standardized approach to handle image data, regardless of the underlying graphics API. AMFSurface enables operations on data residing in GPU or host memory.

4.4.1 AMFData

  1. Timestamps and Sample Durations:

AMFData objects have timestamps and sample durations associated with them. These timestamps represent the presentation or capture time of the data, and the sample durations indicate the duration for which a particular sample should be processed or displayed. The GetPts and SetPts methods are used to access and set the timestamps, while the GetDuration and SetDuration methods are used for sample durations.

  1. AMFData Interface and Memory Conversion:

The AMFData interface is crucial for enabling interoperability between different graphics APIs. It achieves this by providing a mechanism to convert memory types between various APIs. The memory types includes DX9, DX11, DX12, OpenCL, OpenGL, Vulkan, Host memory etc.. The Convert method in the AMFData interface is used for performing these memory conversions.

The cost and efficiency of each conversion operation depend on the specific type of conversion being performed. For instance, converting data from GPU memory to host memory typically involves a copy operation, which can be expensive from a performance standpoint. On the other hand, conversions between different types of GPU memory can be nearly zero-cost since they may involve straightforward memory mappings or transfers within the GPU.

The AMFData interface abstracts the process of performing memory conversions between different types, regardless of the specific conversion involved. This abstraction ensures that developers can use a consistent approach to handle these conversions, simplifying the implementation of interoperability between various graphics APIs. The underlying details and intricacies of each conversion type are encapsulated within the interface, allowing developers to focus on the higher-level aspects of their application without needing to manage the low-level conversion specifics.

AMF provides three functions for performing memory conversion: Interop(), Convert(), and Duplicate().

For developers who are highly sensitive to performance, it is essential to understand the differences between Interop() and Convert() in order to carefully choose the function that best suits their needs based on their specific requirements.

The key distinction is that Interop() works solely within GPU memory, while Convert() involves the use of host memory in the data transfer process between GPUs or between the host and GPU.

  • Interop() is a function that exclusively performs memory conversion on GPU memory. This means it operates directly on the memory located on the GPU and does not involve the use of host (CPU) memory in the data transfer process. Interop() function will fail if interoperability between APIs is not supported.

  • Convert() is a function that, when called, will initially attempt to use Interop(). If Interop() fails or is not available, it will then utilize host (CPU) memory to facilitate the transfer of data between two GPU APIs or between the host (CPU) and the GPU. In this case, data is temporarily copied from GPU memory to host memory or vice versa to enable the conversion or transfer, and then it may be transferred between the GPUs or between the host and GPU.

Duplicate() is used for data copy. It makes a copy via GPU or if not possible via HOST to new allocation.

class AMFData : public AMFPropertyStorage
{
public:
    ……
    virtual AMF_RESULT Duplicate(AMF_MEMORY_TYPE type, AMFData** ppData) = 0;	
    virtual AMF_RESULT Convert(AMF_MEMORY_TYPE type) = 0;
    virtual AMF_RESULT Interop(AMF_MEMORY_TYPE type) = 0;
    virtual AMF_DATA_TYPE   GetDataType()=0;
    virtual void            SetPts(amf_pts pts)=0;
    virtual amf_pts         GetPts()=0;
    virtual void            SetDuration(amf_pts duration)=0;
    virtual amf_pts         GetDuration()=0;
    ……
};

4.4.2 AMFBuffer

  1. Usage of AMFBuffer:

AMFBuffer objects usually serve as a storage mechanism for compressed video and audio frames that are loaded into GPU memory or host memory. These objects are designed to be general-purpose storage containers, allowing for efficient handling of data in various scenarios.

  1. Important Methods:

GetSize(): This method is used to retrieve the size of the buffer in bytes. It provides the information about the total size of the stored data within the AMFBuffer object.

GetNative(): The GetNative() method serves two purposes based on the location of the buffer:

  • If the buffer is located in host memory, the GetNative() method returns the actual pointer for host memory. This allows direct access to the buffer data from the host.

  • If the buffer is located in GPU memory, the GetNative() method returns a handle specific to the API used to allocate the buffer. For example, the GetNative() method will return a handle pointing to ID3D11Buffer if memory type is AMF_MEMORY_DX11. The handle could be a resource handle or a specific identifier that provides access to the buffer within the GPU memory. The exact type of handle returned will depend on the graphics API being used, such as DirectX, OpenGL, Vulkan, etc.

    class AMFBuffer : public AMFData
    {
    public:
        ……
        virtual amf_size  GetSize()=0;
        virtual void*     GetNative()=0;
        ……
    };

4.4.3 AMFSurface

  1. Usage of AMFSurface

AMFSurface represents a 2D image and can store images in various color formats such as RGB, YUV, and others. The specific color format used by a surface depends on the image data being stored.

  1. Color Formats and AMFPlane

Each color format may have different properties and require specific handling during processing. Depending on the color format, an AMFSurface may consist of multiple planes. Each plane is represented by the AMFPlane interface. The number of planes and their respective properties vary based on the color format being used.

The AMFPlane interface allows enumeration and access to these individual planes. The AMFSurface interface provides two methods to enumerate and access planes:

  • a. GetPlane(): This method allows you to retrieve a specific plane from the surface by specifying the plane index or identifier.

  • b. GetPlaneAt(): This method retrieves a plane based on its index in the plane list.

  1. Flags for Surface Type:

An AMFSurface representing a frame carries a set of flags indicating its type. These flags are used to specify the characteristics or properties of the surface. The GetFrameType() method retrieves the flags associated with the surface type, and the SetFrameType() method allows setting the flags for a surface.

In addition, by leveraging the AMFPropertyStorage interface, AMFData can attach and retrieve properties dynamically, allowing for extensibility and customization. This flexibility enables the attachment of user-defined parameters, which can be accessed and utilized during various processing operations.

class AMFSurface : public AMFData
{
public:
    ……
    virtual AMF_SURFACE_FORMAT GetFormat() = 0;
    virtual amf_size       GetPlanesCount() = 0;
    virtual AMFPlane*      GetPlaneAt(amf_size index) = 0;
    virtual AMFPlane*      GetPlane(AMF_PLANE_TYPE type) = 0;
    virtual AMF_FRAME_TYPE GetFrameType() = 0;
    virtual void           SetFrameType(AMF_FRAME_TYPE type) = 0;
    ……
};

4.4.4 AMFPlane

The AMFPlane interface represents a single two-dimensional plane within a multi-plane image format, or the entire image for an interleaved image format.

For multi-plane image formats, such as YUV, an image is typically composed of separate planes representing different color components or channels. Each plane represents one specific aspect of the image, such as luminance (Y) or chrominance (U and V). The AMFPlane interface allows access to these individual planes within the image through the GetNative() methods. For example, The GetNative() method will return a void pointer to ID3D11Texture2D or to other underlying object or a handle if memory type is AMF_MEMORY_DX11, and will returns the actual a void pointer to host memory if memory type is AMF_MEMORY_HOST. Each plane within the AMFPlane interface can have a distinct size, offset, and pitch values. The corresponding Get methods allow retrieving these values for each plane, facilitating proper data access and processing.

On the other hand, for interleaved image formats, the image is stored as a single plane, where the color information for each pixel is interleaved. In this case, the entire image is represented by the AMFPlane interface, providing access to the complete image data.

class AMFPlane : public AMFInterface
{
public:
    ……
    virtual AMF_PLANE_TYPE      GetType() = 0;
    virtual void*               GetNative() = 0;
    virtual amf_int32           GetPixelSizeInBytes() = 0;
    virtual amf_int32           GetOffsetX() = 0;
    virtual amf_int32           GetOffsetY() = 0;
    virtual amf_int32           GetWidth() = 0;
    virtual amf_int32           GetHeight() = 0;
    virtual amf_int32           GetHPitch() = 0;
    virtual amf_int32           GetVPitch() = 0;
    ……
};

4.5 AMFContext

The AMFContext interface indeed serves as an entry point to most AMF functionality and plays a crucial role in initializing graphical APIs, managing device-specific resources, and acting as a factory for various resources within the AMF framework.

  1. Initialization and Management: The AMFContext acts as a facility to initialize and manage graphical APIs and their associated devices. It provides methods to create and set up the necessary contexts, ensuring proper integration with the underlying hardware and software components.

  2. Resource Creation and Initialization: The AMFContext serves as a factory for creating and initializing device-specific resources, such as AMFBuffer, AMFSurface, and AMFCompute. Developers can use the AMFContext interface to create these resources, enabling efficient data storage, manipulation, and processing within the AMF framework.

  3. Wrapping Native Buffers and Surfaces: The AMFContext provides methods to wrap existing native buffers and surfaces, allocated through native APIs like DirectX or Vulkan, into corresponding AMF objects. This allows seamless integration of native resources into the AMF framework, enabling consistent handling and processing of data across different APIs.

class AMFContext : public AMFPropertyStorage
    {
    public:
        // DX9
        virtual AMF_RESULT          InitDX9() = 0;
        virtual void*               GetDX9Device() = 0;
        virtual AMF_RESULT          LockDX9() = 0;
        virtual AMF_RESULT          UnlockDX9() = 0;
        // DX11
        ……   
        // OpenCL
        ……   
        // OpenGL
        ……   
        // Allocation
        virtual AMF_RESULT          AllocBuffer() = 0;
        virtual AMF_RESULT          AllocSurface() = 0;
        virtual AMF_RESULT          AllocAudioBuffer() = 0;
        // Wrap existing objects
        virtual AMF_RESULT          CreateBufferFromHostNative() = 0;
        virtual AMF_RESULT          CreateSurfaceFromHostNative() = 0;
        virtual AMF_RESULT          CreateSurfaceFromDX9Native() = 0;
        ……

With the extension of the AMFPropertyStorage interface, AMFContext objects gain the capability to store additional properties dynamically. These properties can be used to attach custom metadata, configuration settings, or any other relevant information to the AMFContext object.

The existence of three different AMFContext interfaces, namely AMFContext, AMFContext1, and AMFContext2, is to accommodate the inclusion of new APIs while preserving backward compatibility. Despite the separate interfaces, they are implemented by the same object internally. To access different interfaces implemented by the same object, applications can use the QueryInterface method. This method allows obtaining a pointer to any interface from any other interface already held by the object. You can obtain a pointer to the desired interface based on your application's needs and the available functionality you require.

4.6 AMFComponent

The AMFComponent interface represents the major functional blocks within the AMF framework, including encoders, decoders, converters, scalers, and capture functionalities. It acts as a common interface for these components, allowing for standardized interaction and control.

Important Methods:

  • a. Init/Terminate: The Init method is used to configure and initialize a component before it can be used. It typically takes parameters specific to the component's functionality, such as color format and resolution for encoders, or output settings for decoders. The Terminate method is used to release resources and perform cleanup operations associated with the component. AMFComponent extends the AMFPropertyStorageEx interface, which means that properties can be set on the component to configure its behavior. These properties are predefined and are typically defined in the component's corresponding C++ header file. The properties have associated IDs, types, valid ranges, and initial values, allowing for fine-grained configuration of the component. These properties can be either static or dynamic:

    • Static properties are properties of a component that are defined at initialization and typically do not change during the execution of the program. These properties are set once and remain constant throughout the lifetime of the component. They are often used for configuration settings, initial states, or constants.

    • Dynamic properties are properties of a component that can change during runtime based on various factors or interactions with the application. These properties are not fixed and can be modified as the program runs. They are often used to represent states, user inputs, or data that changes over time. Dynamic properties are essential when you need to track and update information that may change while the application is running.

    • It's important to understand whether a property is static or dynamic when working with components in AMF, as it influences how you interact with and manipulate those properties within your application.

  • b. SubmitInput: The SubmitInput method is used to submit input data to the component. In the case of a video encoder, you would typically submit AMFSurface objects containing uncompressed video frames as input. These AMFSurface objects represent the raw video data that needs to be encoded. On the other hand, for a video decoder, you would submit AMFBuffer objects containing compressed frames as input. These AMFBuffer objects contain the compressed video data that needs to be decoded.

  • c. QueryOutput: The QueryOutput method is used to retrieve output data from the component. For a video encoder, the output would typically be in the form of AMFBuffer objects containing compressed frames. These AMFBuffer objects represent the encoded video data. Conversely, for a video decoder, the output would consist of AMFSurface objects containing decoded frames. These AMFSurface objects represent the raw, decoded video frames.

SubmitInput and QueryOutput work in asynchronous mode in AMF. The "asynchronous mode" means that when you call SubmitInput, it doesn't block the execution of your program. Instead, it allows your application to continue processing other tasks while the input data is being handled or processed by the AMF component in the background. Application can use QueryOutput to retrieve output data in separate thread.

The use of asynchronous mode like SubmitInput and QueryOutput in AMF is beneficial because it allows your application to remain responsive and continue performing other tasks while AMF processes data or computations in the background. This asynchronous behavior is common in multimedia processing, data streaming, and similar applications where real-time or continuous data handling is required.

AMFComponent uses factory design pattern. All AMF components are created using the CreateComponent method.

class AMFComponent : public AMFPropertyStorageEx
{
public:
    ……
    virtual AMF_RESULT  Init(AMF_SURFACE_FORMAT format, amf_int32 width, amf_int32 height) = 0;
    virtual AMF_RESULT  Terminate() = 0;
    virtual AMF_RESULT  SubmitInput(AMFData* pData) = 0;
    virtual AMF_RESULT  QueryOutput(AMFData** ppData) = 0;
    virtual AMFContext* GetContext() = 0;
    ……
}

5 Code Samples: Use AMF to Implement an Encoder

In the previous AMF ecosystem diagram, we can see that there can be many ways to use AMF. In this code example, we will provide a complete encoder example using the path marked along the red line.

AMF encoder block path

The Frameworks AMD Native C/C++ code can be visited at: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/tree/master/amf/public/common

This framework provides a C++ wrapper for AMF, serving two purposes. Firstly, it facilitates the usage of AMF in C++ applications, making it easier to integrate AMF functionality into C++ projects. Secondly, it simplifies the process of invoking AMF functions, providing a more straightforward and user-friendly approach to interact with AMF.

5.1 Basic workflow for using AMF

Based on AMF, developing a video encoder application involves four fundamental stages. Each stage further includes specific steps.

  • Set up the AMF context stage
  • Setup encoder component stage
  • Encode each frame stage
  • Finish encoding work stage

The following diagram shows the basic workflow for using AMF.

Encoder basic workflow for using AMF

5.2 Step by step code samples

To use AMF with code samples to implement an encoder, you can follow these steps. The complete C++ code for the samples in this document can be obtained from the code samples of the AMF encoder https://github.com/GPUOpen-LibrariesAndSDKs/AMF/tree/master/amf/public/samples/CPPSamples/SimpleEncoder.

AMF package also provides the plain C code samples of the AMF encoder which can be found at https://github.com/GPUOpen-LibrariesAndSDKs/AMF/tree/master/amf/public/samples/SamplesC/SimpleEncoderC . The C and C++ versions samples can be used to meet the needs of different ISV developers.

  1. Set up the AMF context:
    AMF_RESULT res = AMF_OK; 
    res = g_AMFFactory.Init();
    if (res != AMF_OK)
    {
        wprintf(L"AMF Failed to initialize");
        return res;
    }

    // initialize AMF
    amf::AMFContextPtr context;

    // context
    res = g_AMFFactory.GetFactory()->CreateContext(&context);
    AMF_RETURN_IF_FAILED(res, L"CreateContext() failed");
  • Initialize the context using the Init method which corresponds to specific device (e.g., DirectX, Vulkan) to configure the desired parameters.
    else if (memoryTypeIn == amf::AMF_MEMORY_DX11)
    {
        res = context->InitDX11(NULL); // can be DX11 device
        AMF_RETURN_IF_FAILED(res, L"InitDX11(NULL) failed");
        PrepareFillFromHost(context);
    }
  1. Create an encoder component:
  • Create an instance of the AMFComponent interface, specific to the encoder you want to use (e.g., H.264 encoder).
    // component: encoder
    amf::AMFComponentPtr encoder;
    const wchar_t* pCodec = AMFVideoEncoderVCE_AVC;

    res = g_AMFFactory.GetFactory()->CreateComponent(context, pCodec, &encoder);
    AMF_RETURN_IF_FAILED(res, L"CreateComponent(%s) failed", pCodec);
  • Initialize the encoder component using the Init() method, providing any required configuration parameters. These parameters could include video resolution, frame rate, bitrate, and encoding settings.
if (amf_wstring(pCodec) == amf_wstring(AMFVideoEncoderVCE_AVC))
{
    res = encoder->SetProperty(AMF_VIDEO_ENCODER_USAGE, AMF_VIDEO_ENCODER_USAGE_TRANSCODING);
    AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_USAGE, AMF_VIDEO_ENCODER_USAGE_TRANSCODING) failed");
    res = encoder->SetProperty(AMF_VIDEO_ENCODER_TARGET_BITRATE, bitRateIn);
    AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_TARGET_BITRATE, %" LPRId64 L") failed", bitRateIn);
    res = encoder->SetProperty(AMF_VIDEO_ENCODER_FRAMESIZE, ::AMFConstructSize(widthIn, heightIn));
    AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_FRAMESIZE, %dx%d) failed", widthIn, heightIn);
    res = encoder->SetProperty(AMF_VIDEO_ENCODER_FRAMERATE, ::AMFConstructRate(frameRateIn, 1));
    AMF_RETURN_IF_FAILED(res, L"SetProperty(AMF_VIDEO_ENCODER_FRAMERATE, %dx%d) failed", frameRateIn, 1);

}
res = encoder->Init(formatIn, widthIn, heightIn);
AMF_RETURN_IF_FAILED(res, L"encoder->Init() failed");

In this code sample, an H.264 encoder is used as an example. You can find all the properties related to the AMF H.264 encoder in the AMF H.264 encoder API documentation https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_Video_Encode_API.md.

Based on this code sample, with a few modifications, you can implement HEVC and AV1 encoders. You can find all the properties related to the AMF HEVC encoder in the AMF HEVC encoder API documentation https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_Video_Encode_HEVC_API.md. Similarly, you can find all the properties related to the AMF AV1 encoder in the AMF AV1 encoder API documentation https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_Video_Encode_AV1_API.md.

  1. Set additional encoder properties:
  • Use the SetProperty() method of the AMFComponent interface to set any additional properties specific to the encoder.

Those properties which are configured before the Init() method are called static properties (e.g., profile, level, usage), and those properties will apply until the end of the encoding session .Those properties which can be configured after the Init() method are called dynamic properties. All dynamic properties have default values. Several properties can be changed subsequently and these changes will be flushed to encoder only before the next SubmitInput () call. Typical dynamic parameters include AMF_VIDEO_ENCODER_TARGET_BITRATE, AMF_VIDEO_ENCODER_B_PIC_PATTERN, AMF_VIDEO_ENCODER_FILLER_DATA_ENABLE, AMF_VIDEO_ENCODER_HEADER_INSERTION_SPACING etc.

  1. Prepare input data:

The step "preparing input data" typically consists of three sub-steps.

  • Create or obtain uncompressed video frames that you want to encode.
  • Allocate an AMFSurface object using the AMFContext's AllocSurface() method.
    res = context->AllocSurface(memoryTypeIn, formatIn, widthIn, heightIn, &surfaceIn);
    AMF_RETURN_IF_FAILED(res, L"AllocSurface() failed");
  • Fill the AMFSurface object with the uncompressed video frame data using the AMFMemoryCopy() method or similar.

In this sample, the input of the video encoder is an animation generated programmatically. In this animation, on a purple background, a green block moves slowly from top left towards the bottom right corner. In the following code, the function FillSurfaceDX11 is the program that creates input raw data.

    else
    {
        FillSurfaceDX11(context, surfaceIn);
    }
  1. Submit input data to the encoder:
  • Use the SubmitInput() method of the AMFComponent interface to submit the AMFSurface object containing the uncompressed video frame for encoding.
    res = encoder->SubmitInput(surfaceIn);
    if (res == AMF_NEED_MORE_INPUT) // handle full queue
    {
        // do nothing
    }
    else if (res == AMF_INPUT_FULL || res == AMF_DECODER_NO_FREE_SURFACES)
    {
        amf_sleep(1); // input queue is full: wait, poll and submit again
    }
    else
    {
        AMF_RETURN_IF_FAILED(res, L"SubmitInput() failed");
        surfaceIn = NULL;
        submitted++;
    }
  1. Retrieve encoded output data:

  • Use the QueryOutput() method of the AMFComponent interface to retrieve the encoded data as an AMFBuffer object.

  • Extract the encoded data from the AMFBuffer object and save it or pass it to further processing or storage as needed.

    amf::AMFDataPtr data;
    res = m_pEncoder->QueryOutput(&data);
    if(res == AMF_EOF)
    {
        break; // Drain complete
    }
    if ((res != AMF_OK) && (res != AMF_REPEAT))
    {
        // trace possible error message
        break; // Drain complete
    }
    if(data != NULL)
    {
        amf::AMFBufferPtr buffer(data); // query for buffer interface
        m_pFile.write(reinterpret_cast<char*>(buffer->GetNative()), buffer->GetSize());
    }

The above describes the process of encoding one frame from Step 4 to Step 6. Repeat the operations from Step 4 to Step 6 until all frames are encoded.

  1. Clean up resources:
  • Drain the remaining frames in encoder input queue.
    // drain encoder; input queue can be full
    while (true)
    {
        res = encoder->Drain();
        if (res != AMF_INPUT_FULL) // handle full queue
        {
            break;
        }
        amf_sleep(1); // input queue is full: wait and try again
    }

  • Terminate and release the encoder component using the Terminate() method of the AMFComponent interface.

  • Terminate and release the AMF context using the Terminate() method of the AMFContext interface.

    // cleanup in this order
    surfaceIn = NULL;
    encoder->Terminate();
    encoder = NULL;
    context->Terminate();
    context = NULL; // context is the last

    g_AMFFactory.Terminate();

Please note that the actual implementation details may vary depending on the specific AMF version, encoder type, and programming language you are using. The provided steps give a general outline of the process involved in implementing an encoder using AMF. It is recommended to consult the AMF documentation and code sample provided by the AMF SDK for more detailed and language-specific guidance.

5.3 AMF Asynchronous and synchronous Model

You can design an encoder using either a synchronous model or an asynchronous model. The choice of which model to use depends on the software's system architecture. In this section, we will introduce both models separately.

5.3.1 Asynchronous Model

The performance of a video encoder is a key technical factor. In the above sections, data ingress into the encoder and subsequently querying the encoded data from it can be done sequentially in a single thread. However, a better approach is to design these two steps to work asynchronously to significantly improve performance and reduce latency. Through multi-threaded programming, the steps of SubmitInput and QueryOutput can be designed to work asynchronously. SubmitInput operates in one thread, while QueryOutput operates in another thread, maximizing the encoder's throughput. The code sample of the AMF encoder https://github.com/GPUOpen-LibrariesAndSDKs/AMF/tree/master/amf/public/samples/CPPSamples/SimpleEncoder adopts such a design.

The following diagram illustrates the asynchronous synchronous model of this sample encoder.

Asynchronous model of sample encoder

  1. Declaration of Thread Class
    class PollingThread : public amf::AMFThread
  1. Thread 2 Start
    PollingThread thread(context, encoder, pFileNameOut);
    thread.Start();

This function call thread.Start() will call the thread creating function provided by Operating system, for example _beginthread function in Windows OS or pthread_create in Linux, to create a thread. Eventually PollingThread::Run() is the function that the new thread will execute.

  1. Thread 2 Execution.
    void PollingThread::Run()
    {
        RequestStop();

        AMF_RESULT res = AMF_OK; // error checking can be added later
        while(true)
        {
            amf::AMFDataPtr data;
            res = m_pEncoder->QueryOutput(&data);
            if(res == AMF_EOF)
            {
                break; // Drain complete
            }
            if ((res != AMF_OK) && (res != AMF_REPEAT))
            {
                // trace possible error message
                break; // Drain complete
            }
        }

One thing to note is that AMFThread class is a helper, not mandatory thing when using asynchronous model. Developers can use any other class that supports multithreading.

5.3.2 Synchronous Model

In the synchronous model, input is submitted and output is queried from a single thread. There is a loop that prepares and submits an input sample, then immediately proceeds to check for output. If output is available, it is processed and handled and the loop repeats itself. The following diagram illustrates the synchronous model of this sample encoder. Synchronous mode may appear straightforward, but this apparently simple model can become complex when the number of output samples does not match the number of input samples, as in the case of frame rate conversion. Additionally, the hardware block represented by the component may have internal buffering with limited depth, which could result in SubmitInput failures. This situation would require additional logic to manage input buffering while continuing to request output.

Synchronous model of sample encoder

6 Glossary of Acronyms

Acronym Definition
AMD Advanced Micro Devices
AMF Advanced Media Framework
API Application Programming Interface
APU Accelerated Processing Unit
COM Component Object Model
CPU Central Processing Unit
CU Compute Unit
DLL Dynamic link library
GFX Graphics
GPU Graphics Processing Unit
HEVC High Efficiency Video Coding
ISV Independent software vendor
MFT Media Foundation Transforms
MKV Matroska Video files
MP4 MPEG-4 Part 14
OpenCL Open Computing Language
OpenGL Open Graphics Library
SDK Software Development Kits
UE4 Unreal Engine 4
UVD Unified Video Decoder
UVE Universal Video Encode
VCE Video Coding Engine
VCN Video Codec Next

7 References

  1. AMF API Reference: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_API_Reference.md
  2. H.264 Encoder API: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_Video_Encode_API.md
  3. HEVC Encoder API: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_Video_Encode_HEVC_API.md
  4. AV1 Encoder API: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/blob/master/amf/doc/AMF_Video_Encode_AV1_API.md
  5. AMF Code Samples: https://github.com/GPUOpen-LibrariesAndSDKs/AMF/tree/master/amf/public/samples
Clone this wiki locally