Enable fully custom Win2D effects: new ICanvasImageInterop API

[Sergio0694]

Background and motivation

Win2D is an amazing library that makes it really easy for everyone to build custom effects pipelines and implement all sorts of cool visual effects, especially in combination with the Composition layer. APIs such as CanvasControl, CanvasAnimatedControl, CanvasImageSource and CompositionDrawingSurface make it very convenient for developers to just get a ready to use D2D canvas they can use to create anything they need, and there's lots of built-in APIs to draw images, strokes, shapes, text, and to apply effects. There is a limitation though in cases where advanced users had the need to implement custom effects, as that's currently not supported. That is, there is no way to implement an ICanvasImage outside of Win2D itself.

The only way to add custom rendering steps into a drawing session is via the built-in PixelShaderEffect, which does allow developers to at least draw custom D2D1 pixel shaders, but that comes with its own set of limitations:

• It is restricted to D2D1 pixel shaders using Shader Model 4
• It uses reflection to dynamically create bindings for variables
  • This adds unnecessary overhead during initialization
  • This also prevents users from efficiently set the whole constant buffer in one go
  • This also limits the possible types that can be used in fields (eg. no custom struct types)
• It doesn't support D2D1 resource textures
• It doesn't support fully customizable D2D1 draw transform mappings

That is: PixelShaderEffect itself has several drawbacks, and after experimenting with it for a bit I came to the conclusion that there isn't really a way to address them without introducing any breaking changes (which are of course out of question). Not just that, even that wouldn't actually address the core of the problem: there should be a way to allow any developer to freely extend Win2D's functionality and to implement their own fully customizable effects that can then interop seamlessly with Win2D.

There is a way to do Win2D/D2D interop (as documented here), which is great and can help fill some gaps, but that still isn't an ideal scenario, for two main reasons:

• It doesn't allow developers to author their own ICanvasImage effects that consumers can then easily use just like any other Win2D built-in effect. That includes eg. passing them to a DrawImage call on a drawing session, or using them as inputs for other effects. Retrieving an ID2D1DeviceContext1 from a CanvasDrawingSession object would only allow developers to inject their own rendering logic right there, but it wouldn't enable them to author and publish custom effects that would feel on par with Win2D's ones, thus greatly limiting them and making them not feel "on the same level" as the built-in effects.
• It wouldn't work with some of the other APIs working with ICanvasImage, such as the static CanvasImage.SaveAsync.

This proposal aims to fully address all of these concerns, and to enable all developers to author custom ICanvasImage-s.

Proposed API

The way Win2D handles input ICanvasImage-s from any of the public APIs is by casting them to ICanvasImageInternal, which is a private, C++ API that exposes the necessary extension points to interoperate with these images. The proposal extends this by adding a three publicly documented APIs that developers will be able to use for custom effects.

Three new APIs will be added to ABI.Microsoft.Graphics.Canvas in winrt/published/Microsoft.Graphics.Canvas.native.h:

// COM interface for external, Win2D-compatible effects
class __declspec(uuid("E042D1F7-F9AD-4479-A713-67627EA31863"))
ICanvasImageInterop : public IUnknown
{
public:
    IFACEMETHOD(GetDevice)(ICanvasDevice** device) = 0;

    IFACEMETHOD(GetD2DImage)(
        ICanvasDevice* device,
        ID2D1DeviceContext* deviceContext,
        GetD2DImageFlags flags,
        float targetDpi,
        float* realizeDpi,
        ID2D1Image** ppImage) = 0;
};

// Options for fine-tuning the behavior of ICanvasImageInterop::GetD2DImage.
typedef enum GetD2DImageFlags
{
    None = 0,
    ReadDpiFromDeviceContext = 1,    // Ignore the targetDpi parameter - read DPI from deviceContext instead
    AlwaysInsertDpiCompensation = 2, // Ignore the targetDpi parameter - always insert DPI compensation
    NeverInsertDpiCompensation = 4,  // Ignore the targetDpi parameter - never insert DPI compensation
    MinimalRealization = 8,          // Do the bare minimum to get back an ID2D1Image - no validation or recursive realization
    AllowNullEffectInputs = 16,      // Allow partially configured effect graphs where some inputs are null
    UnrealizeOnFailure = 32,         // If an input is invalid, unrealize the effect and set the output image to null
} GetD2DImageFlags;

// Enables external objects implementing ICanvasImage to fully implement its APIs
extern "C" __declspec(nothrow, dllexport) HRESULT __stdcall GetBoundsForICanvasImageInterop(
    ICanvasResourceCreator* resourceCreator,
    ICanvasImageInterop* image,
    ABI::Windows::Foundation::Numerics::Matrix3x2 const* transform,
    ABI::Windows::Foundation::Rect* rect);

The reason for these three APIs is:

• GetD2DImage matches the internal ICanvasImageInternal::GetD2DImage (though with a stable COM ABI) and is used to retrieve an ID2D1Image by Win2D APIs receiving an ICanvasImage, so they can then draw them onto a device context.
• GetDevice is necessary when custom ICanvasImage effects implementing this interface are receiving other effects as sources: this way they can QueryInterface those input effects (which might either be Win2D effects or other custom ones) to ICanvasImageInterop and then get the device that was used to realize them, if one is available, to ensure it matches the one the current effect is realized on. This matches what Win2D also does internally for its own effects, and is a necessary extension points to allow custom effects to easily interop with Win2D effects and custom effects as their own inputs.
• GetBoundsForICanvasImageInterop is needed so that external objects can implement the ICanvasImage APIs correctly. To do this, they need'd access to a device context, which can only be retrieved internally. To avoid exposing internal implementation details and to make the code simple, this API allows external objects to instead easily leverage the same exact infrastructure that all images are using internally. They'd just call this, and Win2D would run the same logic as other images.

Usage examples

Implementing a fully custom effect is a bit involved, but a full work in progress extension package to support this in ComputeSharp.D2D1 is available at https://github.com/Sergio0694/ComputeSharp/tree/dev/win2d-effect. This is a UWP-specific package (I will also provide a WinUI 3 one) that depends on ComputeSharp.D2D, which will expose a new PixelShaderEffect<T> effect that will be specifically tailored to support D2D1 pixel shaders authored with ComputeSharp.D2D1. This will give developers the ability to write and use custom pixel shaders extremely easily in their native Windows applications 🚀

Here's a short video showcasing a proof of concept effect powered by ComputeSharp.D2D1.Uwp:

devenv_q6vgFmCPgR.mp4

This runs in a Win2D CanvasAnimatedControl. The same effect also works seamlessly as input for other Win2D effects, as well as using other Win2D effects as its own inputs. This is why both those APIs are needed, to get parity with Win2D's implementation.

Alternative Designs

As mentioned above, it would also be possible to extend PixelShaderEffect, but:

• There is no way to do that to fill all those gaps and also not make breaking changes
• It would still limit how authors would setup a custom effect
• It would only enable D2D1 pixel shader effects, not truly custom D2D effects of any kind

Risks

Relatively low: this only adds a COM API in the public header, and no new WinRT APIs. The internal implementation will keep the same behavior for all existing code, and will only use the new interface as a fallback, when one is available. That is, this should cause no functional changes at all for existing code, and it only affects the few public code paths taking an ICanvasImage object (eg. CanvasDrawingSession.DrawImage).

[Sergio0694]

Changes from this PR were pushed to the internal repo, so closing this one.
The merged changes will be pushed back onto this repository from upstream soon 🚀