OpenGL + SMAA 4X

[hartmutbehrens]

Dear All,

I've integrated the SMAA S2X and 4X modes for OpenGL. Getting the integration going was relatively easy (just follow the integration notes in SMAA.hlsl) and don't forget to add some missing defines to SMAA.hlsl in the #if defined(SMAA_GLSL_3) || defined(SMAA_GLSL_4) section:

#define SMAATexture2DMS2(tex) sampler2DMS tex
#define SMAALoad(tex, pos, sample) texelFetch(tex, pos, sample)

Once I got it running, the 4X mode didn't look as good as expected compared to T2X - some edges look better, but others look worse. I think the reason for this is the difference in MSAA2x subsample positions between DirectX and OpenGL.

The integration notes in SMAA.hlsl call for the scene to be rendered with D3D10_STANDARD_MULTISAMPLE_PATTERN for SMAA 4X. This allows the subsamples to match the order in the @SUBSAMPLE_INDICES table. When the scene is rendered this way, you have the following subsample positions in DirectX:

       * Sample positions DirectX:
       *   _______
       *  | S1    |  S0:  0.25    -0.25
       *  |       |  S1: -0.25     0.25
       *  |____S0_|
       *

On the other hand, for MSAA2X in OpenGL you get the following subsample positions (positions adjusted for pixel centre that is reported at (0.5,0.5) - see below):

        * Sample positions OpenGL:
        *   _______
        *  |    S0 |  S0:  0.25     0.25
        *  |       |  S1: -0.25    -0.25
        *  |_S1____|
        *

I've been puzzling on how to adjust the subsampleIndices to make SMAA 4X work in OpenGL. Does anyone have some insight here ? In the DX10 demo it is mentioned that the indices have the following layout : Indices layout: indices[4] = { |, --, /, \ }. How should that be interpreted ?

I suspect I also need to adjust the camera jitter, since with the recommended jitter of (0.125, 0.125) and (-0.125, -0.125) for SMAA4X I would end up with a net jitter in OpenGL of:

           *   ________
           *  |      S0|  S0:  0.3750    0.3750
           *  |    S2  |  S1: -0.1250   -0.1250
           *  |  S1    |  S2:  0.1250    0.1250
           *  |S3______|  S3: -0.3750   -0.3750
           *

As a test, I modified the camera jitter offset to use the original, unadjusted subsample positions of MSAA2X instead: (0.75, 0.75) and (0.25, 0.25) (obtained with glGetMultisamplefv, (0.5, 0.5) is the pixel centre). That improved things quite a bit, but I can't help but think I'm still missing something?

[DuncanHopkinsFoundry]

I think using glGetMultisamplefv() is the way to go.

GL_ARB_sample_locations

[TaaTT4]

Hi @hartmutbehrens,

Can you share your integrations of the various SMAA versions?

[RonnieViklund]

Any hints how to do this implementation this in GLSL? I have 4xMSAA tex's I would like try out in combination with SMAA.

[hartmutbehrens]

@RonnieViklund i'm not sure what you are asking for. The README for this repository does say that despite the extension (SMAA.hlsl), the implementation is OpenGL compatible. The notes in SMAA.hlsl are quite descriptive and extensive..

[RonnieViklund]

Well to me it's not clear at all how to implement 4xMSAA into this... not sure what I need to do in the GLSL shader side of things. I have it working with normal sampler2D but not sampler2DMS textures.

[DuncanHopkinsFoundry]

Well to me it's not clear at all how to implement 4xMSAA into this... not sure what I need to do in the GLSL shader side of things. I have it working with normal sampler2D but not sampler2DMS textures.

Hi @RonnieViklund , I am not totally up to speed on the maths inside SMAA, but the general workflow for using 4xMSAA with SMAA, I am guessing would be:

1. Make sure the polygon render target texture is created as a 4xMSAA format.
2. Render into FBO using texture from (1).
3. Attach texture from (1) to SMAA shader input. MSAA and non-MSAA textures need to use sampler2D and sample2DMS in GLSL respectively. If you using the wrong one the glDraw*() normally fails with an error. (This will vary depending on you GPU, platform and driver)
4. Make sure the SMAA reads each of the 4 MSAA samples per pixel and uses them in its maths.

[RonnieViklund]

@DuncanHopkinsFoundry oh I got MSAA working in my engine and my shaders already, I just don't know how to interface my MSAA textures with SMAA.hlsl, I have a working implementation for normal textures that works just fine but for MSAA it seems you need to do modifications that I don't understand.

Just adding
#define SMAATexture2DMS2(tex) sampler2DMS tex #define SMAALoad(tex, pos, sample) texelFetch(tex, pos, sample)
is not enough to just start using MSAA textures.

For instance SMAAColorEdgeDetectionPS uses SMAASamplePoint which expects a normal non-msaa texture, so my normal texture implementation color = vec4(SMAAColorEdgeDetectionPS(texcoord, offset, Texture0)); wouldn't work if Texture0 is sampler2DMS.

I am probably missing something vital that's why I'm asking.

[hartmutbehrens]

Hey @RonnieViklund I did a bit of reading in SMAA.hlsl (while waiting for THAT SpaceX rocket to launch :) ). To answer your question RE SMAAColorEdgeDetectionPS and using MSAA textures

• for SMAA1x and SMAAT2X, just downsample the frame texture
• for SMAAS2X and SMAA4X, the file says the following:

A shader must be run to output each subsample into a separate buffer
 *    (DX10 is required). You can use SMAASeparate for this purpose, or just do
 *    it in an existing pass (for example, in the tone mapping pass, which has
 *    the advantage of feeding tone mapped subsamples to SMAA, which will yield
 *    better results).

So essentially you need to separate the 2DMS texture into two separate 2D textures. There is a function in SMAA.hlsl that does that for you: have a look at the SMAASeparatePS function.

[turol]

I wrote a demo which uses SMAA (all variants) in both Vulkan and OpenGL
https://github.com/turol/smaaDemo

[RonnieViklund]

Hi guys, thanks everyone for you replies, much appreciated!

A thought occurred to me that haven't occurred to me before, since SMAA is basically like last step in the render pipeline perhaps I can just blit the buffers to resolve the MSAA and save myself a lot of headache since I already have working SMAA for sampler2D and I'm gonna blit it afterwards anyway because this is a VR application and you need to resolve the MSAA before you render to each eye, I will test this in the weekend and report my results.

[turol]

I don't remember where it was but either FXAA or SMAA documentation discourages combining it with naive MSAA. The edge detection doesn't work well in that case and gives bad results.

[RonnieViklund]

An update, blitting the MSAA FBO to a Non-MSAA FBO to resolve it before doing the SMAA passes worked great for me, no need to change anything in my shaders.