Add OpenGL Core renderer and use as default #5655

smoogipoo · 2023-02-21T05:34:06Z

Split out from/improves upon #5647

Abstract

In moving to the new Veldrid renderer, we need to move to using the OpenGL core profile. The primary reason is macOS (on M1 in particular) which does not support the ARB_uniform_buffer_object extension, so we couldn't fallback to OpenGL if anything goes wrong with Metal. This extension is mainlined in the OpenGL core profile.

This has been done by using SPIRV-Cross, which is included with Veldrid, to cross-compile from "Vulkan GLSL" to GLSL. Quite a few changes to shaders are required as a result, not in the least helped by the fact that we've been using a long-deprecated version of GLSL until now, but at least the shaders will be forward compatible with Vulkan/Metal/HLSL/etc...

Breaking shader changes have been documented below.

Testing

Tested:

vNext

The OpenGL Core profile is now used

Shaders now follow "Vulkan GLSL", which is mostly documented in the spec: GL_KHR_vulkan_glsl. The primary differences are documented below:

Vertex shader members

old	new
attribute lowp int In_Value; varying lowp int Out_Value;	layout(location = 0) in lowp int InValue; layout(location = 0) out lowp int OutValue;

The location increases for each additional member in the respective in/out lists.

Fragment shader members

old	new
varying lowp int In_Value;	layout(location = 0) in lowp int InValue;

The location increases for each additional member, and must match the location of the respective member in the vertex shader.

Fragment shader outputs

Fragment shaders are required to define an output variable.

old	new
void main(void) { gl_FragColor = vec4(0.0); }	layout(location = 0) out vec4 colour; void main(void) { colour = vec4(0.0); }

Uniform members

Free-floating "global" uniforms are not supported. They must be placed in "uniform blocks".

old	new
uniform lowp int Uniform_Value;	layout(std140, set = 0, binding = 0) uniform MyUniforms { lowp int Uniform_Value; };

Notes:

std140 should always be used.
The set increases for each unique uniform block.
binding is a special required value which indicates where in the program the block should be bound. osu!framework manages this for you and should be 0 for uniform blocks.
Both shader stages can use the same uniform block from the same set, provided that the physical layout matches.

Furthermore, this means that uniform blocks have become a first-class citizen in osu!framework. The code to make use of the above looks like this:

class MyDrawNode : DrawNode
{
    private IUniformBuffer<MyBufferData>? bufferData;

    public override void Draw(IRenderer renderer)
    {
        base.Draw(renderer);

        // Create the uniform buffer.
        bufferData ??= renderer.CreateUniformBuffer<MyBufferData>();

        // Update the uniform buffer.
        bufferData.Data = bufferData.Data with
        {
            UniformValue = 5
        };

        // Bind the buffer to the shader.
        shader.BindUniformBuffer("MyUniforms", bufferData);

        shader.Bind();
        // Draw...
    }

    protected override void Dispose(bool isDisposing)
    {
        base.Dispose(isDisposing);

        // Ensure that the buffer is disposed.
        bufferData?.Dispose();
    }
}

// Pack = 1 is required.
[StructLayout(LayoutKind.Sequential, Pack = 1)]
// Record structs automatically implement IEquatable<>
public record struct MyBufferData
{
    // Use types provided in the osu.Framework.Graphics.Shaders.Types namespace.
    public UniformInt UniformValue;

    // The struct must be a multiple of 16 bytes.
    private readonly UniformPadding12 pad2;
}

Runtime validation is present to notify of incorrect alignment/packing.

Uniform textures/samplers

The combined sampler of GLSL is not supported.

old	new
uniform lowp sampler2D Sampler;	layout(set = 0, binding = 0) uniform lowp texture2D Texture; layout(set = 0, binding = 1) uniform lowp sampler Sampler;

Notes:

std140 cannot be applied here unlike for uniform blocks.
The set increases along with all other uniform blocks/textures.
The texture and sampler are bound to different points. The texture being bound to 0 and sampler to 1 will be standard for osu!framework going forward.

Sampling textures

old	new
vec4 col = texture2D(my_Sampler, ...);	vec4 col = texture(sampler2D(my_Texture, my_Sampler), ...);

peppy · 2023-02-21T06:07:29Z

iOS startup crash:

osu.Framework.Graphics.OpenGL.Shaders.GLShader+PartCompilationFailedException: A osu.Framework.Graphics.OpenGL.Shaders.GLShaderPart failed to compile: sh_Texture2D.vs:
ERROR: 0:1: '' :  version '330' is not supported
ERROR: 0:1: '' : syntax error: #version
   at osu.Framework.Graphics.OpenGL.Shaders.GLShaderPart.Compile() in /Users/dean/Projects/osu-framework/osu.Framework/Graphics/OpenGL/Shaders/GLShaderPart.cs:line 151
   at osu.Framework.Graphics.OpenGL.Shaders.GLShader.CompileInternal() in /Users/dean/Projects/osu-framework/osu.Framework/Graphics/OpenGL/Shaders/GLShader.cs:line 124
   at osu.Framework.Graphics.OpenGL.Shaders.GLShader.compile() in /Users/dean/Projects/osu-framework/osu.Framework/Graphics/OpenGL/Shaders/GLShader.cs:line 62
   at osu.Framework.Threading.ScheduledDelegate.InvokeTask() in /Users/dean/Projects/osu-framework/osu.Framework/Threading/ScheduledDelegate.cs:line 106
   at osu.Framework.Threading.ScheduledDelegate.RunTaskInternal() in /Users/dean/Projects/osu-framework/osu.Framework/Threading/ScheduledDelegate.cs:line 92
   at osu.Framework.Threading.ScheduledDelegate.RunTask() in /Users/dean/Projects/osu-framework/osu.Framework/Threading/ScheduledDelegate.cs:line 76
   at osu.Framework.Graphics.Rendering.Renderer.BeginFrame(Vector2 windowSize) in /Users/dean/Projects/osu-framework/osu.Framework/Graphics/Rendering/Renderer.cs:line 185
   at osu.Framework.Graphics.OpenGL.GLRenderer.BeginFrame(Vector2 windowSize) in /Users/dean/Projects/osu-framework/osu.Framework/Graphics/OpenGL/GLRenderer.cs:line 86
   at osu.Framework.Graphics.Rendering.Renderer.osu.Framework.Graphics.Rendering.IRenderer.BeginFrame(Vector2 windowSize) in /Users/dean/Projects/osu-framework/osu.Framework/Graphics/Rendering/Renderer.cs:line 1067
   at osu.Framework.Platform.GameHost.DrawFrame() in /Users/dean/Projects/osu-framework/osu.Framework/Platform/GameHost.cs:line 482
   at osu.Framework.Threading.GameThread.processFrame() in /Users/dean/Projects/osu-framework/osu.Framework/Threading/GameThread.cs:line 451

Android:

osu.Framework.Graphics.OpenGL.Shaders.GLShader+PartCompilationFailedException: A osu.Framework.Graphics.OpenGL.Shaders.GLShaderPart failed to compile: sh_Texture2D.vs:
0:60: L0001: Storage qualifier not allowed
   at osu.Framework.Graphics.OpenGL.Shaders.GLShaderPart.Compile()
   at osu.Framework.Graphics.OpenGL.Shaders.GLShader.CompileInternal()
   at osu.Framework.Graphics.OpenGL.Shaders.GLShader.compile()
   at osu.Framework.Threading.ScheduledDelegate.InvokeTask()
   at osu.Framework.Threading.ScheduledDelegate.RunTaskInternal()
   at osu.Framework.Threading.ScheduledDelegate.RunTask()
   at osu.Framework.Graphics.Rendering.Renderer.BeginFrame(Vector2 windowSize)
   at osu.Framework.Graphics.OpenGL.GLRenderer.BeginFrame(Vector2 windowSize)
   at osu.Framework.Graphics.OpenGLCore.GLCoreRenderer.BeginFrame(Vector2 windowSize)
   at osu.Framework.Graphics.Rendering.Renderer.osu.Framework.Graphics.Rendering.IRenderer.BeginFrame(Vector2 windowSize)
   at osu.Framework.Platform.GameHost.DrawFrame()
   at osu.Framework.Threading.GameThread.processFrame()
--- End of stack trace from previous location ---
   at osu.Framework.Platform.GameHost.<>c__DisplayClass133_0.<abortExecutionFromException>b__0()
   at osu.Framework.Threading.ScheduledDelegate.InvokeTask()
   at osu.Framework.Threading.ScheduledDelegate.RunTaskInternal()
   at osu.Framework.Threading.Scheduler.Update()
   at osu.Framework.Threading.GameThread.processFrame()
   at osu.Framework.Threading.GameThread.RunSingleFrame()
   at osu.Framework.Platform.ThreadRunner.RunMainLoop()
   at osu.Framework.Platform.GameHost.windowUpdate()
   at osu.Framework.Platform.GameHost.<Run>b__155_2(Object _, FrameEventArgs _)
   at osuTK.GameViewBase.OnUpdateFrame(FrameEventArgs e)
   at osuTK.Android.AndroidGameView.UpdateFrameInternal(FrameEventArgs e)
   at osuTK.Android.AndroidGameView.RunIteration(CancellationToken token)
   at osuTK.Android.AndroidGameView.<>c__DisplayClass58_0.<StartThread>b__2(Object _)
   at Android.App.SyncContext.<>c__DisplayClass3_0.<Send>b__0()
   at Java.Lang.Thread.RunnableImplementor.Run()
   at Java.Lang.IRunnableInvoker.n_Run(IntPtr jnienv, IntPtr native__this)
   at Android.Runtime.JNINativeWrapper.Wrap_JniMarshal_PP_V(_JniMarshal_PP_V callback, IntPtr jnienv, IntPtr klazz)

smoogipoo · 2023-02-21T06:35:40Z

I'm not entirely sure how make iOS use the core profile since it still uses osuTK 😅 But GL <-> Veldrid needs a compatibility layer anyway (basically some shader macros) so I'll add one in for compat profile until we figure out how to get iOS on the core, or switch away from osuTK (will be required anyway to use Metal).

peppy · 2023-02-21T06:39:30Z

FWIW android runs framework visual tests fine with this patch applied:

diff --git a/osu.Framework/Resources/Shaders/sh_Precision_Internal.h b/osu.Framework/Resources/Shaders/sh_Precision_Internal.h
index 265b627a6..df6e57da5 100644
--- a/osu.Framework/Resources/Shaders/sh_Precision_Internal.h
+++ b/osu.Framework/Resources/Shaders/sh_Precision_Internal.h
@@ -1,6 +1,6 @@
 // This file is automatically included in every shader
 
-#version 330 core
+#version 300 es
 
 #ifndef GL_ES
     #define lowp

iOS not so much.

peppy · 2023-02-21T06:45:58Z

osu.Framework/Graphics/OpenGLCore/Buffers/GLCoreVertexBuffer.cs

+        {
+            ref var currentVertex = ref getMemory().Span[vertexIndex];
+
+            bool isNewVertex = !currentVertex.Vertex.Equals(vertex) || currentVertex.BackbufferDrawDepth != Renderer.BackbufferDrawDepth;


Maybe a bit late / the wrong PR, but we should probably be using BackBuffer in naming.

peppy · 2023-02-21T09:45:18Z

Confirmed latest commit is working on iOS / android (framework visual tests).

bdach · 2023-03-01T20:38:12Z

I was under the presumption that .NET 6 Android TFM is backward-compatible with MonoAndroid in that aspect, at least according to https://github.com/dotnet/designs/blob/main/accepted/2021/net6.0-tfms/net6.0-tfms.md#compatibility-rule

I'm really not sure why it isn't in this case. My naive guess is that currently in Veldrid.SPIRV artifacts, the android binaries are placed inside the build\MonoAndroid10\native directory of the nupkg. So it looks like it's potentially treated as a build dependency, steered by a .targets file, and thus that could not work properly with net6.0, because in the end framework packages are being built for another target framework entirely? But that's only a naive guess because google gives me nothing useful (and it's difficult to figure out a phrase to search for anyways). Unsure if it's worth the time investigating why it doesn't work as-is.

smoogipoo · 2023-03-06T10:35:08Z

I've changed this to use environment variables for the graphics renderer/surface selection for two reasons: 1) I don't want to update every project to pass parameters via HostOptions, and 2) I'm not ready to commit to a structure for HostOptions since I know some people want to create custom-typed renderers.

peppy · 2023-03-07T09:34:40Z

Are we still blocked on android here?

frenzibyte · 2023-03-07T10:01:44Z

Yeah, Android needs a workaround for the library missing exception. Simplest way would be by copying the native library from the upstream repo and placing it inside osu.Framework.Android alongside other native libraries.

frenzibyte · 2023-03-07T17:32:40Z

I've took a look at the approach I mentioned above, but it turns out I can't test it well because I apparently can't reproduce the issue in the first place (testing with emulator). The commit is c3cca77 if anyone wants to test it or cherry-pick it here.

I've copied the ARM ones directly from the NuGet package folder, but the x86 one was generated manually since the package was providing x86_64 instead.

MaxKruse · 2023-03-07T21:55:07Z

Using opengl-core-render branches on all relevant projects, + cherry-picking c3cca77 for the framework still results in the same errors mentioned above ( #5655 (comment) ) . I have double checked that the files actually exist, but they dont seem to be available/packaged to the binary in the end.

This is on actual hardware once again.

frenzibyte · 2023-03-08T03:34:54Z

From testing with emulator, no native libraries get updated in the output folder unless bin/obj folders are cleaned before building again.

peppy · 2023-03-08T04:42:48Z

Tested on android hardware to work with 8e090c3.

frenzibyte

Took a slightly quick skim on all commits in this PR and caught some things, mostly minor/quality notes (minus #5655 (comment)).

osu.Framework/FrameworkEnvironment.cs

osu.Framework/Graphics/OpenGL/Textures/GLTexture.cs

osu.Framework/Graphics/Rendering/Renderer.cs

frenzibyte · 2023-03-08T05:55:13Z

osu.Framework/Graphics/OpenGL/Shaders/GLShader.cs


            int blockBindingIndex = 0;
            int textureIndex = 0;

-            for (int i = 0; i < uniformCount; i++)
+            foreach (ResourceLayoutDescription layout in crossCompileResult.Reflection.ResourceLayouts)


With this change, Uniforms is no longer being populated, causing TestSceneTextureUnit to crash. Uniforms is likely to be removed any time now, but not sure how the crashing test scene would be fixed.

Uniforms is to be removed, yeah. As for the test scene, it can't really be done that way anymore. You'd have to bind each subsequent texture to unit 0 and draw with that unit.

I think there's no reason for that test scene to exist as TestSceneVideo already does multiple texturing. I've removed it.

frenzibyte · 2023-03-08T05:58:19Z

osu.Framework/Platform/Windows/WindowsGameHost.cs

+        protected override IRenderer CreateRenderer()
+        {
+            if (FrameworkEnvironment.PREFERRED_GRAPHICS_RENDERER != null || FrameworkEnvironment.PREFERRED_GRAPHICS_SURFACE != null)
+                return base.CreateRenderer();
+
+            return new WindowsGLRenderer(this);
+        }


This looks like it needs a minor refactor. CreateRenderer should be changed to CreateDefaultRenderer and there should be another factory method that would first check the environment variables and then fall back to the GameHost's default renderer such as the one here. Can be left for another PR though if it embeds too much changes.

That being said, if a user sets environment variables to OpenGL, GLRenderer would be used instead of WindowsGLRenderer, which breaks fullscreen capability detection method, wouldn't it?

Yeah I'm not sure that it's incorrect the way it is right now. Potentially this could also switch on the "gl"/"opengl"/"wingl" renderer types and handle it that way? But I think it's fine for renderer=gl to bypass WindowsGLRenderer for the time being.

Let's leave this as a follow-up change if required.

peppy · 2023-03-10T02:55:24Z

osu.Framework/Graphics/Containers/BufferedContainer_DrawNode.cs


+                    blurParametersBuffer.Data = blurParametersBuffer.Data with


I'm not sold on the readability of with...

More details for others unfamiliar with this new expression.

Also correct me if I'm wrong but this looks to be initialising every property in this case, so I think using new would suffice right?

Also see at least one code inspection where this is to be applied.

Yes, new would suffice here. with is only really that helpful in Renderer itself, where there's a massive buffer that's changed by many methods. I think new is actually usable for most cases other than that one...

Let's go with it for now. I think it makes sense to use with in that case because it will handle a property potentially being added to the record in the future more correctly.

peppy · 2023-03-10T09:13:22Z

This is in as good a state as it will be I think. Have done multiple review passes and tested quite extensively on mobile and macOS. I intend to do some CPU/alloc regression testing on the weekend to make sure nothing has changed too much, but let's get this in as it is and work from there.

smoogipoo added 6 commits February 17, 2023 16:40

Add GL core renderer

22b0f3a

Fix extensions string

188f56a

Fix use of unsupported method on desktop platforms

e221321

Use in/out in shaders

abdc9dd

Remove deprecated constructs from shaders

1764ea7

Remove nullable disables

b58d4cd

pull-request-size bot added the size/XL label Feb 21, 2023

smoogipoo added area:graphics and removed size/XL labels Feb 21, 2023

smoogipoo mentioned this pull request Feb 21, 2023

Update shaders for OpenGL Core ppy/osu-resources#247

Merged

1 task

peppy self-requested a review February 21, 2023 05:39

pull-request-size bot added the size/XL label Feb 21, 2023

peppy reviewed Feb 21, 2023

View reviewed changes

smoogipoo added 4 commits February 21, 2023 16:58

Add GL/GLCore shader compatibility layers

ffe38a3

Update shaders to use macros

ac4f8cb

Add compatibility layer for fragment shader output

6aab647

Add compatibility profile type

02b6b64

pull-request-size bot added size/XXL and removed size/XL labels Feb 21, 2023

Fix compatibility profile on mobile devices

1cace44

frenzibyte self-requested a review February 21, 2023 09:26

bdach closed this Feb 21, 2023

bdach reopened this Feb 21, 2023

smoogipoo added 2 commits February 21, 2023 18:37

Add newlines and fix nullref

8eda0ab

Fix Veldrid nullref

ea4a72e

Add startup-time surface selection

f00a789

smoogipoo added 3 commits March 6, 2023 14:49

Ensure shader is compiled before BindUniformBlock()

eeef6a3

Mode env variables to FrameworkEnvironment

a5bf178

Use env variables for renderer/surface

10b7eec

Unbox nullable

4f691c5

Add SPIRV native libraries for Android

8e090c3

Merge branch 'master' into opengl-core-renderer

f85aef5

frenzibyte reviewed Mar 8, 2023

View reviewed changes

smoogipoo added 4 commits March 9, 2023 22:33

Use PascalCase for env vars

cc19945

Revert change to GL.Hint()

e755c07

Output readable type names to exceptions

7122320

Remove broken/unsupported test scene

e8d2f12

peppy reviewed Mar 10, 2023

View reviewed changes

Change tuple with suggestion to hint level

255d760

peppy self-requested a review March 10, 2023 09:04

Merge branch 'master' into opengl-core-renderer

86c00bc

peppy approved these changes Mar 10, 2023

View reviewed changes

peppy enabled auto-merge March 10, 2023 09:10

peppy merged commit 3aa9920 into ppy:master Mar 10, 2023

frenzibyte mentioned this pull request Mar 10, 2023

HSV colour picker causes GPU-level crash on M1 with OpenGL Core #5674

Closed

smoogipoo deleted the opengl-core-renderer branch September 11, 2023 02:25

Frederisk mentioned this pull request Oct 13, 2023

Game won't launch in Android Emulator: DllNotFoundException: libveldrid-spirv #6028

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add OpenGL Core renderer and use as default #5655

Add OpenGL Core renderer and use as default #5655

smoogipoo commented Feb 21, 2023 •

edited by peppy

Loading

peppy commented Feb 21, 2023 •

edited

Loading

smoogipoo commented Feb 21, 2023 •

edited

Loading

peppy commented Feb 21, 2023

peppy Feb 21, 2023

peppy commented Feb 21, 2023

bdach commented Mar 1, 2023 •

edited

Loading

smoogipoo commented Mar 6, 2023

peppy commented Mar 7, 2023

frenzibyte commented Mar 7, 2023

frenzibyte commented Mar 7, 2023 •

edited

Loading

MaxKruse commented Mar 7, 2023

frenzibyte commented Mar 8, 2023 •

edited

Loading

peppy commented Mar 8, 2023

frenzibyte left a comment •

edited

Loading

frenzibyte Mar 8, 2023

smoogipoo Mar 9, 2023 •

edited

Loading

frenzibyte Mar 8, 2023 •

edited

Loading

smoogipoo Mar 9, 2023 •

edited

Loading

peppy Mar 10, 2023

peppy Mar 10, 2023

peppy Mar 10, 2023

smoogipoo Mar 10, 2023

peppy Mar 10, 2023

peppy commented Mar 10, 2023

Add OpenGL Core renderer and use as default #5655

Add OpenGL Core renderer and use as default #5655

Conversation

smoogipoo commented Feb 21, 2023 • edited by peppy Loading

Abstract

Testing

vNext

The OpenGL Core profile is now used

Vertex shader members

Fragment shader members

Fragment shader outputs

Uniform members

Uniform textures/samplers

Sampling textures

peppy commented Feb 21, 2023 • edited Loading

smoogipoo commented Feb 21, 2023 • edited Loading

peppy commented Feb 21, 2023

peppy Feb 21, 2023

Choose a reason for hiding this comment

peppy commented Feb 21, 2023

bdach commented Mar 1, 2023 • edited Loading

smoogipoo commented Mar 6, 2023

peppy commented Mar 7, 2023

frenzibyte commented Mar 7, 2023

frenzibyte commented Mar 7, 2023 • edited Loading

MaxKruse commented Mar 7, 2023

frenzibyte commented Mar 8, 2023 • edited Loading

peppy commented Mar 8, 2023

frenzibyte left a comment • edited Loading

Choose a reason for hiding this comment

frenzibyte Mar 8, 2023

Choose a reason for hiding this comment

smoogipoo Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

frenzibyte Mar 8, 2023 • edited Loading

Choose a reason for hiding this comment

smoogipoo Mar 9, 2023 • edited Loading

Choose a reason for hiding this comment

peppy Mar 10, 2023

Choose a reason for hiding this comment

peppy Mar 10, 2023

Choose a reason for hiding this comment

peppy Mar 10, 2023

Choose a reason for hiding this comment

smoogipoo Mar 10, 2023

Choose a reason for hiding this comment

peppy Mar 10, 2023

Choose a reason for hiding this comment

peppy commented Mar 10, 2023

smoogipoo commented Feb 21, 2023 •

edited by peppy

Loading

peppy commented Feb 21, 2023 •

edited

Loading

smoogipoo commented Feb 21, 2023 •

edited

Loading

bdach commented Mar 1, 2023 •

edited

Loading

frenzibyte commented Mar 7, 2023 •

edited

Loading

frenzibyte commented Mar 8, 2023 •

edited

Loading

frenzibyte left a comment •

edited

Loading

smoogipoo Mar 9, 2023 •

edited

Loading

frenzibyte Mar 8, 2023 •

edited

Loading

smoogipoo Mar 9, 2023 •

edited

Loading