Add headless / server mode support to all platforms as a DisplayServer

[akien-mga]

Describe the project you are working on:
The Godot Engine ;)

Describe the problem or limitation you are having in your project:
Historically, Godot's headless (no window/no rendering) platform was only supported on Linux/*BSD via the platform=server port, which was basically a copy of platform=x11 with some changes to allow headless use. It was then extended to also support macOS, albeit still via platform=server (so the same platform code actually covers two platforms).

On Windows, only a --no-window flag is provided to avoid showing a window, but it's not a proper headless mode.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
The hacky server platform should go away, as maintaining a custom platform port for this purpose is overkill.

Now that we split OS from DisplayServer in the master branch, we can implement a headless DisplayServer backend that could be used on all support platform ports (notably Windows, but in theory also Android, iOS, etc.).

Moreover, as this would be registered as an additional DisplayServer backend, it would no longer be necessary to have a custom binary for headless (CI/CD) and server workflows, as the native editor and template builds for the host platforms could be used, with a command line switch to disable window creation and rendering (and possibly automatic fallback on real headless systems that do not support the graphical DisplayServer implementations).

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
We currently have a "Dummy" rasterizer and texture loader implemented in https://github.com/godotengine/godot/tree/master/drivers/dummy for the server platform.

It should be modified to be paired to a headless DisplayServer implementation (possibly implementations if there are platform-specific requirements to handle) that can be enabled on Linux, macOS and Windows via the --display-driver switch (possibly other platforms later on if deemed relevant and technically possible).

The platform/server code should be removed.

Help welcome to flesh out this proposal with more technical details on the implementation. (CC @Faless)

Replaces and supersedes godotengine/godot#11389.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
No.

Is there a reason why this should be core and not an add-on in the asset library?:
This is about platform code, which is core.

[Calinou]

Moreover, as this would be registered as an additional DisplayServer backend, it would no longer be necessary to have a custom binary for headless (CI/CD) and server workflows, as the native editor and template builds for the host platforms could be used, with a command line switch to disable window creation and rendering (and possibly automatic fallback on real headless systems that do not support the graphical DisplayServer implementations).

Sounds good, but is it possible to dlopen() X11 libraries to sidestep the requirement to install them on a server?

[ezdiy]

@Calinou You don't want to do lazy loading by hand (and sadly, ld-linux has no -znolazyload like solaris). Best you can do is external tool that generates dlopen()/dlsym() wrapper library thus simulating MSVC late binding feature.

[akien-mga]

BTW, whoever implements this gets the privilege to git rm -rf platform/server :)
That's 758 deletions(-) (meh, less than I thought), but still good to take (well, remove ;)).

[akien-mga]

Sounds good, but is it possible to dlopen() X11 libraries to sidestep the requirement to install them on a server?

That's an option indeed. The other option which would likely make sense to implement is add a compilation option to disable building the graphical DisplayServers, so you can have a build for any platform with just the headless DisplayServer and therefore no dependency on X11, Wayland, etc. (or equivalent for macOS and Windows).

4.0 should provide enough flexibility so that you can choose at compile time which DisplayServer(s) and which rendering drivers (Vulkan, GLES3, GLES2) you want to compile. The default being to compile all the things of course, but we could still provide a Linux headless and server binaries with only the headless DisplayServer for convenience. (Built from platform/linuxbsd and not from platform/server like now.)

[jordo]

@akien-mga I've implemented a dummy display driver for headless and server support (enabled via --display-driver). Only tested on osx and linux below:

linux-dummy-driver.mov

osx-dummy-display2.mov

The headless driver would just be available on all platforms and can be selected via --display-driver. If nothing is specified it defaults to the existing default platform specific driver.

Last thing to note, is that using this driver will just peg the CPU, as nothing under this driver currently throttles main::iteration(). Currently the dummy display server just noop's swap_buffers() (see below), but I think this would be good place to throttle. I would propose to just link it to the physics tick rate which would be specified in the project, or potentially configurable for a headless tick/process() hz.

void DisplayServerDummy::swap_buffers() {
	// TODO: throttle main here
}

[jordo]

Sorry, i guess I should kinda clarify the posted videos. They just load my test project with a scene that prints "server process". Both are just built for platform=osx, and platform=linuxbsd, and they run headless with the --display-driver "dummy" flag.

[Calinou]

Currently the dummy display server just noop's swap_buffers() (see below), but I think this would be good place to throttle. I would propose to just link it to the physics tick rate which would be specified in the project, or potentially configurable for a headless tick/process() hz.

In 3.2 at least, the Debug > Force Fps project setting should be obeyed in headless mode. You can also set a fixed frame delay in the project settings. See also #2001.

[Faless]

@jordo not sure if this helps, but since godotengine/godot#39994 OSes have a dedicated function to add frame delay, that could be moved to the DisplayServer I guess.