OpenGL Renderer

[Kethku]

I recently swapped to an OpenGL backend away from the vulkan one with the hope that it would increase linux compatibility. However it seems that some folks are having issues.

I don't have the tools or know-how required to fix these problems on my own.

The opengl binding is handled via https://github.com/rust-windowing/glutin and is done here: https://github.com/Kethku/neovide/blob/main/src/window/window_wrapper/renderer.rs which was based in part on alacritty's implementation https://github.com/alacritty/alacritty/tree/master/alacritty/src/display and the sample in skia-safe https://github.com/rust-skia/rust-skia/blob/master/skia-safe/examples/gl-window/main.rs.

If anybody is able to synthesize something that works more consistently that would be amazing.

NOTE: I reverted the main change as it wasn't ready for prime time like I thought. The code is currently held in the OpenGL branch

[kiuKisas]

❯ ./neovide
Ignored client type property: "methods"
Ignored client type property: "attributes"
fish: “./neovide” terminated by signal SIGSEGV (Address boundary error

Arch Linux, build from: 7c18e92

[calvinkosmatka]

Building from 4aec0d5

I was able to at least get a window open on my old X201T, Ubuntu 20.04, sway wm, wayland-only with a little massaging.

1. A couple dependency issues - typeface-open-stream was merged upstream, so I had to change the skia-safe url to git = "https://github.com/rust-skia/rust-skia". flexi_logger was causing a version conflix for regex so I switched it to flexi_logger = { version = "0.15", default-features = false }.
2. I didn't have the consolas font, so had to switch the fallback to a font I had installed (not really an issue)
3. Had to comment out .with_double_buffer(Some(true)) at src/window/window_wrapper/mod.rs L425 to get a window to open, otherise it would panic with the message thread 'main' panicked at 'called 'Result::unwrap()' on an 'Err' value: NoAvailablePixelFormat', src/window/window_wrapper/mod.rs:430:10. This might just be a quirk of my hardware, but others might hit this as well. Also tried running with LIBGL_ALWAYS_SOFTWARE=1 (which I need to make Alacritty run on this machine), made no difference.
4. I can't type most special characters, only @ and :. Looks like the issue is that glutin WindowEvent::KeyboardInput events already take the modifier state into account and they don't have virtual_keycodes for most special characters. I really want to get this running on this machine so I'd be willing to take a crack a refactoring keyboard input to make this work

[Kethku]

This gives traction. I will try these things. I didn't realize the double buffer could be the issue.

[Kethku]

I built a Ubuntu install on an extra harddrive explicitly for testing this stuff out. The experience was predictably mixed.

At the moment I ran into a segmentation fault on run. After running in lldb, I got this:

* thread #1, name = 'neovide', stop reason = signal SIGSEGV: invalid address (fault address: 0xfffffffffffffc68)
  * frame #0: 0x0000555556620b1e neovide`eat_space_sep_strings(SkTArray<SkString, false>*, char const*) + 46
    frame #1: 0x0000555556620851 neovide`GrGLExtensions::init(GrGLStandard, GrGLFunction<unsigned char const* (unsigned int)>, GrGLFunction<unsigned char const* (unsigned int, unsigned int)>, GrGLFunction<void (unsigned int, int*)>, GrGLFunction<char const* (void*, int)>, void*) + 577
    frame #2: 0x00005555569f5e5b neovide`GrGLMakeAssembledGLInterface(void*, void (* (*)(void*, char const*))()) + 475
    frame #3: 0x0000555556933330 neovide`GrGLMakeAssembledInterface(void*, void (* (*)(void*, char const*))()) + 224
    frame #4: 0x00005555567261ae neovide`::C_GrGLInterface_MakeAssembledInterface(ctx=0x00007fffffffa9a8, get=(neovide`skia_safe::gpu::gl::interface::gl_get_proc_fn_wrapper::h359798421d1ad615 at interface.rs:32)) at gl.cpp:147:38
    frame #5: 0x0000555555739595 neovide`skia_safe::gpu::gl::interface::_$LT$impl$u20$skia_safe..prelude..RCHandle$LT$skia_bindings..bindings..GrGLInterface$GT$$GT$::new_load_with::h10453dd3aeb4a6b8(loadfn=closure-1 @ 0x00007fffffffa9a8) at interface.rs:24:13
    frame #6: 0x00005555556fe4ee neovide`neovide::window::window_wrapper::renderer::SkiaRenderer::new::hfdf2e1af3baa2074(windowed_context=0x00007fffffffc0f0) at renderer.rs:53:25
    frame #7: 0x00005555557b3a1e neovide`neovide::window::window_wrapper::start_loop::h2e07a824b49b4e33(window_command_receiver=Receiver<neovide::editor::WindowCommand> @ 0x00007fffffffabb0, ui_command_sender=TxBlocking<neovide::bridge::ui_commands::UiCommand, crossfire::mpsc::unbounded::UnboundedSharedFuture> @ 0x00007fffffffd5f0, running=Arc<core::sync::atomic::AtomicBool> @ 0x00007fffffffabc0, logical_size=(__0 = 804, __1 = 801), renderer=Renderer @ 0x00007fffffffd610) at mod.rs:435:25
    frame #8: 0x000055555576f70d neovide`neovide::window::create_window::he86e9e84cff00986(batched_draw_command_receiver=Receiver<alloc::vec::Vec<neovide::editor::DrawCommand, alloc::alloc::Global>> @ 0x00007fffffffd320, window_command_receiver=Receiver<neovide::editor::WindowCommand> @ 0x00007fffffffd330, ui_command_sender=TxBlocking<neovide::bridge::ui_commands::UiCommand, crossfire::mpsc::unbounded::UnboundedSharedFuture> @ 0x00007fffffffdd50, running=Arc<core::sync::atomic::AtomicBool> @ 0x00007fffffffd340) at mod.rs:106:5
    frame #9: 0x00005555558a8538 neovide`neovide::main::h8a34d2494d865a2e at main.rs:179:5
    frame #10: 0x000055555571767b neovide`core::ops::function::FnOnce::call_once::hb6835360d127917d((null)=(neovide`neovide::main::h8a34d2494d865a2e at main.rs:34), (null)=<unavailable>) at function.rs:227:5
    frame #11: 0x000055555570192e neovide`std::sys_common::backtrace::__rust_begin_short_backtrace::h98ed3d2a87357883(f=(neovide`neovide::main::h8a34d2494d865a2e at main.rs:34)) at backtrace.rs:125:18
    frame #12: 0x00005555556cb001 neovide`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h8129eb8637f80ccd at rt.rs:66:18
    frame #13: 0x0000555556a43b2a neovide`std::rt::lang_start_internal::h965c28c9ce06ee73 [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::hbcc915e668c7ca11 at function.rs:259:13
    frame #14: 0x0000555556a43b23 neovide`std::rt::lang_start_internal::h965c28c9ce06ee73 [inlined] std::panicking::try::do_call::h6b0f430d48122ddf at panicking.rs:379
    frame #15: 0x0000555556a43b23 neovide`std::rt::lang_start_internal::h965c28c9ce06ee73 [inlined] std::panicking::try::h6ba420e2e21b5afa at panicking.rs:343
    frame #16: 0x0000555556a43b23 neovide`std::rt::lang_start_internal::h965c28c9ce06ee73 [inlined] std::panic::catch_unwind::h8366719d1f615eee at panic.rs:431
    frame #17: 0x0000555556a43b23 neovide`std::rt::lang_start_internal::h965c28c9ce06ee73 at rt.rs:51
    frame #18: 0x00005555556cafe0 neovide`std::rt::lang_start::ha74f856e07cf29c4(main=(neovide`neovide::main::h8a34d2494d865a2e at main.rs:34), argc=1, argv=0x00007fffffffe068) at rt.rs:65:5
    frame #19: 0x00005555558a888c neovide`main + 28
    frame #20: 0x00007ffff795e0b3 libc.so.6`__libc_start_main + 243
    frame #21: 0x00005555556b909e neovide`_start + 46

Which says to me that running is failing at creating the interface here: https://github.com/Kethku/neovide/blob/opengl/src/window/window_wrapper/renderer.rs#L53

@calvinkosmatka did you install any particular drivers? I'm running with an rtx 3070, so maybe driver support is the issue given that the gpu is relatively new? I installed the nvidia drivers, because I couldn't get multiple monitors to display with the default ones.

[Kethku]

Weirdly I get passed that segmentation fault when run in the rr debugger. Not sure what thats about...
Edit: nope egl fails before, so I think rr screws something else up and was a distraction. The real problem is constructing the interface as mentioned in the above comment

[Kethku]

I pushed changes matching your suggestions (including some default font fixes).

[kiuKisas]

It seems to work ! I gonna give a try today and let you know if I have any issue :)

[kiuKisas]

I didn't have any issue during the all day, I guess it's fine now :)

[Kethku]

@kiuKisas were you running from the opengl branch? I reverted the opengl change on main because it was causing problems.

[calvinkosmatka]

@Kethku the machine I'm testing this on just has intel integrated graphics (I don't recall ever installing any special drivers, if I did it would have been 8ish years ago). I have yet to hit a SIGSEV.

I did a little more digging into my keyboard input issues - looks like it's (mostly) an upstream issue in winit/glutin and there's not much hope of solving it on the Neovide end (see rust-windowing/winit#1806). Winit just don't send a bunch of virtual keycodes on Wayland/X11 (didn't check other platforms). Good news is I think this would give Neovide non-qwerty support basically for free once this is fixed.

[Kethku]

I'm watching rust-windowing/winit#1788 as my hope for revamping the text input. In the meantime, we could likely create another keyboard layout like this: https://github.com/Kethku/neovide/blob/main/src/window/winit/layouts/qwerty.rs for whatever layout you need. This would fix your issues in the short run.

[calvinkosmatka]

Ooh that branch looks promising. Have you tried building against it? I'm going to give it a shot right now.

I would need to change the API a bit to make a new layout work since I'd have to rely on scancodes - think I'd rather look towards a more permanent solution

[Kethku]

I have not tried building against it, but its gone through extensive review and appears to be getting close

[Kethku]

rust-windowing/winit#1888 this is based on that same PR and indicates effort to port it to other platforms as well which makes me more confident it will land in the not too distant future.

[jdu]

Is it worth maybe looking at gfx-hal or even wgpu-rs as a backend to make portability across backends (vulkan, DX, opengl, metal, webgl) a little easier? gfx-hal and wgpu-rs seem to have a lot of weight behind them and are ticking along pretty quickly.

I've been trying to see if there's a way to replicate what skupin was doing by sort of shimming skia into the vulkan backend, but with gfx-hal, but i'm not as experienced with graphics and the render pipeline to see how to do that (though i'm not quitting), but it might yield a bit more consistency for the backend and allow compiling for the different backends more easily.

[Kethku]

Theres two strategies in my mind:

1. somehow get skia to use gfx-hal or wgpu-rs as a backend
2. build a replacement for skia which uses wgpu-rs

For option 1 I have no idea how feasible that should be. We may be able to get some support for such a thing by talking to the maintainer of skia-safe. They have way more knowledge about Skia's internals than I do and may have some advice for how to approach the problem.

For option 2 we attempted to create something like that in the past based on wgpu. @jonvaldes advised some work on it before I integrated the blurred backgrounds. A significant part of me would really like to have that level of control of the renderer system especially since it would push a significant source of non rust code out. Further, building skia in its entirety makes build dependencies more complex and increases the weight of the resulting app.

That said, skia takes care of a lot of hard problems for us. The big ones in my mind are:

• snapshotted images, so we don't have to rerender the entire window every change
• blurred window backgrounds using an easy to use and understand api
• good font rendering (including upcoming subpixel aliasing support)

Since I have been working on the subpixel font rendering problem a bit, I understand more of the requirements, and I don't think its an impossible problem to tackle, but it would push it into Neovide's responsibility in a way that its not today.

Edits: wording

[Kethku]

All of that is to say, I'm very interested in proposals for making driver support easier for end users. And I would LOVE a solution that increases our flexibility or simplifies the complexity of the code we have to manage.

Skia is a simplifying force for this app which takes care of a lot of the hard rendering problems so that I (and contributors) can focus on the hard problem of neovim compatibility. As neovim compatibility increasingly becomes a solved problem in neovide, we could potentially turn our focus toward replacing skia. But it does look like a long road.

It occurs to me that an interesting solution to this problem would be to abstract neovide's renderer into its own library for drawing terminal like applications in rust. Such a library could then expose a high level api which neovide could depend on. Then any progress in this area could be made in that library without increasing the complexity of neovide itself. Plus such a library would likely have uses outside of just neovim. If anybody finds such a proposal interesting, I would be very interested in discussing it more...

[jdu]

I'm currently playing around looking at how skulpin impements its renderer to see if I can approximate something similar with gfx-hal. I have a couple projects that i'd like to use skia-safe as the drawing layer, but I'd rather have the more fine-grained control over the graphics pipeline similar to how skulpin is shimming skia into the rafx backend for vulkan.

I have a working gfx-hal implementation using winit and building and working for a basic draw with metal, vulkan and DX12 backends that I got working today. Just now fiddling with seeing how I can shim in skia-safe and attach its context to the gfx-hal components like the queue and the pipeline.

It's tricky because i'm not used to the material_pass rendering that skulpin/skia-safe is using with the texturing, I can get a triangle to draw alright using basic vert/frag shaders but there's a couple things I need to learn in here in order to get skia-safe playing nice with gfx-hal.

I'd really rather not implement a whole drawing backend, and would rather use skia-safe as well, especially with the text layout stuff as that stuff is painfully hard to implement. So i'm keen to get skia working.

I'll keep you updated on where I get with it and if I do get somewhere I'll see if I can lay out what I had to do to get it to work along with a minimal example drawing a rect and some text from skia.

[Kethku]

I would greatly appreciate it! I think it would improve neovide significantly so I'm all ears.

[calvinkosmatka]

I've gotten this to the point where I can actually edit Neovide using opengl Neovide! So one more issue I've discovered is that the cursor starts to not line up with the text as you get further along in the line. Using font.measure_str to calculate font width in caching_shaper.rs, the cursor position tends to lag behind the grid text. I tried swapping this out for:

let bounds = self.shape_cached(STANDARD_CHARACTER_STRING, false, false).first().expect("").bounds();
let text_width = bounds.right - bounds.left;

but then the text lagged behind the cursor. I've tried with a handful of different fonts with the same results. Is there a better way to get skia to tell us "how big will this text be when rendered"?

[Kethku]

@calvinkosmatka if you divide the text_width by the number of characters in STANDARD_CHARACTER_STRING does that work better? Its also possible that the shaped blob is scaled by some known factor. I have a ubuntu install finally running and working, so I should be able to test it more myself.

Do you have a branch with any changes you had to make? Do you know if there were any specific dependencies you needed to get it running?

[Kethku]

I've also been thinking about the text layout approach we've been taking. Generally its pretty dumb because we depend on the shaper to layout what should be a monospace font. The only reason we use the shaper at all is because I need to know what glyph substitutions are necessary to handle ligatures. However if we could use the shaper just for ligatures and then position the glyphs manually our selves based on some estimate (such as the size of "Z") then we wouldn't ever run into this cursor to text positioning mismatch. It would always be positioned the same way every time.

The opengl branch goes part of the way towards doing this by using Rustybuzz instead of skribo for the harfbuzz binding. This way we have a nice api on top of actual harfbuzz rather than a higher level wrapper. I'm struggling to understand how glyphs line up to the original text codepoints though. My understanding is that this is done with clusters, but the documentation is somewhat dense https://harfbuzz.github.io/working-with-harfbuzz-clusters.html and my manual tests get weird results such as skipped cluster indexes when I don't expect it. So I took a break fiddling with it to regroup.

TLDR
To solve these types of problems once and for all, we need some function which when given a string returns a list of glyph ids with matching unicode characters from that string. From that function would could then use skia's font fallback to fill in any gaps, and could draw the glyphs in the correct position assuming they are monospace. If anybody knows anything about how that my work, I'm all ears as it would solve a bunch of problems in neovide today

[AckslD]

Probably the same issue but I just installed neovide-git from AUR and got the following error when running:

> neovide
Ignored client type property: "methods"
Ignored client type property: "attributes"
thread 'main' panicked at 'Failed to create window: SdlError("Installed Vulkan doesn\'t implement the VK_KHR_surface extension")', src/window/sdl2/mod.rs:442:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

EDIT: same error with neovide from AUR.

[AckslD]

Thanks @Kethku! What dependencies does that branch have? I guess I'm missing something since I get

> neovide
Ignored client type property: "methods"
Ignored client type property: "attributes"
thread 'main' panicked at 'Failed to create renderer: CreateInstanceError(InstanceError(VkError(ERROR_EXTENSION_NOT_PRESENT)))', src/window/sdl2/mod.rs:458:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace 

[proudmuslim-dev]

I'm getting the same error as the guy above, running Arch on Swaywm (stable)

[Kethku]

Thats very strange because the VkError in your message is from vulkan dependencies. I'm pretty confused... Further window/sdl2 isn't present in the opengl branch https://github.com/Kethku/neovide/tree/opengl/src/window

Are you sure you are actually on the opengl feature branch?

[AckslD]

Sorry, I think that was my bad! Anyway, now I'm on the opengl branch and getting instead the error:

Ignored client type property: "methods"
Ignored client type property: "attributes"
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/renderer/fonts/caching_shaper.rs:59:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[haringsrob]

Hey @Kethku ,

For what its worth, I could build on Mac after removing -resolver = "2" from the cargo.toml file.

It starts up fine, font renders perfect compared to latest main branch.

However, there is some difference in the font sizes:

Cursor is tiny, and floating window as well. vim.o.guifont = "JetBrainsMono Nerd Font" (also tested without font config)

Cursor is below the first namespace line (it highlighs a letter from another line.)

---

I basically have no idea what I am doing but:

in mod.rs (line 376): If I set let scaling = 1.0 everything is rendering fine (but small). Any other float gives bad results.

I had to set a bigger font size (example :h90 but works well with :h36)

[romhml]

Hey @AckslD, I ran into the same issue and solved it by installing the Ubuntu font, which is the fallback font on Linux (pacman -S ttf-ubuntu-font-family). I guess another way to fix the problem would be to define a proper guifont in your vim config.

[romhml]

I took a deeper look into the renderer::font module and noticed that this unwrap also causes neovide to crash when setting an invalid guifont, instead of falling back to the default value.

I don't know if this is relevant to this particular issue, but I think it can be improved by checking that the font is valid / can be found on the system before updating it. Another thing would be to remove that unwrap and implements Errors in this module to give "user-friendly" feedback when something goes wrong.

I can give it a go if needed!

[shaunsingh]

Hello! ive been using the openGl build for a while now and the only issue is that my minimap doesn't show. The minimap im using is: https://github.com/wfxr/minimap.vim. It works fine in the Vulkan build

[haringsrob]

Build the latest OpenGL branch today. All working perfect (m1 Mac), I do need to "resize" the window after launch to get all the things to show up on the correct places.

[shaunsingh]

Ive also noticed some symbols with the openGL branch don't appear. For example, I have some characters set up for the error and warning symbols in ale for listing. In iterm2 it looks like this:

Whereas in neovide openGl it looks like this:

In both cases im using the same font, and I believe it works in the Vulkan branch

[shaunsingh]

additionally, every time a symbol like that comes up, in the terminal I see

Is it something to do with fg_indexed?

edit: not sure if it is related but in the background of iterm you can see a faint fg indexed:

[j4qfrost]

Digging into the segfault, looks like eglGetCurrentDisplay is the issue. I would guess that this is a driver issue, but I have no idea.

[j4qfrost]

@Kethku looks like this little hack bypasses the segfault:

if name == "eglGetCurrentDisplay" {
    return std::ptr::null();
}

You can add into into the closure here https://github.com/Kethku/neovide/blob/opengl/src/window/window_wrapper/renderer.rs#L53 Neovide works, but I am running into a spacing issue. I'll look into a bit more.

[Kethku]

@j4qfrost amazing. You're a life saver. I will try to add this tonight

[j4qfrost]

@Kethku no problem, kind of sucks that I can't figure out why the function pointer is invalid. Maybe someone with better assembly skills can take a look.

[bryce-carson]

I haven't been able to compile or run Neovide on any platform until now; I've tried OS X Catalina (simple "Cannot run this app"; allowing apps from any publisher didn't work. 🤷🏻), Ubuntu, and Fedora before. It's now working (built and opened without error, at least) on Fedora 34 with the following in mind.

Hey @AckslD, I ran into the same issue and solved it by installing the Ubuntu font, which is the fallback font on Linux (pacman -S ttf-ubuntu-font-family). I guess another way to fix the problem would be to define a proper guifont in your vim config.

I am on Fedora, which does not package the whole Ubuntu font family because of the licensed used for the font.

I am going to +1 the above, and note that on Fedora 34 this solution worked:

sed -i 's/Ubuntu/Cantarell/' ./src/renderer/fonts/caching_shaper.rs

P.S.: Woops, don't use Cantarell. It's not monospaced! 🤦🏻

Additionally, @j4qfrost's suggestion worked. It took me a couple tries, as I don't even know C++ let alone Rust, but this allows the project to build. ./src/window/window_wrapper/renderer.rs

        // Lines 53-58 below
        let interface = skia_safe::gpu::gl::Interface::new_load_with(|name| {
            if name == "eglGetCurrentDisplay" {
                return std::ptr::null();
            }
            else { windowed_context.get_proc_address(name) }
        })

I also experience the error with symbols that @calvinkosmatka observed.

[bryce-carson]

I've gotten this to the point where I can actually edit Neovide using opengl Neovide! So one more issue I've discovered is that the cursor starts to not line up with the text as you get further along in the line. Using font.measure_str to calculate font width in caching_shaper.rs, the cursor position tends to lag behind the grid text. I tried swapping this out for:

let bounds = self.shape_cached(STANDARD_CHARACTER_STRING, false, false).first().expect("").bounds();
let text_width = bounds.right - bounds.left;

but then the text lagged behind the cursor. I've tried with a handful of different fonts with the same results. Is there a better way to get skia to tell us "how big will this text be when rendered"?

I observed that as well when using a non-monospace font, Cantarell, but when using avage Nerd Font Mono it was not observed.

[lakshyarao22]

guys i have installed vulkan drivers and its running flawlessly

[Kethku]

I'm glad its working for you! I've come to the conclusion though that for some people vulkan is a non starter. Hence the effort to swap to opengl.

[Kethku]

This has just been merged!

[Noobertoo]

if you use LIBGL_ALWAYS_SOFTWARE=1 you need to edit neovide command .
or you can create a command called nvide wich make LIBGL_ALWAYS_SOFTWARE=0 and run neovide at the same time how? :
1- create nvide file in ~/.local/bin/ :
vim ~/.local/bin/nvide
2- past this lines inside the nvide file :

#!/bin/sh
#~/.local/bin/nvide"
export LIBGL_ALWAYS_SOFTWARE=0
exec neovide

3- Then give it the proper execution permissions :
chmod +x ~/.local/bin/nvide

now you can run nvide command