Discussion: Providing GLSL Transitions to Movie Editors

[gre]

One of the goal of GLSL.io is to provide transitions for Video Editors.

from @rectalogic :

The goal of rendermix was to implement a video mixer using GLSL shaders to implement transitions/effects. This is also what it's precursor WebVfx does - WebVfx allows you to write transitions/filters/effects using WebGL and use those in a video processing pipeline as an MLT plugin. WebVfx is based on QtWebKit, but you need a version built to enable WebGL. QtWebKit has been abandoned and the replacement QWebEngine will not have the features WebVfx requires.

So I wrote v8-webgl - a partial implementation of WebGL using desktop OpenGL and the v8 JavaScript engine, along with ANGLE to translate the WebGL glsl shaders to desktop OpenGL - independent of a web browser and designed to be used as a video editor plugin. Then I wrote rendermix (based on the jMonkeyEngine videogame engine), and some others as attempted replacements.

All that said, I think the most straightforward way to make glsl.io work with video editors is to write a frei0r plugin. ffmpeg, mlt and others all already support frei0r. I think the plugin would need to use ANGLE to translate the WebGL GLSL transition shader to OpenGL GLSL.

So basically a frei0r mixer2 or mixer3 plugin that would initialize an OpenGL context, convert the transition glsl from WebGL to OpenGL using ANGLE and compile the shader program, then in f0r_update2 upload the image bits for each frame with glTexImage2D, run the shader program, download the resulting bits using glReadPixels and set those in the f0r_update2 outframe argument.

[gre]

@rectalogic: I'm a complete neophyte in OpenGL, so I was about to naively try to run my GLSL in OpenGL, how is that not possible? I'm curious, how far ANGLE goes with this WebGL -> OpenGL conversion, does it even run JavaScript?

frei0r sounds awesome, for the example of ffmpeg, do you think that would be possible to have this kind of CLI usage #43 or do you see limitation I don't see?

Thanks

[rectalogic]

WebGL GLSL is based on OpenGL ES 2.0 and has a number of differences with the various versions of OpenGL GLSL http://stackoverflow.com/a/8872201/239247 . Google Chrome uses ANGLE to translate WebGL GLSL into GLSL that can be used with OpenGL (since the browser is probably using OpenGL). It doesn't use JavaScript, ANGLE basically takes the WebGL GLSL shader source and transforms it to OpenGL GLSL - just translating between two versions of GLSL. You can see that being done here - I'm converting shader source code from SH_WEBGL_SPEC to SH_GLSL_OUTPUT.
https://github.com/rectalogic/v8-webgl/blob/master/src/shader_compiler.cc#L44

And yes, I think you can do a CLI like #43 - see the examples of how to use frei0r with ffmpeg today:
https://www.ffmpeg.org/ffmpeg-filters.html#frei0r-1

So we would have a new shad0r frei0r plugin which would load a glsl.io transition and parse the filter_params to get the uniform values and the glsl.io shader source file.

[rectalogic]

Crap, I wrote most of this (a frei0r mixer2 transition using GLSL) then realized ffmpeg does not support frei0r mixer2 transitions - it only supports frei0r generators and filters.

[rectalogic]

MLT supports frei0r mixer2. Anyway here's a work in progress, currently uses a hardcoded WebGL transition and isn't yet working for a number of reasons.
https://github.com/rectalogic/shad0r

Another approach might be to use movit, which MLT also supports.
http://www.mltframework.org/bin/view/MLT/OpenGL

movit does shader concatenation, and requires you to use macros for your uniforms and shader function, so this would complicate transforming glsl.io shaders.

[gre]

Awesome!!

very cool progress :-)

[gre]

Hi @rectalogic ,
I recently wrote (1) https://github.com/glslio/glsl-transition-validator-cli and (2) https://github.com/glslio/glsl-transition-thumbnail-cli which was 2 attempts to use nodejs + node-webgl as a way to (1) validate a transition and (2) compute a thumbnail image for a transition from the command line.
The idea is to have this running on an headless server validating user gists and computing thumbnail of transitions.

That's nice because this approach allows me to share all the glsl-transition logic (also glsl-transition-validator is used on glsl.io) and I was able to write my cli wrapper in 100 LoC. However, node-webgl is not very stable (it works on my MacOS but it is quite heavy, and it even open a window during the process).
Another attempt would be to use http://npmjs.org/package/gl , but it is not very stable neither, but the author told me he will work with ANGLE in future releases.
I'm not so far from a working solution. I think there could be a way to refactor using gl when it will be based on ANGLE.
As you are the author of v8-webgl I was wondering what do you think of that.

---

Of-course this would only be for the purpose of validation + thumbnail. It won't change the fact we need proper C/C++ implementation if we want to plug in native web editors.

Your frei0r plugin looks awesome I've read the source code, haven't tried to run it yet. I was wondering if we can make a "core" library which would be independent of frei0r and build the frei0r plugin on top of it. Maybe it doesn't make sense?

I think we will need a "glsl parser" to parse the uniforms and dynamically bind them from the JSON uniforms. OpenGL/C++ is new for me but if you want I could try to work on this.

Thanks

[rectalogic]

ANGLE will validate the GLSL and report human readable errors. It also has an API to extract the uniforms from the GLSL it parsed. See ShGetVariableInfo.

Once shad0r is working, you can use it to write a simple thumbnail extractor - just write a small app that loads the shad0r plugin and calls f0r_init, f0r_construct and then f0r_update2 passing in the from and to image buffers and the time you want to render at (e.g. 0.5), and getting out the rendered GLSL image buffer.

[gre]

Awesome if we can get uniforms programmatically :-)

When I say validation it is more functional validation. A transition have a
few properties that could be programmatically checked (all the required
uniforms are defined, from is displayed when progress=0, to is displayed
when progress=1, and a few other stuff).

Thanks

2014-06-20 22:30 GMT+02:00 Andrew Wason notifications@github.com:

ANGLE will validate the GLSL and report human readable errors. It also has
an API to extract the uniforms from the GLSL it parsed. See
ShGetVariableInfo
https://chromium.googlesource.com/angle/angle/+/master/include/GLSLANG/ShaderLang.h
.

Once shad0r is working, you can use it to write a simple thumbnail
extractor - just write a small app that loads the shad0r plugin and calls
f0r_init, f0r_construct and then f0r_update2 passing in the from and to
image buffers and the time you want to render at (e.g. 0.5), and getting
out the rendered GLSL image buffer.

Reply to this email directly or view it on GitHub
#56 (comment).

Gaëtan Renaudeau, greweb.fr

[gre]

Just a little 🆙

A few people are requesting this feature. This is not an easy task to implementing. But it would be incredible for GLSL Transitions ecosystem to grow.

[transitive-bullshit]

@gre @rectalogic I needed this functionality for a project, so I decided to implement it as a new ffmpeg filter. You can check out the source here -- would love to hear your thoughts!

I'm not transpiling the shader source via ANGLE yet because ANGLE seems like a beast. I'm also not sure it's necessary, as webgl glsl appears to be stricter than desktop glsl and the gl-transitions I've tested with have all worked so far.

[gre]

@transitive-bullshit this is awesome!!!