Feature insert time

[KoltesDigital]

Hi!

The first commit was integrated on a testing branch, but still not on the main one.

The second is a feature proposal for inserting time when pressing Ctrl+Shift+T. A friend asked me for this feature. I watched him use the feature very effectively, in order to trigger nice transitions, so given the code below, he would change t0 and immediately reload the shader.

  float t0 = 2755.312;
  float t1 = t0 + 10.;
  float ratio= mix(.2,.5,sat(smoothstep(t0,t1,time)));

Because of the server player, I chose to add a new query system. Whereas commands are broadcasted to all clients, queries require that a single browser is connected (i.e. another socket different from the Atom socket).

I think this can be useful!

Requires fand/vedajs#34

[fand]

Hi @KoltesDigital ! Thanks for proposing a cool feature.
It seems very useful, but I'm not sure this is a best solution to provide a way to work with timings.
Key bindings are easy to use but it can cause confliction, so we have to be careful to add new ones.

I'm also thinking about adding GUI for uniforms to VEDA, yet this is just an idea.
When GUI were implemented there should be more intuitive way to manage timings; such as timeline UI.

Anyways, I don't have enough time to review this PR this week.
I'll review the details next week :) thanks!

[KoltesDigital]

Shortcuts are always a scarce resource... Ctrl+Shift+T reopens the last closed tabs.

Haven't we discussed about the right-pane branch some months ago? Anyway yes it would be cool, I used synthclipse and found this feature interesting, I even wrote a tool for making 4k demos using synthclipse because of this feature (https://github.com/CookieCollective/4k-Demo-Oven) and I'd love to use VEDA instead.

However in this case I'm not sure to have this into a specific uniform controller. The goal is not to have a full demo timeline, but just being able to do transitions. Do you imagine like a uniform float transitionTime; // controller: time-button which would correspond to a button on the panel, and when clicked the uniform's value would be updated to the current time? Seems pretty specific to me. Maybe a uniform controller for transitions in general, where we can specify the transition duration, and the uniform's value would simply be a ratio between 0 and 1.

[fand]

Hi @KoltesDigital , sorry for late reply.
I read though the changes. It's almost acceptable, but I have a few concerns.

First is about the keybinding.
How about using alt-shift-t instead? It's not used by Atom.

The second is, i'm not sure inserting text is the best way to send values to GLSL world.
Using uniform float transitionTime looks better to me.

You can create uniform variables by veda.SetUniform() .
Would you fix it before merge?
Otherwise I can do that after merging this PR.

BTW, I still wanna add GUI uniform controller to VEDA. (yet the PR is abandoned... I will create another PR😅)
Once we have GUI uniform controller, we might need multiple transitionTimes...
So we might change the uniform type to uniform sampler2D transitionTime someday.

[fand]

Wow, I just wanted this kind of methods.. this is useful in many situations, like querying available audio inputs from the browser! 😍

[KoltesDigital]

I mirrored your command system. Although I'm concerned with CommandData... I suggest to change both systems to discriminated unions, so that in the switch (command.type) cases, the data type would be automatically inferred. Safer and prettier!

[KoltesDigital]

I'm fine with any shortcut. BTW atom-terminal also both ctrl-shift-t and alt-shift-t!

I'm also fine with the transitionTime uniform. Although I don't see any other use, maybe naming it transitionTime is too specific. A uniform float userTime and a command/shortcut Set userTime to current time would let users come up with new usages :)

[KoltesDigital]

BTW, have a look at how synthclipse manages uniform controllers, and value curves over time. It may give you new ideas.

[fand]

Oh I overlooked atom-terminal... maybe ctrl-alt-t is available? 😹

For naming, How about using triggerTime?
I also thought about using keyframe, but it maybe confusing when GUI is implemented.

[fand]

Whoa, I tried synthclipse once but didn't know it's that powerful... 🙀
http://synthclipse.sourceforge.net/user_guide/fragx/uniform_controls.html

I love ISF and I thought extending ISF is the best way to bind uniforms to GUI.
But Fragx looks more intuitive, yet it'll need dirty hacks to manipulate shader code. Hmmmm

[KoltesDigital]

Well I don't think it's worth to find an unused key combination. The chances of using VEDA with Atom-terminal are almost zero. And if I were to use a terminal, I want it to be outside of Atom.

As I understand, FragX only means GLSL + uniform name convention + #include, which you already have anyway (another name convention and glslify). Moreover I prefer a descriptive format, especially in JSON, because it allows for easy manipulation (e.g. https://github.com/marcinbiegun/isf-touchdesigner). So I'm totally for supporting ISF's INPUTS property.

What if the panel retrieves the list of active uniforms (glGetProgramiv, glGetActiveUniform) and it would automatically propose default controllers based on the uniform type for each one, which can then be overridden. And what if this manual override would store this customization in the file's settings header, i.e. the header would be overwritten? Synthclipse stores that in an extra file, but in the case of VEDA it makes sense to change the header. So settings would still be written in ISF, there's the GUI which is more convenient than writing INPUTS: [{name: ..., type: ..., min: ... by hand, and there's no need to parse the shader to find annotations.

I'm fine with a uniform triggerTime and a command Trigger time although I find it less descriptive.

In the meantime, could you please integrate the first commit in the main branch? Fix convertPathForServer for Windows. I have another feature in mind and I'll have to use it again.

[fand]

Ok, I cherry-picked fc652f4 into master and pushed

[fand]

So settings would still be written in ISF, there's the GUI which is more convenient than writing INPUTS: [{name: ..., type: ..., min: ... by hand, and there's no need to parse the shader to find annotations.

OH my god, this is the way to go

[eliemichel]

Hello, as the aforementionned "friend who asked for this", I wanted to highlight that one may use many transitions at the same time, not especially starting at the same time. Though the triggerTime is a cleaner way to implement it, it does not scale well to many triggers!

[fand]

Hi @eliemichel !
I agree with you, but I think triggerTime can handle multiple triggers using textures:

uniform sampler2D triggerTime;

// If we have 2 transitions
float t1 = texture2D(trigterTime, vec2(0, 0));
float t2 = texture2D(triggerTime, vec2(1, 0));

// If we have 5 transitions
float t1 = texture2D(trigterTime, vec2(0, 0));
float t2 = texture2D(triggerTime, vec2(0.25, 0));
float t3 = texture2D(trigterTime, vec2(0.5, 0));
float t4 = texture2D(triggerTime, vec2(0.75, 0));
float t5 = texture2D(triggerTime, vec2(1, 0));

For now we can't set multiple values since we don't have GUI, so the width of the texture must be 1.
Once we implement GUI we can store arbitrary count of values in triggerTime - it's all up to the users.

VEDA's OSC textures works like this; the width of the OSC textures are detemined by the length of OSC message.

[KoltesDigital]

However, I don't know how you plan to manage multiple triggers within Atom. Multiple shortcuts are a UI flood, while asking which trigger to trigger defeats its purpose. You could achieve that with a MIDI controller and the existing MIDI binding, where you detect a button event by storing the state of the MIDI texture at previous frame using a pass.

I think we're trying to push even more information to the user with predefined uniforms, while we're trying at the same time to let users choose what they want with ISF's generic INPUTS and uniform controls. This would also resolve issues with other hardcoded inputs, e.g. webcams (#122).

So, why not using INPUT type event, which would translate as a button in the uniform controls panel? And users are free to name it as they want. Bonus, we could add "triggerOnNoteOn": XX to conveniently bind to MIDI messages.

Besides, I'm in favor of adding new INPUTS types and new properties because the specs only provide basic ones. For instance, a type osc with a property route or url would be prettier than replacing /
with _ in the url.

Regarding webcams, we could introduce an INPUT type videoinput with a new property index for selecting the source (as we can retrieve them now #131) and the corresponding uniform controls would be a <select>.

The point is, there would be a preparation step before the performance, where the users need to configure inputs with the panel, and then during the performance the panel shouldn't be used anymore. I still want that current uniforms work out of the box, because it's very nice for new users. Forcing a preparation step would make new users go away.

[fand]

@KoltesDigital

However, I don't know how you plan to manage multiple triggers within Atom.
Multiple shortcuts are a UI flood, while asking which trigger to trigger defeats its purpose.
You could achieve that with a MIDI controller and the existing MIDI binding, where you detect a button event by storing the state of the MIDI texture at previous frame using a pass.

I'm thingking about adding these features gradually.
How about this?

1. Use uniform float triggerTime for this PR
   • Add keybinding to ctrl-alt-t or alt-shift-t
2. Add uniform sampler2D triggerTimes after GUI is implemented
3. Add MIDI keyboard bindings to GUI input forms in another PR

If you agree with this and fix this PR, I'll merge this.
I can't wait playing with triggers!!

[fand]

The point is, there would be a preparation step before the performance, where the users need to configure inputs with the panel, and then during the performance the panel shouldn't be used anymore. I still want that current uniforms work out of the box, because it's very nice for new users. Forcing a preparation step would make new users go away.

Actually I wanna let users touch GUI panels during performance.
GUI panel will have 2 sections: Settings and Uniform Contoller.
Settings section is for setting audio input, video input, etc.
Uniform Controller allows users to add/remove input forms for uniforms, and the input forms will work just like MIDI controllers.

just an idea, though 👻

[eliemichel]

TBH I don't totally get what's wrong with the raw "insert current time" function. This texture-based solution is smart, but feels a bit overkill. As @KoltesDigital noticed, if one wants such an elaborate setup one may use multiple buffers to cache some MIDI/OSC trigger input.

Also, I have some small concerns about texture-based input. The first one is about mip maps. I guess there is no mip map at all for those textures, but should I query the pixel at vec2(.25, .0) or vec2(.375, .5) to get the second input out of four?

My second point is about performance, and I'm asking because I really don't know if it has any impact: let's say I have an OSC input providing me with the current BPM, and that I use this bpm info in many many places around the code. Is it ok to perform a call to texture() everytime, or should I take care of passing the bpm around as an argument to all my functions? In the latter case, it would be more convenient to get the OSC/MIDI/trigger/... input as a float/vec uniform rather than a sampler. I am asking because I've been taught in regular shaders to minimize the use of `texture()``calls, but it might not be relevant in the case of very small textures.

[fand]

Inserting raw float constant is very interesting, but it's too magical, IMO.

Inserting code is a completely new feature.
For now VEDA is just a GLSL runner, it doesn't have any features which extend Atom's basic text editing experience.

Extending text editing features, itself is not bad idea.
For example, I like glslViewer style float input.
But I personally don't like to add magic numbers in GLSL code...

Also, when we've implemented GUI controller and trigger buttons we'll have 2 different ways to manipulate trigger values - sounds confusing to me.

[fand]

should I query the pixel at vec2(.25, .0) or vec2(.375, .5) to get the second input out of four?

vec2(.25, 0), vec2(.375, 0) and vec2(.49, 0) will return same value.
VEDA uses THREE.DataTexture to create texture from data.
This class uses gl.NEAREST for min/mag filter, so vec2(.25, 0) and vec2(.49, 0) points to same texel.

Is it ok to perform a call to texture() everytime, or should I take care of passing the bpm around as an argument to all my functions?

It depends on how many times you call texture().
It doesn't matter if you only call texture() several times, but if you call it dozens of times it'll start to drop frames.
If you need to do so, I suggest caching the value to global variables like this:
(This is a real code I used before)

uniform sampler2D osc_foo;

float o0;
float o1;
float o2;
float o3;

float osc(in float ch) {
  return texture2D(osc_foo, vec2(ch / 4.));
}

void main() {
  o0 = osc(0.);
  o1 = osc(1.);
  o2 = osc(2.);
  o3 = osc(3.);

  ...
}

[fand]

FWIW, I like to write tiny scripts which cooperates with VEDA over OSC.
For example, I once created an OSC sequencer:

https://twitter.com/amagitakayosi/status/1106768974463668225
https://github.com/fand/launchpad-to-osc

[eliemichel]

Thanks for all these replies! :) I understand that you insist on keeping the core design of Veda clean. I care about clean code as well when I take the time to write it, but doing live-coding is a different matter for which tricks like global variable caching will be helpful!
And funny you mention writing a companion OSC-based tool, I just published mine ;)
https://twitter.com/exppad/status/1134217836057022464
https://github.com/eliemichel/Cooldown

[fand]

Hi folks,
While the discussion is not fully solved yet, I merged this PR because now I think VEDA should cover as many use cases as possible.
Sorry for leaving this PR unmerged for long time...