Sverchok update system RFC

[ly29]

A description about how updates happens in Sverchok since I know people wonder about it. Please ask questions.

Update events

Update can happen in the following ways:

• Editor changes
• Value change in a node
• Frame change
• Update button

What happens when each event is issued.

Editor changes

The whole node group is evaluated and a list of nodes is created. Then the update function is called on each node in a safe order. The update event from blender is issued to the class SverchCustomTree(NodeTree):
Since the it is hard to know what has changed between two editor change updates the whole list rebuilt, the socket cache cleared and update is called for the whole node group.

def update(self):
        makeTreeUpdate2(tree_name = self.name)
        speedUpdate(tree_name = self.name)

Value change in a node

If a numeric value in for example a float node is changed the following code is run.

def updateNode(self, context):
    speedUpdate(start_node = self.name,tree_name =self.id_data.name)

Note These list should be cached but at the moment they are not. Done!

Animation update

Animation updates are issued from a frame change handler.

 def update_ani(self):
        if self.sv_animate:
            speedUpdate(tree_name = self.name)

@persistent
def sv_update_handler(scene):
    '''Sverchok update handler'''
    for name,tree in bpy.data.node_groups.items():
        if tree.bl_idname =='SverchCustomTreeType' and tree.nodes:
            try:
                tree.update_ani()                
            except Exception as e:
                print('Failed to update:',name,str(e))

Update buttons

The various update buttons use the following operator.

class SverchokUpdateAll(bpy.types.Operator):

  def execute(self, context):
        makeTreeUpdate2()
        speedUpdate()
        return {'FINISHED'}

Note Perhaps we should separate this to have a update All and update current node tree now that we can show more than one node tree at the same time.

Functions

speedUpdate()

speedUpdate is the master update function which calls the update system and dispatches the various calls.
It has several modes.

• Partial update only part of a node group starting in a named node
• Update a named node group
• Update all Sverchok node group
• Update an Animation tree (not used yet)

Note It needs some cleaning to be honest.

makeTreeUpdate2()

If tree_name is passed an update list is created for that node group is stored in a update list cache. Without tree_name all node groups are processed.

Note The name comes since this is rewrite of the original update makeTreeUpdate2() function.

Building update lists

There are some function the can be used to create update lists. The list are flat and for the whole tree.

Make update list

Create an update list for a node group or a specified subset of node group. The node_set options is used by the next partial updates but can also be used for creating animation trees and other partial update trees.

Gives the following list:

1. Float
2. List Series
3. Vectors in
4. List Length
5. Vectors out
6. List Series.001
   7 Vectors in.001
   8 Text Output

def make_update_list(node_tree,node_set = None):
    """ Makes a list for updates from a node_group
    if a node set is passed only the subtree defined by the node set is used. Otherwise
    the complete node tree is used.
    """ 

A short description of the algorithm used:

• Select nodes that have inputs or outputs.
• Create a dict of {nodeA :[nodeB, nodeC nodeD]} where the nodes in the list are the nodes that needs be updated before nodeA.
• Select the wifi nodes and crete dependecies for them if needed.
• Start with nodes without outputs and try to satisfy the dependecies while maintaining a stack of nodes and creating the output list for each node where the dependencies are satisfied.

It does some error checking on the way and the whole system fails without notifying the user. Because there are many fail invalid node trees states created during normal editing. For example every time you drag a noodle from one node to another 2 updates event are issued and one socket is connected and the other is not.

make_tree_from_nodes

Create an update list of nodes affected by a change by a set of nodes.

def make_tree_from_nodes(node_names,tree_name):
    """
    Create a partial update list from a sub-tree, node_names is a list of nodes that
    drives change for the tree
    Only nodes downtree from node_names are updated
    """

Future issues.

Some issued for the future.

Node state and Error handling

Node need a way to mark that they are not ready and in such cases they will not be updated. Also they need should be able to be marked as failed. I think 3 node states are needed.

• Not ready
• Ready
• Failed

Separating UI and execute

Right now the ui changes and data processing events are not separated. They should be since editor changes might affect the node function and cause incorrect states. Also the other types of events (frame change and value change) do not need the UI update code to run.

I propose the following.

def update(self):
     handle sockets
     set state

def execute(self):
    process data

Animation node trees

Detect driven, animated etc nodes and update only update the parts of the node tree that are needed. Low priority.

Cache animation states

Not sure about how to do this. Low priority.

[ly29]

Another issue is to make the terminolgy consistant. Instead of tree, node group is a better description since it matches blender.

[zeffii]

the first line under the header "update events" doesn't make sense in English reads

[ly29]

My written English is bad sometimes. Please point out more unclear points.

[zeffii]

Giving feedback maybe less useful because I have little experience with parallelism + graph programs.

I'm wondering if with Blender shipping with numpy that it may be in our best interest to migrate all the code that could benefit from that. Also what may be done for processing on multiple cores, when the node tree would allow it..

[ly29]

I have probably have less knowledge than you do.

[ly29]

numpy would speed it up I think.
to factor out parts of node tree that can be updated in parallell wouldn't be hard to do. but from what I understand python only allow one thread in blender.

[zeffii]

@ly29 I hadn't planned on immersing myself in the lower levels of Sverchok, but even that is becoming increasingly interesting. Yeah if the single core issue is still the case, it won't stay that way for very long especially with the possibility of leveraging multicore graphics cards as matrix workers.

[ly29]

@zeffii I stayed away from until it broke a bit to much for me.
Also I wrote this so others could improve it or understand some of the issues.

[ly29]

next topic is the socket data cache...

[zeffii]

is that to check if work needs to be done, by keeping track of which nodes have unchanged socket inputs + outputs?

[ly29]

That is something that could be done. Right now we just assume all nodes that are linked downstream from a changed input have to be updated on a partial update.
But for example for Kd Tree it would be great if would cache the KD Tree and you can adjust the other inputs if the vertices don't change.

[ly29]

New development in #152 instead.