Switch the version of litegraph used to litegraph.ts

[space-nuko]

This PR is for migrating the version of LiteGraph used by ComfyUI to litegraph.ts.

NOTE: There will be some amount of incompatibilities with existing frontend extensions with this approach, I would encourage testing them to see what has to be changed before deciding this is a good idea. All core extensions have been migrated already. And of course please test your existing workflows because something will probably break sooner or later!

No compilation is needed if you check out this PR, the intention is to have litegraph.core.js precompiled in this repo and track litegraph.ts changes in a separate repo (no submodules). Also I left ComfyUI using JavaScript to avoid compilation and keep things simple

Benefits:

1. Can be community supported by ComfyUI developers instead of waiting for litegraph upstream to merge desirable changes, since it seems most adoption of litegraph has come from ComfyUI in the recent past. If this is merged I would like to give ownership of litegraph.ts to those in the ComfyUI development community that want to help support the library into the future
2. The codebase was converted to TypeScript and cleaned up. Now all relevant parts of the code are divided into separate files and use ES modules and classes. This makes it much easier to make changes to the litegraph codebase without introducing type errors and some other classes of bugs. This also makes it easier to create custom node types with extends. (When compiling for use with ComfyUI it outputs a single .js file like before)
3. Better support for litegraph feature extension. An example is a new callback to customize context menus. Previously the only way to do that was to manually replace the ContextMenu constructor in litegraph.
4. Can be used to add subgraph support to ComfyUI since the subgraph implementation in litegraph.ts was fixed. Since litegraph's implementation of subgraphs depends on its own internals, using that implementation would mean modifying litegraph itself
5. Can be expanded for custom widget support later
6. Feature parity with ComfyBox workflows. The most significant feature that's been added to litegraph.ts/ComfyBox is the ability to add graph-internal logic using node-based events. So there wouldn't be a need to implement branching nodes on the backend to have conditional workflows anymore. This can open up a lot of future possibilities like conditionally turning off nodes based on workflow state, firing notifications when jobs finish, and so on

Drawbacks:

1. There may be a lot of bugs and growing pains owing to switching the implementation of litegraph, and the codebase of litegraph.ts has diverged significantly from upstream
2. Most frontend extensions will likely need some amount of changes to remain compatible, usually by adding import statements to the top of each file since LiteGraph and its classes are no longer globals. The changes are relatively minor from what I've handled so far and all core extensions were migrated save for any bugs I missed
3. litegraph.ts requires a TypeScript toolchain and a compilation step (but not for any changes local to ComfyUI's own JavaScript)

To build litegraph.ts for ComfyUI:

1. Check out litegraph.ts at the comfyui branch
2. Install pnpm
3. Run pnpm install in the root directory of litegraph.ts
4. Run bash build.sh with git-bash or similar
5. Copy out/litegraph.core.js to ComfyUI/web/lib and replace the existing version

The version of litegraph.core.js in this PR is precompiled to match the latest changes

[0xdevalias]

Wanted to include this idea/note I posted on another issue here; RE: litegraph bugs/limitations and a potentially better alternative:

I think it will require several changes in litegraph to fix bugs.

These changes are in my fork of litegraph which isn't API compatible with upstream litegraph however as the pace of development is too significant

This is probably a little off topic for this issue, and I suspect that it would be a rather major / potentially breaking change to do so.. but I'm curious if there's a reason that litegraph is used instead of something like xyflow / reactflow / etc? I haven't deeply evaluated both projects yet, but at least in my initial cursory exploration, xyflow seems like it's capable of everything liteflow does, but in a more standard are modern developed/maintained way.

• https://github.com/xyflow/xyflow

  • React Flow | Svelte Flow - Powerful open source libraries for building node-based UIs with React (https://reactflow.dev) or Svelte (https://svelteflow.dev). Ready out-of-the-box and infinitely customizable.

  • https://www.xyflow.com/

    • Powerful open source libraries for building node-based UIs with React or Svelte. Ready out-of-the-box and infinitely customizable

  • https://reactflow.dev

    • Wire Your Ideas with React Flow A customizable React component for building node-based editors and interactive diagrams

  • https://svelteflow.dev

    • Wire Your Ideas with Svelte Flow A customizable Svelte component for building node-based editors and interactive diagrams by the creators of React Flow

Originally posted by @0xdevalias in #669 (comment)

[YunYouJun]

Wanted to include this idea/note I posted on another issue here; RE: litegraph bugs/limitations and a potentially better alternative:想包括我在这里发布的另一个问题的想法/注释;回复：litegraph 错误/限制和可能更好的替代方案：

I think it will require several changes in litegraph to fix bugs.我认为需要对 litegraph 进行一些更改才能修复错误。

These changes are in my fork of litegraph which isn't API compatible with upstream litegraph however as the pace of development is too significant这些变化在我的 litegraph 分支中，它与上游 litegraph 的 API 不兼容，但是由于开发速度太快了

This is probably a little off topic for this issue, and I suspect that it would be a rather major / potentially breaking change to do so.. but I'm curious if there's a reason that litegraph is used instead of something like xyflow / reactflow / etc? I haven't deeply evaluated both projects yet, but at least in my initial cursory exploration, xyflow seems like it's capable of everything liteflow does, but in a more standard are modern developed/maintained way.对于这个问题来说，这可能有点偏离主题，我怀疑这样做将是一个相当重大/潜在的突破性变化。但我很好奇是否有原因使用 litegraph 而不是 xyflow / reactflow / 等？我还没有深入评估这两个项目，但至少在我最初的粗略探索中，xyflow 似乎能够完成 liteflow 所做的一切，但以更标准的方式是现代开发/维护。

• xyflow/xyflow

  • React Flow | Svelte Flow - Powerful open source libraries for building node-based UIs with React (reactflow.dev) or Svelte (svelteflow.dev). Ready out-of-the-box and infinitely customizable.反应流 |Svelte Flow - 强大的开源库，用于使用 React （ reactflow.dev） 或 Svelte （ svelteflow.dev） 构建基于节点的 UI.开箱即用，可无限定制。

  • xyflow.com

    • Powerful open source libraries for building node-based UIs with React or Svelte. Ready out-of-the-box and infinitely customizable强大的开源库，用于使用 React 或 Svelte 构建基于节点的 UI。开箱即用，可无限定制

  • reactflow.dev

    • Wire Your Ideas with React Flow A customizable React component for building node-based editors and interactive diagrams用 React Flow 连接你的想法 一个可定制的 React 组件，用于构建基于节点的编辑器和交互式图表

  • svelteflow.dev

    • Wire Your Ideas with Svelte Flow A customizable Svelte component for building node-based editors and interactive diagrams by the creators of React Flow用 Svelte Flow 连接你的想法 一个可定制的 Svelte 组件，用于由 React Flow 的创建者构建基于节点的编辑器和交互式图表

Originally posted by @0xdevalias in #669 (comment)引用自 in #669 （comment）

Agreed, I once tried to implement a script workflow using litegraph, but its lack of customization and well-documented API made it a bad development experience that I had to give up. Then I used xyflow, which did almost all of my customization needs, along with a modern front-end development experience.

[0xdevalias]

I once tried to implement a script workflow using litegraph, but its lack of customization and well-documented API made it a bad development experience that I had to give up. Then I used xyflow, which did almost all of my customization needs, along with a modern front-end development experience.

@YunYouJun One thing that litegraph seems to be able to do is to 'execute' the graphs; but in my exploration of xyflow / reactflow / etc, I haven't yet come across a good simple example of how to 'execute' a graph made with them. Do you have any good examples of this; or is it more of a 'manually implement based on your specific needs' sort of thing?

Originally I would have expected that for each node I could implement something like a 'transformer' function, that would then process the data flowing through the graph; but I'm thinking maybe that's not the case?

ChatGPT seems to suggest that it's up to us to implement the execution flow separately:

Details

Implementing a graph execution logic in React Flow (previously known as ReactFlow) or a similar library like XyFlow involves a few key steps. These libraries are primarily focused on the visualization and interaction part of node-based graphs, so the execution logic is something you need to implement yourself. Here's a basic guide on how to approach this:

1. Graph Setup: In React Flow, your graph is represented by a set of nodes and edges. Each node and edge has its own properties and data. You can set up these nodes and edges in the state of your React component.

2. Defining Node Behavior: Each node should have a defined behavior. This could be a function or a class that determines what the node does when it receives input.

3. Processing Nodes: You'll need to create a function that processes the nodes. This function will take the input of a node, use the node's defined behavior to process this input, and then pass the output to the next node.

4. Managing Connections: React Flow manages the visual connections between nodes. You need to use this connection information to determine the flow of data in your execution logic.

5. Executing the Graph: Create a function that initiates the execution of the graph. This could start from nodes without inputs (like start nodes) and then traverse through the graph based on the connections.

Here's a simplified example in a React component:

import React, { useState } from 'react';
import ReactFlow from 'react-flow-renderer';

const initialNodes = [
  // ... define your nodes
];

const initialEdges = [
  // ... define your edges (connections)
];

const nodeBehaviors = {
  // Define behaviors for different node types
  // e.g., 'number': (node) => node.data.value,
  //       'add': (node, inputs) => inputs.reduce((a, b) => a + b, 0)
};

const processNode = (node, nodes, edges) => {
  // Process the node based on its type and inputs
  // ...
};

const executeGraph = (nodes, edges) => {
  // Logic to execute the graph
  // ...
};

function GraphComponent() {
  const [nodes, setNodes] = useState(initialNodes);
  const [edges, setEdges] = useState(initialEdges);

  const handleExecute = () => {
    executeGraph(nodes, edges);
  };

  return (
    <div>
      <ReactFlow nodes={nodes} edges={edges} />
      <button onClick={handleExecute}>Execute Graph</button>
    </div>
  );
}

export default GraphComponent;

In this example:

• initialNodes and initialEdges define the initial state of your graph.
• nodeBehaviors is an object where you define what each node type should do.
• processNode is a function that processes a single node.
• executeGraph is a function that handles the execution logic of the entire graph.
• The GraphComponent is a React component that renders the graph and has a button to trigger the execution.

This is a basic framework. The actual implementation of processNode and executeGraph will depend on the specific requirements of your graph and the complexity of the node behaviors. React Flow provides the visual and interactive part, but the execution logic is up to you to define.

To execute a graph and process nodes in a React Flow-based setup, you need to establish a systematic approach for traversing the graph, processing each node according to its type and the data it carries, and then passing the output to connected nodes. This involves a few key concepts:

1. Node Processing Function: You should have a function for each type of node that knows how to process that node's inputs and produce an output. This function will be specific to what you want each node type to do.

2. Graph Traversal: The execution function should traverse the graph in a way that respects the flow of data. Typically, you'd start with nodes that have no input (start nodes) and then move to nodes that those nodes output to, and so on.

3. Data Management: You need a way to manage the data that is being processed. This includes storing the output of nodes and passing it as input to connected nodes.

4. Cyclic Graph Handling: If your graph allows cycles, you need to handle them carefully to avoid infinite loops.

5. Asynchronous Processing: If some nodes perform asynchronous operations (like fetching data from an API), your execution logic should be able to handle these operations correctly.

Here is a more detailed example of how you might implement this:

import React, { useState } from 'react';
import ReactFlow, { isNode } from 'react-flow-renderer';

const initialNodes = [
  // ... define your nodes
];

const initialEdges = [
  // ... define your edges (connections)
];

// Define behaviors for different node types
const nodeBehaviors = {
  number: (node) => node.data.value,
  add: (node, inputs) => inputs.reduce((a, b) => a + b, 0),
  // ... other node types
};

const processNode = (node, processedNodes) => {
  // Find inputs from the edges
  const inputs = node.inputs.map(inputId => processedNodes[inputId]);

  // Process the node based on its type
  return nodeBehaviors[node.type](node, inputs);
};

const executeGraph = (nodes, edges) => {
  const processedNodes = {};
  const nodesToProcess = nodes.filter(n => isStartNode(n, edges));

  while (nodesToProcess.length > 0) {
    const node = nodesToProcess.shift();

    // Process the node
    const output = processNode(node, processedNodes);

    // Store the output
    processedNodes[node.id] = output;

    // Add connected nodes to the processing queue
    edges.filter(edge => edge.source === node.id)
         .forEach(edge => nodesToProcess.push(getNodeById(nodes, edge.target)));
  }

  return processedNodes;
};

// Helper functions
const isStartNode = (node, edges) => !edges.some(edge => edge.target === node.id);
const getNodeById = (nodes, id) => nodes.find(node => node.id === id);

function GraphComponent() {
  const [nodes, setNodes] = useState(initialNodes);
  const [edges, setEdges] = useState(initialEdges);

  const handleExecute = () => {
    const outputs = executeGraph(nodes, edges);
    console.log(outputs); // You can do something with the outputs here
  };

  return (
    <div>
      <ReactFlow nodes={nodes} edges={edges} />
      <button onClick={handleExecute}>Execute Graph</button>
    </div>
  );
}

export default GraphComponent;

In this example:

• nodeBehaviors defines how different node types are processed.
• processNode processes a single node using nodeBehaviors.
• executeGraph traverses the graph, starting with nodes that have no incoming edges (start nodes), and processes each node.
• The GraphComponent includes a button to trigger the graph execution.

This is a simplified framework. Depending on your application, you might need to add error handling, manage state more robustly, and handle asynchronous operations. Also, the handling of inputs and outputs, as well as the traversal order, might need to be adjusted based on your specific graph structure and requirements.

[YunYouJun]

I used xyflow api to implement a set of task nodes execution mechanism by myself.

I used mitt to build an event management mechanism. When the node completes, the event information about the completion of the node is emitted.
Based on the entry of the node, listen to its upstream nodes.

When we click Run, we complete the workflow by starting the execution of the node that is not entered by default.

This is an example can preview:

[0xdevalias]

I used xyflow api to implement a set of task nodes execution mechanism by myself.

I used mitt to build an event management mechanism.

@YunYouJun Ah true. That makes sense. Thanks! :)

This was another node UI project + graph execution project I came across that looks pretty interesting/useful as well:

• https://retejs.org/

  • A tailorable TypeScript-first framework for creating processing-oriented node-based editors

• https://github.com/retejs/rete

  • JavaScript framework for visual programming

    Rete.js is a framework for creating visual interfaces and workflows. It provides out-of-the-box solutions for visualization using various libraries and frameworks, as well as solutions for processing graphs based on dataflow and control flow approaches.

[YunYouJun]

retejs.org

@0xdevalias

Cool, it looks more like it was designed specifically for node workflows.

Perhaps using it directly can save some development time.

Unfortunately, I already have a lot of customization for xyflow, which gives me no incentive to try it in the short term.

[clarkmcc]

@YunYouJun is your example open source? Would love to look into it in more depth.

[YunYouJun]

@clarkmcc

Sorry, it's still in development and I'm not ready to open source it just yet.
But in the future it will be open source.

[mcmonkey4eva]

This would need a total rebuild against current code before it could be mergeable