Execution Model Inversion

[guill]

This PR inverts the execution model -- from recursively calling nodes to using a topological sort of the nodes. This change allows for modification of the node graph during execution. This allows for two major advantages:

1. The implementation of lazy evaluation in nodes. For example, if a
"Mix Images" node has a mix factor of exactly 0.0, the second image
input doesn't even need to be evaluated (and visa-versa if the mix
factor is 1.0).

2. Dynamic expansion of nodes. This allows for the creation of dynamic
"node groups". Specifically, custom nodes can return subgraphs that
replace the original node in the graph. This is an incredibly
powerful concept. Using this functionality, it was easy to
implement:
    a. Components (a.k.a. node groups)
    b. Flow control (i.e. while loops) via tail recursion
    c. All-in-one nodes that replicate the WebUI functionality
    d. and more
All of those were able to be implemented entirely via custom nodes,
so those features are *not* a part of this PR. (There are some
front-end changes that should occur before that functionality is
made widely available, particularly around variant sockets.)

The custom nodes associated with this PR can be found at:
https://github.com/BadCafeCode/execution-inversion-demo-comfyui

Note that some of them require that variant socket types ("*") be enabled.

[comfyanonymous]

If anyone can test and report if it works or not with their most complex workflows that would be very helpful.

[Bocian-1]

All errors I get on this one seem to be from custom nodes. The missing input and invalid image file errors happen regardless due to how this workflow is built
Interaction OpenPose.json

got prompt
ERROR:root:Failed to validate prompt for output 2003:
ERROR:root:* ImageReceiver 3135:
ERROR:root:  - Custom validation failed for node: image - Invalid image file: ImgSender_temp_uvqrp_00001_.png [temp]
ERROR:root:  - Custom validation failed for node: link_id - Invalid image file: ImgSender_temp_uvqrp_00001_.png [temp]
ERROR:root:  - Custom validation failed for node: save_to_workflow - Invalid image file: ImgSender_temp_uvqrp_00001_.png [temp]
ERROR:root:  - Custom validation failed for node: image_data - Invalid image file: ImgSender_temp_uvqrp_00001_.png [temp]
ERROR:root:  - Custom validation failed for node: trigger_always - Invalid image file: ImgSender_temp_uvqrp_00001_.png [temp]
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 3862:6:
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 3133:5:
ERROR:root:* (prompt):
ERROR:root:  - Return type mismatch between linked nodes: a, INT != INT,FLOAT,IMAGE,LATENT
ERROR:root:  - Return type mismatch between linked nodes: b, FLOAT != INT,FLOAT,IMAGE,LATENT
ERROR:root:* MathExpression|pysssss 3133:5:
ERROR:root:  - Return type mismatch between linked nodes: a, INT != INT,FLOAT,IMAGE,LATENT
ERROR:root:  - Return type mismatch between linked nodes: b, FLOAT != INT,FLOAT,IMAGE,LATENT
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 3059:11:
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 3822:16:
ERROR:root:* MathExpression|pysssss 3133:4:
ERROR:root:  - Return type mismatch between linked nodes: a, FLOAT != INT,FLOAT,IMAGE,LATENT
ERROR:root:  - Return type mismatch between linked nodes: b, INT != INT,FLOAT,IMAGE,LATENT
ERROR:root:* ImpactConditionalBranch 3822:15:
ERROR:root:  - Required input is missing: tt_value
ERROR:root:* ImpactConditionalBranch 3822:14:
ERROR:root:  - Required input is missing: tt_value
ERROR:root:* KSampler 3822:3:
ERROR:root:  - Required input is missing: latent_image
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 3678:6:
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 3244:
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 2322:
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 3861:6:
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 3133:4:
ERROR:root:* (prompt):
ERROR:root:  - Return type mismatch between linked nodes: a, FLOAT != INT,FLOAT,IMAGE,LATENT
ERROR:root:  - Return type mismatch between linked nodes: b, INT != INT,FLOAT,IMAGE,LATENT
ERROR:root:Output will be ignored
Exception in thread Thread-8 (prompt_worker):
Traceback (most recent call last):
  File "threading.py", line 1045, in _bootstrap_inner
  File "threading.py", line 982, in run
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\main.py", line 111, in prompt_worker
    e.execute(item[2], prompt_id, item[3], item[4])
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 470, in execute
    execution_list.add_node(node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 110, in add_node
    self.add_strong_link(from_node_id, from_socket, unique_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 136, in add_strong_link
    super().add_strong_link(from_node_id, from_socket, to_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 87, in add_strong_link
    self.add_node(from_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 110, in add_node
    self.add_strong_link(from_node_id, from_socket, unique_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 136, in add_strong_link
    super().add_strong_link(from_node_id, from_socket, to_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 87, in add_strong_link
    self.add_node(from_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 110, in add_node
    self.add_strong_link(from_node_id, from_socket, unique_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 136, in add_strong_link
    super().add_strong_link(from_node_id, from_socket, to_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 87, in add_strong_link
    self.add_node(from_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 110, in add_node
    self.add_strong_link(from_node_id, from_socket, unique_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 136, in add_strong_link
    super().add_strong_link(from_node_id, from_socket, to_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 87, in add_strong_link
    self.add_node(from_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 110, in add_node
    self.add_strong_link(from_node_id, from_socket, unique_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 136, in add_strong_link
    super().add_strong_link(from_node_id, from_socket, to_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 87, in add_strong_link
    self.add_node(from_node_id)
  File "C:\AI\ComfyUI_windows_portable\ComfyUI\comfy\graph.py", line 108, in add_node
    is_lazy = "lazy" in input_info and input_info["lazy"]
              ^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

[blepping]

started trying to test this. first the job-iterator extension has to be disabled. it also is not compatible with rgthree's nodes even if you disable the executor stuff in it because it will always try to patch the executor regardless. i commented out that part and rgthree didn't seem to be causing problems after that.

those problems are basically expected, stuff that messes with the executor is not going to be compatible with these changes.

>> info 363 VAE -- None None None
Exception in thread Thread-6 (prompt_worker):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/raid/vantec/ai/models/sd/ComfyUI/main.py", line 111, in prompt_worker
    e.execute(item[2], prompt_id, item[3], item[4])
  File "/raid/vantec/ai/models/sd/ComfyUI/execution.py", line 470, in execute
    execution_list.add_node(node_id)
  File "/raid/vantec/ai/models/sd/ComfyUI/comfy/graph.py", line 109, in add_node
    is_lazy = "lazy" in input_info and input_info["lazy"]
              ^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

this is more of an issue. i added a debug print after line 107 which calls self.get_input_info: print(">> info", unique_id, input_name, "--", input_type, input_category, input_info)

the method can return None however the downstream code doesn't check for that.

>> info 363 VAE -- None None None

seems like it failed fetching info for the standard VAE node. maybe other weird stuff is going on here but there definitely should be more graceful handling for methods that can return None.

don't want to seem negative, i definitely appreciate the work you put into these changes and a better approach to execution is certainly very welcome and needed!

---

edit: did some more digging, the issue seemed to be an incompatibility with the use everywhere nodes (was using that to broadcast the VAE so maybe it wasn't a standard VAE node after all).

are these changes expected to be compatible with use everywhere?

[ltdrdata]

started trying to test this. first the job-iterator extension has to be disabled. it also is not compatible with rgthree's nodes even if you disable the executor stuff in it because it will always try to patch the executor regardless. i commented out that part and rgthree didn't seem to be causing problems after that.

those problems are basically expected, stuff that messes with the executor is not going to be compatible with these changes.

>> info 363 VAE -- None None None
Exception in thread Thread-6 (prompt_worker):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/raid/vantec/ai/models/sd/ComfyUI/main.py", line 111, in prompt_worker
    e.execute(item[2], prompt_id, item[3], item[4])
  File "/raid/vantec/ai/models/sd/ComfyUI/execution.py", line 470, in execute
    execution_list.add_node(node_id)
  File "/raid/vantec/ai/models/sd/ComfyUI/comfy/graph.py", line 109, in add_node
    is_lazy = "lazy" in input_info and input_info["lazy"]
              ^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'NoneType' is not iterable

this is more of an issue. i added a debug print after line 107 which calls self.get_input_info: print(">> info", unique_id, input_name, "--", input_type, input_category, input_info)

the method can return None however the downstream code doesn't check for that.

>> info 363 VAE -- None None None

seems like it failed fetching info for the standard VAE node. maybe other weird stuff is going on here but there definitely should be more graceful handling for methods that can return None.

don't want to seem negative, i definitely appreciate the work you put into these changes and a better approach to execution is certainly very welcome and needed!

edit: did some more digging, the issue seemed to be an incompatibility with the use everywhere nodes (was using that to broadcast the VAE so maybe it wasn't a standard VAE node after all).

are these changes expected to be compatible with use everywhere?

ping @ali1234, @rgthree.

I see that compatibility of existing custom nodes may be compromised with the new structure.

More important than immediate compatibility breaking is verifying whether each extension can provide compatibility patches for the new structure.

[ali1234]

Job iterator only patches the executor to let it run multiple times in a loop. This pull request should make it obsolete.

[rgthree]

Thanks for the ping and opportunity to fix before breaking. The execution optimization was meant to be forward compatible, but it did assume methods weren't being removed.

I just pushed rgthree/rgthree-comfy@6aa0392 which is forwards compatible with this PR by no longer attempting to patch the optimization to ComfyUI's execution if the recursive methods don't exist, as is the case in this PR. I've patched this PR in and ensured it.

Question: I assume this PR will render the rgthree optimization obsolete? Currently, the patch I provide reduces iterations and times by 1000s of percent (from 250,496,808 to just 142, and 158.13 seconds to 0.0 as tested on my machine).

Also, I noticed the client API events have changed (client-side progress bar shows 100% complete, even though the workflow is still running). Are there more details on breaking changes?

[Trung0246]

Crashed for the following nodes while attempt to create info['input_order']:

ttN pipeLoader
ttN pipeLoaderSDXL

Potentially problematic line:
https://github.com/TinyTerra/ComfyUI_tinyterraNodes/blob/main/tinyterraNodes.py#L1534

It looks like the problematic key is my_unique_id, which result in "UNIQUE_ID" string, therefore .keys()call will fail.

---

[FIXED]

Assertion crash during execution. Looks like there's some incorrect assumption when calling BasicCache.set_prompt. The following are simplified call hierarchy:

https://github.com/guill/ComfyUI/blob/36b2214e30db955a10b27ae0d58453bab99dac96/execution.py#L457
https://github.com/guill/ComfyUI/blob/36b2214e30db955a10b27ae0d58453bab99dac96/comfy/caching.py#L141

Then the rest:

CacheKeySetInputSignature.add_keys
CacheKeySetInputSignature.get_node_signature
CacheKeySetInputSignature.get_immediate_node_signature
IsChangedCache.get
get_input_data
cached_output = outputs.get(input_unique_id)
BasicCache._get_immediate

Crash at https://github.com/guill/ComfyUI/blob/36b2214e30db955a10b27ae0d58453bab99dac96/comfy/caching.py#L175

The root cause is kinda hard to explain, but take this workflow for instance (workflow embedded in the image):

This should run fine if every is at it is (notice that the node id order MUST BE 10, 8, 9 such that the middle node always have the lowest id). But the moment when I change the source code of Concat Text_O to this:

class concat_text_O:
    """
    This node will concatenate two strings together
    """
    @ classmethod
    def INPUT_TYPES(cls):
        return {"required": {
            "text1": ("STRING", {"multiline": True, "defaultBehavior": "input"}),
            "text2": ("STRING", {"multiline": True, "defaultBehavior": "input"}),
            "separator": ("STRING", {"multiline": False, "default": ","}),
        }}

    RETURN_TYPES = ("STRING",)
    FUNCTION = "fun"
    CATEGORY = "O/text/operations"

    @ staticmethod
    def fun(text1, separator, text2):
        return (text1 + separator + text2,)
    
    @classmethod
    def IS_CHANGED(cls, *args, **kwargs):
        return float("NaN")

Comparing with the https://github.com/omar92/ComfyUI-QualityOfLifeSuit_Omar92/blob/ebcaad0edbfbd8783eb6ad3cb979f23ee3e71c5e/src/QualityOfLifeSuit_Omar92.py#L1189, the new code added IS_CHANGED, which is typical for some nodes. However now IsChangedCache.get is forced to call get_input_data, which at that point BasicCache.cache_key_set is not initialized yet, hence the assertion crash.

---

[FIXED]

Another bug when a node is being added but does not get executed. This occurs when the node has all of it inputs being optional (with may includes link inputs, not value inputs) AND when the same output being connected to two nodes: the process node and the prompt expand node. When this node get added to expand by prompt expand node, it somehow ignore such node and will no longer execute and this includes all subsequent process node.

Not sure how to process with this. The only way I could think is adding a way to force add a node to execution_list through the same expand dict by checking certain keys with the current codebase.

[asagi4]

I managed to trigger an error like this by running out of VRAM mid-generation and then trying to run another generation.

Traceback (most recent call last):                                                   
  File "/home/sd/.conda/envs/sd/lib/python3.11/threading.py", line 1045, in _bootstrap_inner                                                                              
    self.run()                                                                       
  File "/home/sd/.conda/envs/sd/lib/python3.11/threading.py", line 982, in run       
    self._target(*self._args, **self._kwargs)                                        
  File "/home/sd/git/ComfyUI/main.py", line 111, in prompt_worker                        e.execute(item[2], prompt_id, item[3], item[4])                                  
  File "/home/sd/git/ComfyUI/execution.py", line 476, in execute                     
    self.handle_execution_error(prompt_id, dynamic_prompt.original_prompt, current_ou
tputs, executed, error, ex)
  File "/home/sd/git/ComfyUI/execution.py", line 438, in handle_execution_error
    "current_outputs": error["current_outputs"],
                       ~~~~~^^^^^^^^^^^^^^^^^^^
KeyError: 'current_outputs'

ComfyUI then seems to get stuck unable to do anything.

I use ComfyBox as a frontend to ComfyUI. Didn't yet try reproducing this without it.

I fixed it by changing the code to use error.get("current_outputs", []) instead. I don't know why current_outputs is missing in this case, but using get makes ComfyUI recover from the OOM without having to restart.

[WeeBull]

Update: I've managed to test the PR now, and apart from having to disable rgthree, my flows seem to work fine. My fear that mapping over lists might be broken seems to be unfounded.

Original: I'm not able to test this at the moment, but does this preserve the behaviour that a node will map over lists given as inputs? Looking at the change it appears like it might have removed it.

For example, using the impact pack read prompts from file followed by unzip prompts will give you lists of +ve and -ve prompts, which a CLIP prompt encode will turn into lists of conditioning, which a ksampler will turn into lists of latents.

Also really useful for shmoo-ing parameters over ranges with CR Float / Integer range list.

[Seedsa]

ERROR:root:Failed to validate prompt for output 197:
ERROR:root:* (prompt):
ERROR:root:  - Return type mismatch between linked nodes: model, * != MODEL
ERROR:root:  - Return type mismatch between linked nodes: latent_image, * != LATENT
ERROR:root:* Efficient Loader 190:
ERROR:root:  - Return type mismatch between linked nodes: positive, * != STRING
ERROR:root:* KSampler (Efficient) 197:
ERROR:root:  - Return type mismatch between linked nodes: model, * != MODEL
ERROR:root:  - Return type mismatch between linked nodes: latent_image, * != LATENT
ERROR:root:Output will be ignored
ERROR:root:Failed to validate prompt for output 9:
ERROR:root:* (prompt):
ERROR:root:  - Return type mismatch between linked nodes: images, * != IMAGE
ERROR:root:* SaveImage 9:
ERROR:root:  - Return type mismatch between linked nodes: images, * != IMAGE
ERROR:root:Output will be ignored
invalid prompt: {'type': 'prompt_outputs_failed_validation', 'message': 'Prompt outputs failed validation', 'details': 'Return type mismatch between linked nodes: model, * != MODEL\nReturn type mismatch between linked nodes: latent_image, * != LATENT\nReturn type mismatch between linked nodes: images, * != IMAGE', 'extra_info': {}}

[doctorpangloss]

suppose i already have a way to distributed-compute whole workflows robustly and transparently inside python. with these changes, is it a small lift to distribute pieces of the graph / individual nodes? the idea would be to make it practicable to integrate many models together performantly - each individual model may take up the entire VRAM, but distributed on different machines. this is coming from the POV of having a working implementation.

if the signature of node execution were async, it would be a very small lift to parallelize individual node execution among consumers (aka workers). it would bring no changes to the current blocking behavior in ordinary 1-producer-1-consume.

[guill]

Update: I've managed to test the PR now, and apart from having to disable rgthree, my flows seem to work fine. My fear that mapping over lists might be broken seems to be unfounded.

@WeeBull Yeah, the intention is that functionality is maintained. I tested it via some contrived test cases, but it's not functionality I use much in my actual workflows. Good to hear that it seems to work for you!

@Seedsa As mentioned in the PR summary, this PR does not enable "*"-type sockets. You'll have to manually re-enable those via a patch just as you would on mainline. @comfyanonymous How do you feel about a command-line argument that enables "*" types so people don't have to circulate another patch file for it after this PR?

@doctorpangloss That should be a good bit easier with this architecture than the old one, though it's likely any old working code won't transfer over unchanged. In this execution model, nodes can already return ExecutionResult.SLEEPING to say "do other nodes that don't require my result and come back to me later". You could fairly easily kick off whatever remote process you want in that first call to Execute, add an artificial prerequisite to the node (that gets removed when you get the result of the remote process), and sleep in the meantime.

There might be a way to wrap the Execute call in such a way that normal Python async semantics do that automatically for you, but I'd have to do some research. Python isn't really my area of expertise. 😅

[guill]

@Trung0246 I believe the issue with ttN pipeLoader is actually a bug in the node. The declaration of "my_unique_id": "UNIQUE_ID" should be inside of the "hidden" category for it to actually function. Right now, it's doing nothing at all (and the my_unique_id argument will always be None). Honestly, an error with a callstack might be preferable to the current situation where it silently ignores that argument. I can add additional checks if people disagree though.

I'm unable to reproduce any of your other issues. Any chance you could upload some workflows? (I'm not sure what node pack the 'process' or 'prompt_expand' nodes are from.)

[guill]

Pasting a code block that @Trung0246 gave me on Matrix here so that it's documented:

class TautologyStr(str):
	def __ne__(self, other):
		return False

class ByPassTypeTuple(tuple):
	def __getitem__(self, index):
		if index > 0:
			index = 0
		item = super().__getitem__(index)
		if isinstance(item, str):
			return TautologyStr(item)
		return item

class StubNode:
	@classmethod
	def INPUT_TYPES(cls):
		return {
			"required": {
				"text": ("STRING", {
					"default": "TEST",
					"multiline": False
				}),
			},
			"optional": {
				"_stub_in": (TautologyStr("*"), ),
				"_stub": ("STUB_TYPE", )
			},
			"hidden": {
				"_id": "UNIQUE_ID",
				"_prompt": "PROMPT",
				"_workflow": "EXTRA_PNGINFO"
			}
		}
	
	RETURN_TYPES = ByPassTypeTuple(("*", "*"))
	RETURN_NAMES = ("_stub_out", "_stub_out_all")
	INPUT_IS_LIST = True
	OUTPUT_IS_LIST = (True, True)
	# OUTPUT_NODE = True
	FUNCTION = "execute"
	CATEGORY = "_for_testing"

	def execute(self, **kwargs):
		return (kwargs.get("text", ["???"]), kwargs.get("_stub_in", ["STUB"]))

[guill]

I believe all the issues reported in this PR so far have been addressed. Please let me know if you encounter any new ones (or I'm wrong about the existing reports being resolved).

[ricklove]

Getting this error in a complex workflow - working on identifying the source so I can upload a minimal test case:

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "D:\Projects\ai\comfyui\ComfyUI\execution.py", line 313, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "D:\Projects\ai\comfyui\ComfyUI\execution.py", line 191, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "D:\Projects\ai\comfyui\ComfyUI\execution.py", line 166, in map_node_over_list
    results.append(getattr(obj, func)(**input_dict))
TypeError: LoadImage.load_image() got an unexpected keyword argument 'upload'

[ricklove]

Getting this error in a complex workflow - working on identifying the source so I can upload a minimal test case:

ERROR:root:!!! Exception during processing !!!
ERROR:root:Traceback (most recent call last):
  File "D:\Projects\ai\comfyui\ComfyUI\execution.py", line 313, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "D:\Projects\ai\comfyui\ComfyUI\execution.py", line 191, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "D:\Projects\ai\comfyui\ComfyUI\execution.py", line 166, in map_node_over_list
    results.append(getattr(obj, func)(**input_dict))
TypeError: LoadImage.load_image() got an unexpected keyword argument 'upload'

Looks like just the Load Image node will cause this

Edit: I verified this is still happening after merging in the latest of master branch, so something is breaking LoadImage node

[hananbeer]

nice.

but... I actually liked workflows going only in one direction. it simplifies things a lot.
yea some things are more difficult without it. and if batching is possible it can just take too much ram.

I may be late here but if I may join the discussion:
I would approach it differently and use functions instead.
lets say you want to decrease denoise over time - instead of the proposed loop nodes over ksamplers, I would just prefer to pass a generator object as returned in python by enumerate(np.arange(1, 0, -0.2)) which evaluates to the list [1.0, 0.8, 0.6, 0.4, 0.2]
(a bit of a side track - since range() doesn't support float and np.arange isn't a generator I simply wrapped it with enumerate)

then the ksampler node (and any node for that matter) needs to unpack this generator.
in python it's just calling next(), e.g:

>>> enumerate(np.arange(1,0,-0.2))
<enumerate object at 0x0000028898F91A80>
>>> a=_
>>> next(a)
(0, 1.0)
>>> next(a)
(1, 0.8)
>>> next(a)
(2, 0.6000000000000001)
>>> next(a)
(3, 0.40000000000000013)
>>> next(a)
(4, 0.20000000000000018)
>>> next(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

it should probably just be an arbitrary function instead of generator, just wanted to use the range() example for the sake of making my point.

now nodes should treat every input as a function, a "normal" value (constant) will just be the identity function.
this solution is cleaner imo and reduces the complexity of loops.

the only issue is existing nodes don't expect functions, but perhaps a possible solution could be a new input type of type function and if the execution engine sees functions that are going to non-function input types, it will just unpack it automatically. (just take the first value and error if there is more than one value)

[LiChangyi]

@guill Good job, I found that now we are selecting one to execute based on the get_ready_nodes results. Why not execute them all together? Because during the running process, there may be some CPU dependent, some GPU dependent, and even interface requests. If parallel execution is possible, the speed of running graphs would be enormous.

[guill]

@LiChangyi You might be interested in this discussion: #3683

Parallel execution is definitely something I think we all want to see, but there are some challenges in accomplishing it -- especially since so many users are already running close to their machine's RAM/VRAM limits.

[mijuku233]

@guill

Nodes in loops don't update state.

As shown below, control_after_generate is set to decrement, but except for the first run, the subsequent seed value will not decrease, and the image preview will not update.

I tried to write a node that "outputs different values ​​in sequence", which depends on loops.
But the node in the loop does not update state, it can only output the same value, so I can't realize my idea.

My Workflow: loop.json

[Amorano]

@guill

Nodes in loops don't update state.

I tried to write a node that "outputs different values ​​in sequence", which depends on loops. But the node in the loop does not update state, it can only output the same value, so I can't realize my idea.

My Workflow: loop.json

You should use the examples from his repository. You need to accumulate the result in a list. You cannot preview the execution as it runs like that.

https://github.com/BadCafeCode/execution-inversion-demo-comfyui

hth

[mijuku233]

@guill

Nodes in loops don't update state.

I tried to write a node that "outputs different values ​​in sequence", which depends on loops. But the node in the loop does not update state, it can only output the same value, so I can't realize my idea.
My Workflow: loop.json

You should use the examples from his repository. You need to accumulate the result in a list. You cannot preview the execution as it runs like that.

https://github.com/BadCafeCode/execution-inversion-demo-comfyui

hth

I was using his example node to demonstrate the loop flaw.
This flaw prevented the node I was writing from working as expected.

[guill]

@mijuku233 "control_after_generate" means "update after the workflow executes", not "update after the node executes". If you want the seed to decrement with each iteration of the loop, you'll want to convert it to an input and use an Int Math operation on it (either adding remaining to the initial seed or subtracting 1 from the seed and passing it to the next iteration of the loop).

[mijuku233]

@guill
I just used "control_after_generate" to show you the problem.

There is a simple logic in my node code, which count +1 after each run so that I can use the count to switch different values.
I connected my node in loop node, similar to the ksampler in the gif display, and the returned value will not increment according to the loop.

    def __init__(self):
        self.count = 0

    def doit(self):

        self.count += 1

        return (self.count,)

[guill]

You generally shouldn't rely on nodes being stateful. Data stored on node objects should be limited to cached information that can be recalculated if necessary. If you rely on statefulness in the way you're doing now, results will change depending on what workflow you happened to run previously.

As to why you're seeing this within a single execution: nodes aren't actually executed multiple times in a loop. Iterations after the first are entirely different instances that map to the original node in the UI.

[mijuku233]

unique_id of new instances generated in a loop is abnormal

[Amorano]

unique_id of new instances generated in a loop is abnormal

If you are seeing multiple problems, I would suggest you make multiple NEW tickets.

This PR is already merged and this is just adding actual problems into a dead end ticket.

[acorderob]

This seems to be backwards, because selftest will have more elements than otherset if self has multiple types and we are comparing with only one type.

Also, shouldn't all this be available for custom nodes instead of being for tests?