Nested categories

[gregvari]

Atm. you can only go one two level deep in categories, which is quite (only if I'm putting many nodes into one package) limiting.

E: and you can't even do top-level nodes, now that I see it's basically Pack( List[Category( List[Node] )] )

What I'd propose is to just handle it almost as a file manager, where you can do folders and nodes (and they're marked as such to avoid confusion, like folders may get an arrow.)

Obv. this would require that the menu can stack these indefinitely.

[gregvari]

Obviously splitting things up into more nodes kinda works, but it's not the most ideal solution.

[KennedyRichard]

I've been indeed looking for ways to improve node hierarchy organization in Nodezator. It wouldn't be honest to say that the current design is ideal.

However, I simply didn't manage to come up with a better model yet. And I'm not saying that yours is not a good idea. It is just that there is other things to consider before deciding on changing the current structure adopted. I'll give more info/context on this.

First of all, things like the ones you just requested/suggested have been presented by other users as well. For instance, check this comment from Mr. Císař.

In it, he also expresses his desire to be able to have a more nested structure of the node packs, or as you worded, more levels.

I understand the desire behind it and, as I said, I do recognize there may be better ways to structure the node packs than as they are structured right now, but I don't think allowing more levels as you both suggested will actually improve the node pack functionality.

Certainly, as you pointed out, it would improve the organization of categories with many node script folders.

However, nodes in Nodezator are meant to be highly modular and completely independent from each other. They are considered completely autonomous applications, independent from each other. The category folders are actually just a cosmetic addition to give users a way to organize their nodes thematically.

In other node editors and similar visual programming apps, nodes or other visual representation of operations can be nested without harm because they are usually meant to always be used together anyways, never separated from each other, so it makes sense.

Adding this possibility to Nodezator would in the same way just constitute a mere formality, because the nodes further nested below an existing one would not be part of that node, but just independent nodes that only happen to be nested together for the sake of organization. They'd still be completely independent from each other, so much so that plucking the inner nodes would not harm the upper node at all. The upper nodes would just provide a "home" for the nodes lower in the hierarchy.

But again, this doesn't make sense, because nodes must be completely independent anyways, so the fact that a node script folder is holding another node script folder would just be for organization.

In other words, there's actually no hierarchy in Nodezator nodes. In fact, in addition to category folders, even node packs can be seen as a formality, because it just servers to hold nodes together, but the nodes themselves are completely independent from each other. There's no "node tree".

That is why the concept of nesting doesn't apply to Nodezator.

However, that's not to say that your request/suggestion as well as the others' are not legitimate. The need for some organization is a legitimate one. That's why for now I won't accept nor deny your request, but leave it here so other users can see it and share their opinions. I don't plan on doing anything about this soon, given our other priorities, like the need to implement subgraphs/group nodes, but rest assured that this is definitely something for which I care and will properly deal with as soon as the other priorities are dealt with. While we wait for that time to arrive, of course I'll be listening close to yours and any other feedback that pops up in the meantime, looking for alternatives that satisfy everyone as much as possible, under the nonnegotiable condition that they don't harm the simplicity, maintainability and power of Nodezator's design.

Speaking of simplicity, maintainability and power, don't you agree that up to this point I merely talked about why the node pack structure was designed like that, but didn't properly talk about the advantages of such design? As a creator/maintainer of a piece of software, it is also my duty to not only make sound design decisions, but also justify such decisions to the users when such decisions incur intrinsic costs.

I impose a lack of better organization to node packs, by requiring a fixed structure to node packs. Which are, then, the benefits of submitting users to this limitation? What do I offer users in return? For what purpose did I sacrifice a bit of organization?

The short answer is that by not allowing the node scripts to nest or even share resources I reinforce their complete independence, modularity and autonomy. This, in turn, facilitates the maximum interoperability between nodes across all node packs, resulting in a high degree of flexibility, customization and even scalability. What all those overused words mean in practice? Think of it like this: since every node in a node pack is completely autonomous and are gathered into categories just for the mere sake of organization, you can simply pluck a node script folder from an existing node pack and add it to your own node pack by just copying the folder into your own node pack (within an arbitrary category). It's a simple Ctrl+C/Ctrl+V and you are done.

This is substantially more easy and straightforward to do in node definitions (node packs) for Nodezator than it would be in node definitions from other node editors.

For instance, here is a possible scenario: imagine that you are using another node editor that allows or even requires users to define nodes that can be organized however the user wants and that can also share resources. If you have a set of node definitions (like a node pack) and want to use a specific node from a set of node definitions from another user, you now have several potential problems. You can't simply copy that definition from the other set. You'll actually have to dive into the source code to check whether that specific node definition uses resources shared among the nodes within that set of definitions. If that node does use some resources, than you'll also have to figure out which are such resources and copy them as well in your set of node definitions. Since your own set of definitions also have their own resources they share, you'll also have to make sure the node definition and respective resource(s) you brought into your set of definitions don't clash with the code/resources from your set of definitions. In other words, every change you make to your set of node definitions can potentially affect the rest of the set.

Now, something that is often overlook in software development is the fact that software often grows indefinitely. For simple sets of node definitions with only a handful of nodes, what I described earlier is not much work. Some functions or class definitions, a few resources here and there. It is not hard to figure out your way across the source and identify what you need to copy a node definition to your own set. However, as your work becomes more complex and the scope increases, you'll likely create many sets of node definitions, each with several nodes. You'll also may want to share some them. With time, you are likely to use node packs from other users as well. Now, imagine if you and the other users adopted a node editor that allowed node definitions of arbitrary structure and which shared resources among them. The task described in the scenario presented earlier would be even more daunting and sometimes prohibitive.

In Nodezator, because nodes are completely independent/autonomous/modular, you can simply copy the ones you want right away. Or you can study them and create your own version. The point is, because the nodes are isolated from each other, everything about a node is contained in the same spot and everything that node needs to operate is either present there or is received during execution via its inputs. Nodes in Nodezator are just like cards in a Pokémon Trading Card Game, for instance. No matter how they are categorized, each of them exist on their own as an indivisible unit while at the same time the game mechanics allow them to interact with each other.

Of course, if you wanted to use the entire foreign node definition in another node editor, it probably wouldn't be a problem, because then the entire set of definitions and shared resources would be entirely available. However, as I stated earlier, valuable and relevant software grows often, and you may find yourself in a position to need some specific behaviour from the foreign set of node definitions. When this happens, you'll want to simply copy it to your own set of node definitions and make the changes necessary. However, as we discussed earlier, it won't always be so easily if the node editor used allows arbitrary structures and shared resources.

Let's go even more practical by discussing the implications to an existing programming task. You are involved with AI content generation, right? Naturally, it is a virtually limitless area. Nodezator allows easy definition of nodes as well as almost automatic conversion of existing callables from imported libraries into nodes. Because Nodezator enforces modularity and independence, you can treat all existing nodes, the ones you create and the ones from others, as Lego® blocks. While this doesn't solve all problems, it at least immensely simplifies interaction between such nodes and any other work needed to be done on them, like copying, extending, etc.

That's where the flexibility, customization and scalability I mentioned earlier comes from. You can easily integrate other workflows or their individual nodes into your own AI generation graphs. Need to edit images? Need to create and customize graphs? Need to parse text? In Nodezator you'll be able to create your own nodes and/or use nodes from others with the confidence that all the existing nodes in the entire ecosystem are completely independent/modular, because Nodezator enforces this. Because of that, if the need arises, each of the existing nodes, regardless of their origin, can be more easily studied or modified/extended. Without this trait, node editors are just fancy GUIs to combine operations that may end up growing too complex to be manageable/maintainable in the long term. If the nodes are not completely independent, the node editor loses much of its power.

Granted, some nodes do share resources in a way. For instance, when they use import internally. Two distinct and independent nodes may import the matplotlib library, one to create a pie chart and the other to create a line chart. In this case, both of them pulls existing code from the same library. However, in this case, the shared resources doesn't constitute any harm to them, but in this context it is considered a more generally available resource. Stable and healthy libraries won't often change their APIs even across different versions, because every time they do so something from someone somewhere may break. Often, it does happen from time to time, but this is not a problem inherent to node editing, and not even to Python, but to programming and tech in general.

However, because third-party libraries like matplotlib are usually very stable, there's no problem in it being imported by different nodes, which thus constitutes and example of a resource being shared. In this case it is okay though, because the resource is a very general, widespread one and it is reasonable to expect some stability from it. It is like watching a tutorial on farming on Youtube. The tutorial may require you to acquire your own shovel to use on your land, but you and everyone watching will use the same sun, because it is a generally available and very stable resource, even though, scientifically speaking, it is expected to run out of fuel in a few billion years.

Not everything can be or should be made modular. Nodes in a node editing interface however, provide much more power when they are as much independent from each other as possible.

Sorry for the big wall of text. Don't feel compelled to comment back on each aspect I discussed or give any form of long reply. I like to be comprehensive on my replies to provide as much relevant information as possible, but it can feel overwhelming when people think I expect the same level of understanding and depth from their reply. The entire Indie Python project and its child projects like Nodezator are supposed to be places for learning and having fun, not to demand effort nor time from people. Just interact/reply at your own pace.

Thank you for your feedback, it is much appreciated. Please, post more ideas any time you want, no matter whether it is a very reasonable and legitimate one like this one or a seemingly absurd one. The discussion tab of the Nodezator project is a great place for ideas to clash respectfully and evolve as a result. This helps improve Nodezator.

I also write a lot because I love programming and open source, learning new things and meeting new, talented people. Thank you for being part of this community.

Peace

[gregvari]

I totally see how limiting nodes is beneficial now: just like how putting restrictions on yourself can improve creativity.

As you said, smaller and more modular nodes allow easier maintenance and changes later on.

Is there an upfront cost? Surely, if someone isn't used to this idea: but I can see how it's worth it in the long run.

Thanks for the detailed response (and how you related it to my personal use-case. 😉)

[GrimPixel]

I would circumvent the hierarchy by breaking everything down: each calculation process is a node; each node is named after the calculation process. This requires a label system as discussed in #73 so I can have an idea what they are meant to do; as well as the ability to have integrated nodes, like that Heron's formula one, where two nodes can be integrated into one.

[gregvari]

@GrimPixel Do you mean subgraphs with your last remark?

[GrimPixel]

Yes. Or it maybe be called a group of nodes with defined connections.