Merge pull request #67 from Visionatrix/feat/models_catalog_editor

Model Catalog Editor UI tool

.pre-commit-config.yaml

@@ -11,10 +11,11 @@ repos:
     rev: 5.13.2
     hooks:
     -   id: isort
-        files: scripts/
+        files: (scripts/|models_catalog_editor.py)
+        args: ["--profile", "black"]
 
 -   repo: https://github.com/psf/black
     rev: 24.10.0
     hooks:
     -   id: black
-        files: scripts/
+        files: (scripts/|models_catalog_editor.py)

README.md

@@ -16,8 +16,9 @@ For any problems with Visionatrix or suggestions for improvement, go to the [mai
   - Flows Developing:
     - [Vix Workflows](https://visionatrix.github.io/VixFlowsDocs/FlowsDeveloping/vix_workflows/)
     - [Technical information](https://visionatrix.github.io/VixFlowsDocs/FlowsDeveloping/technical_information/)
-    - [Gated Models](https://visionatrix.github.io/VixFlowsDocs/FlowsDeveloping/gated_models/)
     - [Creating Workflows](https://visionatrix.github.io/VixFlowsDocs/FlowsDeveloping/comfyui_vix_migration/)
+    - [Models Catalog](https://visionatrix.github.io/VixFlowsDocs/FlowsDeveloping/models_catalog/)
+    - [Gated Models](https://visionatrix.github.io/VixFlowsDocs/FlowsDeveloping/gated_models/)
   - Common information:
     - [FAQ](https://visionatrix.github.io/VixFlowsDocs/faq/)
     - [Hardware FAQ](https://visionatrix.github.io/VixFlowsDocs/hardware_faq/)

docs/FlowsDeveloping/comfyui_vix_migration.md

@@ -93,7 +93,7 @@ that you can use as example:
 
 Visionatrix simplifies and automates the process of downloading the models.
 
-As the third step of the migration, you need to map the models that are used in your workflow (see [models-mapping](./vix_workflows.md#automatic-models-mapping)).
+You need to ensure that models used in workflow are known to Visionatrix, see [models catalog](./models_catalog.md)
 
 ---
 

docs/FlowsDeveloping/gated_models.md

@@ -2,41 +2,35 @@
 title: Gated Models
 ---
 
-It often happens that the model you are using is not available for
-download without authentication. These are referred to as [Gated Models](https://huggingface.co/docs/hub/models-gated).
+Sometimes, the model you want to use requires authentication to access it. These are referred to as [Gated Models](https://huggingface.co/docs/hub/models-gated).
 
-Flows with such models have a separate mark in the Visionatrix UI.
+Flows with such models are distinctly marked in the Visionatrix UI.
 
-To be able to install such a flow, you need to specify an `Access Token`
+To install such a flow, you need to provide an `Access Token` for HuggingFace or an `API Key` for CivitAI.
 
-!!! note
+### HuggingFace Token
 
-    Currently, only HuggingFace Access Tokens are supported.
+Steps to access gated models from HuggingFace:
 
+1. Register on [HuggingFace](https://huggingface.co) if you are not already registered.
+2. Generate an access token in the settings of HuggingFace (click on your icon → Settings → Access Tokens).
+3. Click `Set Permissions` for the token after generation and select `Read access to contents of all public gated repos you can access`.
+4. Enter this access token in the Visionatrix settings.
+5. Gain access to the model on your account by visiting its page (you can click on the model from the Visionatrix UI) and filling out the required form.
 
-Steps to Access Gated Models:
+### CivitAI API Key
 
-1.  Register on [HuggingFace](https://huggingface.co) if you are not
-    already registered
-2.  Gain access to the model on your account by going to its page (you
-    can click on the model from Visionatrix UI) and filling out the form
-3.  Generate an access token in the settings of HuggingFace (click on
-    your icon -\> settings -\> access tokens)
-4.  Click on `Set Permissions` of the token after generation and select
-    `Read access to contents of all public gated repos you can access`
-5.  Go to the Visionatrix settings and enter this access token
+Steps to access gated models from CivitAI:
 
-Alternatively, you can set an environment variable named `HF_AUTH_TOKEN`
-with the token value, but this requires setting up the environment
-variable for each worker if you have many of them.
+1. Register on [CivitAI](https://civitai.com) if you are not already registered.
+2. Create an API key in the settings of CivitAI (click on your icon → Add API Key).
+3. Enter this API key in the Visionatrix settings.
+4. Gain access to the model on your account by visiting its page (you can click on the model from the Visionatrix UI) and filling out the required form.
 
-#### I'm a user and want to connect my own worker to process flows with closed models.
+#### Connecting a Worker to Process Flows with Gated Models
 
-As user's workers cannot receive global access tokens from the server
-to avoid leaks, you have two options:
+If you want to connect your own worker to process flows with gated models, note that user workers cannot receive global access tokens from the server to prevent leaks. You have two options:
 
-1.  Download the model yourself and place it in the folder specified in
-    `models_catalog.json` under the `save_path` key.
-2.  Set the `HF_AUTH_TOKEN` environment variable with your own public
-    access token, and the worker will be able to install flows with
-    gated models.
+1. Download the model yourself and place it in the folder specified in `models_catalog.json` under the `filename` or `types` keys.
+2. Set the `HF_AUTH_TOKEN` environment variable with your own public access token. The worker will then be able to install flows with gated models from HuggingFace.
+3. For CivitAI, set the environment variable `CA_API_KEY`.

docs/FlowsDeveloping/models_catalog.md

@@ -0,0 +1,156 @@
+---
+title: Adding New Model to Catalog
+---
+
+## Introduction
+
+Visionatrix uses a `models_catalog.json` file to automatically map nodes in ComfyUI workflows to their corresponding models, ensuring a seamless user experience.
+
+If you use a model that is missing in the current catalog, you can now easily add it using our new Model Catalog Editor tool.
+
+This tool helps you create consistent and accurate entries in `models_catalog.json` by guiding you through setting the model’s URL, homepage, filename, hash, and classification ("types").
+
+---
+
+## Optional Auth Tokens
+
+If you have authentication tokens for [Hugging Face](https://huggingface.co/) or [CivitAI](https://civitai.com/), the tool can use them to fetch metadata and check gated models.
+It will attempt to read these tokens from the Visionatrix database if available.
+
+The script looks for the database in the `../Visionatrix` directory, so if both repositories are cloned in the same parent directory, the setup should work seamlessly.
+
+!!! note
+
+    Lack of access to the database or missing tokens does not prevent adding models, but certain features (like checking gated models) may be limited.
+
+---
+
+## Launching the Editor
+
+The `models_catalog_editor.py` file is located in the `VixFlowsDocs` repository at its root. You will also find a `requirements_catalog_editor.txt` file there. To start:
+
+1. **Install Dependencies** (if needed):
+
+   ```bash
+   pip install -r requirements_catalog_editor.txt
+   ```
+
+2. **Run the Editor**:
+
+   ```bash
+   python models_catalog_editor.py
+   ```
+
+This opens a GUI window:
+
+![Model Catalog Editor](../assets/models_catalog_editor_empty.png)
+
+---
+
+## Using the Model Catalog Editor
+
+### Step 1: Provide the Model Source URL
+
+In the "URL" field, enter the direct link to your model. Supported sources:
+
+- **Hugging Face**: e.g., `https://huggingface.co/{user}/{repo}/blob/main/model.safetensors`
+  *The editor automatically corrects `/blob/` links to `/resolve/` if necessary.*
+
+- **CivitAI**: Provide a link that includes a `modelVersionId` query parameter or a link to a model/version page. The tool will attempt to fetch metadata and files associated with that model.
+
+Click **Process**. The editor then tries to:
+
+- Fetch a homepage URL.
+- Propose a default filename.
+- Check if the model is gated (requires auth token).
+- For CivitAI, it lists all available files and their hashes, prompting you to choose one if multiple are found. It also attempts to determine the correct model `type` from the metadata.
+
+---
+
+### Step 2: Confirm or Edit the Download URL, Homepage, and Filenames
+
+- **Download URL**: The tool sets this automatically for known sources. You can adjust it if needed.
+- **Homepage**: For Hugging Face, this is usually the repository root. For CivitAI, it's the model’s main page. You can also enter a custom homepage if necessary.
+- **Filenames**:
+  The editor attempts to determine a good filename. If your model came from Hugging Face and also exists on CivitAI, you might see a "HugFace name" and a "CivitAI name".
+  If the suggested filenames are not suitable (e.g., too generic like `model.safetensors`), provide a "Force Filename" to ensure uniqueness and clarity.
+
+---
+
+### Step 3: Verify the Hash
+
+The editor tries to fetch a SHA256 hash for the model:
+
+- For Hugging Face models with a supported configuration, it reads the `X-Linked-ETag` header as a hash.
+- For CivitAI models, it uses the provided file hash from the metadata.
+
+If the hash is not found, enter it manually in the "Hash" field. The hash is crucial for ensuring model integrity and facilitating quick checks by Visionatrix.
+
+---
+
+### Step 4: Set Model "Types"
+
+"Types" determines where the model file will be stored inside ComfyUI’s directory structure. Common types include:
+
+- `checkpoints`
+- `diffusion_models`
+- `loras`
+- `controlnet`
+- `embeddings`
+- `vae`
+- ... and more.
+
+The tool tries to infer a type from the model’s metadata. However, for certain models (like SD3.5 or Flux models), the correct type may not be determined automatically. In such cases, **you must manually select the correct type** from the checkboxes.
+
+**Tip**: If unsure, consider:
+
+- `diffusion_models` or `checkpoints` for base diffusion models.
+- `loras` for LoRA-based models.
+- `controlnet` for ControlNet-based models.
+- `embeddings` for textual inversion embeddings.
+
+---
+
+### Step 5: Set Gated (if Needed)
+
+If the model is gated (e.g., private or requires a Hugging Face token to download), check the "Gated" box. This ensures Visionatrix knows it needs an authorization token for downloading.
+
+---
+
+### Step 6: Define the Regexes
+
+Regexes help Visionatrix identify which model entry applies to a given node in a ComfyUI workflow. You can set:
+
+- `class_type` regex: Matches the ComfyUI node’s `class_type`.
+- `input_name` regex: Matches the input parameter name of the node parameter that holds the model filename.
+- `input_value` regex: **Required** to match the actual input value from the workflow (e.g., the filename or part of it).
+
+**Important**:
+- `input_value` is mandatory. It should match the final filename or a portion of it.
+- Use a pattern that is flexible enough to capture variations in filenames, but not so broad it conflicts with other models.
+
+For example, if your filename is `my-special-model.safetensors`, a suitable `input_value` might be:
+
+```regex
+(?i)(?:[^\/\\]*[\/\\]?)?my-special-model\.safetensors$
+```
+
+If you have multiple filenames (like a Hugging Face name and a CivitAI name), the tool tries to create a regex that matches them all. Adjust the regex if necessary to ensure uniqueness and correctness.
+
+---
+
+### Step 7: Save the Entry
+
+Once you have verified all fields:
+
+- Click **Save**.
+- Choose or confirm a unique model name key. The tool ensures no conflicts with existing keys. If there's a hash conflict, you can overwrite or rename the key.
+- On successful save, the model entry is added to `models_catalog.json`.
+
+---
+
+## Conclusion
+
+With this utility, adding new models has become easy and efficient.
+
+After saving your changes locally, consider sharing your `models_catalog.json` updates by submitting a pull request to the VixFlowsDocs repository.

docs/FlowsDeveloping/technical_information.md

@@ -33,6 +33,7 @@ Visionatrix by default install and update these nodes:
 > -   [ComfyUI-PhotoMaker-Plus](https://github.com/Visionatrix/ComfyUI-PhotoMaker-Plus)
 > -   [ComfyUI-VideoHelperSuite](https://github.com/Visionatrix/ComfyUI-VideoHelperSuite)
 > -   [ComfyUI-Frame-Interpolation](https://github.com/Visionatrix/ComfyUI-Frame-Interpolation)
+> -   [ComfyUI-KJNodes](https://github.com/Visionatrix/ComfyUI-KJNodes)
 
 We are gradually expanding the list.
 

docs/FlowsDeveloping/vix_workflows.md

@@ -30,13 +30,11 @@ file.
 By default, it is taken and updated from the Visionatrix repository on
 GitHub, in case you add a new flow and need to add new model mappings
 you can change its path using an environment variable to a local file
-path.
+path or add additional places from where to fetch it.
 
 !!! note
 
-    We hope that after you add something locally, you will open a pull
-    request so that the community can benefit from it.
-
+    UI for easily adding models without going into too much detail, you can find it on this [documentation page](./models_catalog.md).
 
 The file structure consists of a set of objects, each describing a
 ComfyUI Node class that loads or uses a model.
@@ -83,10 +81,11 @@ Together, `types` and `filename` should provide enough information to correctly
 
 ### "filename"
 
-Overrides the default file name for the model. This is particularly useful when:
+Overrides the default file name for the model.
+
+This is particularly useful when the model has a generic name (e.g., `"model.safetensors"`) that could conflict with others.
 
-- The model is hosted on platforms like "CivitAI" where the name cannot be determined from the URL without starting the download.
-- The model has a generic name (e.g., `"model.safetensors"`) that could conflict with others. Using a unique name avoids such conflicts.
+Using a unique name avoids such conflicts.
 
 ### "url"
 

docs/index.md

@@ -15,8 +15,9 @@ Flows Developing:
 
   - [Vix Workflows](FlowsDeveloping/vix_workflows.md)
   - [Technical Information](FlowsDeveloping/technical_information.md)
-  - [Gated Models](FlowsDeveloping/gated_models.md)
   - [ComfyUI to Vix Workflows Migration](FlowsDeveloping/comfyui_vix_migration.md)
+  - [Models Catalog](FlowsDeveloping/models_catalog.md)
+  - [Gated Models](FlowsDeveloping/gated_models.md)
 
 Integrations Manual:
 

mkdocs.yml

@@ -47,8 +47,9 @@ nav:
   - Flows Developing:
       - Overview: FlowsDeveloping/vix_workflows.md
       - Technical Information: FlowsDeveloping/technical_information.md
-      - Gated Models: FlowsDeveloping/gated_models.md
       - ComfyUI to Vix Workflows Migration: FlowsDeveloping/comfyui_vix_migration.md
+      - Models Catalog: FlowsDeveloping/models_catalog.md
+      - Gated Models: FlowsDeveloping/gated_models.md
   - Integrations Manual:
       - Getting started: IntegrationsManual/getting_started.md
   - FAQ: faq.md