Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Major overhaul / better native integration #92

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft

Major overhaul / better native integration #92

wants to merge 15 commits into from

Conversation

city96
Copy link
Owner

@city96 city96 commented Dec 11, 2024

The plan is to do a full rewrite/refactor of this repo to have better integration with most of the native comfy code.
This should make things less fragile (and less cumbersome in general).

Progress/steps:

  • Remove HyDiT (already supported in base ComfyUI)
  • Rewrite PixArt base code
  • Add T5 for PixArt
  • Rewrite Sana base code
  • Add Gemma for Sana [mostly done, missing logic]
  • Rewrite DiT base code
  • Rewrite VAE loader
  • Add back LoRA support

Major changes:

  • Text encoders return comfy compatible CLIP objects instead of custom types
  • Auto config detection wherever possible to minimize user error
  • Single node with dropdown for all models instead of separate node sets per model
  • PixArt/DiT/etc models are loaded from unet (diffusion_models) folder instead of checkpoints folder

Other possible ideas/plans:

  • GGUF support if ComfyUI-GGUF is installed
  • Generic resolution select node with dropdown and slider/float input
  • Change to native comfy attention for PixArt/Sana
  • Add proper ControlNet for PixArt (current version never worked correctly)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant