Skip to content

Sxela/WarpFusion

BranchesTags

Repository files navigation

WarpFusion

WarpFusion

Latest public version:

If you find a public version elsewhere, before running as admin/root, make sure to check it for malware by comparing it to the latest notebook in this repo.

Greatly inspired by Cameron Smith's neural-style-tf

Example videos

Example videos and settings

Guides made by users:

Local installation guide for Windows (venv)

Run once per notebook version (recommended)

  1. Create a folder for WarpFusion. It's recommended to have a general folder for WarpFusion and subfolders for each version. Like C:\code\WarpFusion\0.16.11\ for version 0.16.11
  2. Download install.bat and save it into your WarpFolder, C:\code\WarpFusion\0.16.11\ in this example.
  3. Run install.bat. It will download and install python, git, and create a virtual python environment called "env" inside our folder and install dependencies, required to run the notebook and jupyter server for local colab. When git install window appears, use the default settings. The installation will contiinue after you install git.
  4. Download run.bat and save it into your WarpFolder, C:\code\WarpFusion\0.16.11\ in this example.

Run to launch

  1. Execute run.bat. It will activate the environment and start jupyter server.
  2. After the server has launched, go to https://colab.research.google.com
  3. Click File -> Upload Notebook and upload the *.ipynb file
  4. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
  5. Select "connect to a local runtime" and paste the URL that will be generated below, which looks like "http://localhost:8888/?token=somenumbers"
  6. Click "Connect" and CTRL+F9 to run all cells.

Local installation guide for Linux-Ubuntu 22.04 (venv):

Pre-requisites:

  • Make sure that Ubuntu packages for CUDA toolkit and the latest NVIDIA utils are installed. Check using nvidia-smi command.

  • ⚠️ Warning: Destructive Process Ahead ⚠️

    Clean Python Environment:

    • If you haven't followed best practices for Python virtual environments, you may want to clean your system.

    • Be warned, this is a destructive process and will remove all Python packages installed in the global environment.

      pip freeze > uninstall.txt
pip uninstall -r uninstall.txt
sudo pip freeze > uninstall.txt
sudo pip uninstall -r uninstall.txt
rm -rf ~/.cache

Installation Steps:

  1. Choose Directory:

    • Open a terminal and navigate to your home directory or a directory of your choice.
    cd $HOME or cd ~

  2. Clone Repository:

    • Clone the WarpFusion repository.
    git clone https://github.com/WarpFusion/WarpFusion.git

  3. Navigate to Folder:

    • Enter the WarpFusion directory.
    cd WarpFusion

  4. Run the Linux Installer:

    • Make the script executable and run it.
    chmod +x linux_install.sh
./linux_install.sh
    • Script will prompt you to enter a "version" to create your working folder, this can be any name you choose as at will append to "WarpFusion", ie: "WarpFusion0.23.11

Run to launch

  1. Navigate to your WarpFusion(version) folder and execute the run script: 
    cd $HOME/WarpFusion(version)
./run.sh
  2. After the server has launched, go to https://colab.research.google.com
  3. Click File -> Upload Notebook and upload the *.ipynb file
  4. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
  5. Select "connect to a local runtime" and paste the URL that will be generated below, which looks like "http://localhost:8888/?token=somenumbers"
  6. Click "Connect" and CTRL+F9 to run all cells.

Troubleshoot python virtual environment issues

  • Delete your python virtual environment "warpenv" and re-run the running the script (backup your models, images and videos just in case). 
    cd $HOME/WarpFusion(version)
rm -rf warpenv

Docker install

Run once to install (and once per notebook version)

  1. Create a folder for warp, for example d:\warp
  2. Download Dockerfile and docker-compose.yml to d:\warp
  3. Edit docker-compose.yml so that volumes point to your model, init_images, images_out folders that are outside of the warp folder. For example, d:\models\:/content/models will expose d:\models as /content/models to the notebook
  4. Download and install docker from here - https://docs.docker.com/get-docker/
  5. Run docker-compose up --build inside the warp folder.
  6. Go to https://colab.research.google.com
  7. Click File -> Upload Notebook and upload the *.ipynb file
  8. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
  9. Select "connect to a local runtime" and paste the token that was generated in your docker container, but leave the url as localhost. Should look like "http://localhost:8888/?token=somenumbers"
  10. Click "Connect" and CTRL+F9 to run all cells.

Run to launch

  1. Run docker-compose up inside the warp folder.
  2. Go to https://colab.research.google.com
  3. File -> open notebook -> open your previouslty uploaded notebook
  4. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
  5. Select "connect to a local runtime" and paste the token that was generated in your docker container, but leave the url as localhost. Should look like "http://localhost:8888/?token=somenumbers"
  6. Click "Connect" and CTRL+F9 to run all cells.

Credits

This notebook uses:

Stable Diffusion by CompVis & StabilityAI
K-diffusion wrapper by Katherine Crowson
RAFT model by princeton-vl
Consistency Checking (legacy) from maua
Color correction from pengbo-learn
Auto brightness adjustment from progrockdiffusion

AUTOMATIC1111: weighted prompt keywords, lora, embeddings, attention hacks
Reconstructed noise - based on changes suggested by briansemrau

ControlNet
TemporalNet, Controlnet Face and lots of other controlnets (check model list)
BLIP by SalesForce
RobustVideoMatting (as external cli package)
CLIP
FreeU Hack
Experimental ffmpeg Deflicker
Dw pose estimator
SAMTrack Segment-and-Track-Anything (with cli my wrapper and edits)
ComfyUI: sdxl controlnet loaders, control loras
animatediff base
animatediff wrapper for compvis models from comfyui-animatediff
IP Adapters implementation from sd-webui-controlnet


DiscoDiffusion legacy credits:

Original notebook by Somnai, Adam Letts and lots of other awesome people!

Turbo feature by Chris Allen

Improvements to ability to run on local systems, Windows support, and dependency installation by HostsServer

Warp and custom model support by Alex Spirin

Citation

If you find this code useful for your research, please cite:

@misc{Spirin2022,
  author = {Spirin, Alex},
  title = {warpfusion},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Sxela/WarpFusion}},
}

About

WarpFusion

Topics

video vid2vid img2img text2video stablediffusion video2video

Resources

Readme

License

View license
Activity

Stars

978 stars

Watchers

17 watching

Forks

106 forks
Report repository

Releases 18

v0.24 Latest
Oct 6, 2024
+ 17 releases

Sponsor this project

Packages

No packages published

Contributors 7

Languages