Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(release): v0.0.1 add forge/svd support #1

Merged
merged 25 commits into from
Mar 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
92a317f
feat(forge): init a1111 forge
mnb3000 Mar 18, 2024
9703ee1
ci: add CI configs
mnb3000 Mar 18, 2024
0740f30
ci: add metadata labels to image
mnb3000 Mar 18, 2024
62acfb6
ci: update main publish workflow
mnb3000 Mar 18, 2024
a33bb33
fix(scripts): add separate ensure model functions
mnb3000 Mar 18, 2024
0170dd4
feat(sha256): add weight checksum validation after download
mnb3000 Mar 18, 2024
5b38bcf
fix(scripts): add workspace to model paths
mnb3000 Mar 18, 2024
8c7bab8
ci(diskSpace): leave large packages in for faster buildtime
mnb3000 Mar 18, 2024
e7e2a52
fix(scripts): fix local variables
mnb3000 Mar 18, 2024
5b1a0b8
fix(checksums): add checksums to image
mnb3000 Mar 18, 2024
4c23ab6
Revert "ci(diskSpace): leave large packages in for faster buildtime"
mnb3000 Mar 18, 2024
a9ad241
fix(scripts): update start script path
mnb3000 Mar 19, 2024
edcc4be
fix(checksums): create checksum folder
mnb3000 Mar 19, 2024
70dc832
feat(civitai): add civitai token support
mnb3000 Mar 19, 2024
64a516b
fix(checksums): update checksum formatting
mnb3000 Mar 19, 2024
beb7740
fix(scripts): update checksum flow
mnb3000 Mar 19, 2024
9d1f353
fix(dockerfile): pass civitai token arg
mnb3000 Mar 19, 2024
265099c
feat: add env check on startup
mnb3000 Mar 19, 2024
7750531
feat(scripts): add optional (`ENABLE_SDXL_TURBO`) sdxl turbo download
mnb3000 Mar 19, 2024
0c0b886
ci(release): add changelog generation
mnb3000 Mar 19, 2024
56e28d8
fix(scripts): fix svd downloader
mnb3000 Mar 19, 2024
20a0de4
ci(cache): clean docker cache without buildkit
mnb3000 Mar 19, 2024
a0cb49d
chore(cleanup): add default sdxl turbo value
mnb3000 Mar 19, 2024
f86dc6c
feat(scripts): add exit on failed checksum validation
mnb3000 Mar 19, 2024
4189ce9
chore: update readme
mnb3000 Mar 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/docker-publish-testing.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Docker publish testing

on:
push:
branches:
- "*"
- "!main"

jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Maximize build space
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
docker-images: false

- name: Clean docker image cache
run: |
docker image rm -f \
$( \
docker image ls -a | \
grep -v "buildkit" | \
grep -v "ubuntu.*22\.04" | \
awk 'NR>1 {print $3}'
)

- name: Checkout
uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: mnb3000/a1111-forge-svd-docker
tags: |
type=ref,event=branch
type=ref,event=branch,suffix=-${{ github.run_number }}

- name: Build and push
uses: docker/bake-action@v4
with:
files: |
./docker-bake.hcl
${{ steps.meta.outputs.bake-file }}
targets: default
push: true
env:
RELEASE: dev-build-${{ github.run_number }}
88 changes: 88 additions & 0 deletions .github/workflows/docker-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
name: Docker publish

on:
push:
branches:
- main

permissions:
contents: write

jobs:
release-on-push:
runs-on: ubuntu-latest
permissions:
contents: write
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- id: release
name: Release
uses: rymndhng/release-on-push-action@master
with:
bump_version_scheme: patch
use_github_release_notes: true

- name: Create changelog
uses: TriPSs/conventional-changelog-action@v5
with:
github-token: ${{ secrets.github_token }}
skip-tag: "true"
skip-bump: "true"
release-count: 20

build:
runs-on: ubuntu-latest
needs: release-on-push
steps:
- name: Maximize build space
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
docker-images: false

- name: Clean docker image cache
run: |
docker image rm -f \
$( \
docker image -a | \
grep -v "buildkit" | \
grep -v "ubuntu.*22\.04" | \
awk 'NR>1 {print $3}'
)

- name: Checkout
uses: actions/checkout@v4

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: mnb3000/a1111-forge-svd-docker
flavor: |
latest=auto
tags: |
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=ref,event=branch

- name: Build and push
uses: docker/bake-action@v4
with:
files: |
./docker-bake.hcl
${{ steps.meta.outputs.bake-file }}
targets: default
push: true
env:
RELEASE: ${{jobs.release-on-push.steps.release.outputs.tag_name}}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.idea
.DS_Store
*.safetensors
.env
170 changes: 67 additions & 103 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -11,48 +11,49 @@ ENV DEBIAN_FRONTEND=noninteractive \
RUN apt update && \
apt -y upgrade && \
apt install -y --no-install-recommends \
build-essential \
software-properties-common \
python3.10-venv \
python3-pip \
python3-tk \
python3-dev \
nodejs \
npm \
bash \
dos2unix \
git \
git-lfs \
ncdu \
nginx \
net-tools \
inetutils-ping \
openssh-server \
libglib2.0-0 \
libsm6 \
libgl1 \
libxrender1 \
libxext6 \
ffmpeg \
wget \
curl \
psmisc \
rsync \
vim \
zip \
unzip \
p7zip-full \
htop \
screen \
tmux \
bc \
pkg-config \
plocate \
libcairo2-dev \
libgoogle-perftools4 \
libtcmalloc-minimal4 \
apt-transport-https \
ca-certificates && \
build-essential \
software-properties-common \
python3.10-venv \
python3-pip \
python3-tk \
python3-dev \
nodejs \
npm \
bash \
coreutils \
dos2unix \
git \
git-lfs \
ncdu \
nginx \
net-tools \
inetutils-ping \
openssh-server \
libglib2.0-0 \
libsm6 \
libgl1 \
libxrender1 \
libxext6 \
ffmpeg \
wget \
curl \
psmisc \
rsync \
vim \
zip \
unzip \
p7zip-full \
htop \
screen \
tmux \
bc \
pkg-config \
plocate \
libcairo2-dev \
libgoogle-perftools4 \
libtcmalloc-minimal4 \
apt-transport-https \
ca-certificates && \
update-ca-certificates && \
apt clean && \
rm -rf /var/lib/apt/lists/* && \
Expand All @@ -72,26 +73,15 @@ RUN pip3 install --no-cache-dir torch==${TORCH_VERSION} torchvision torchaudio -
# Stage 2: Install applications
FROM base as setup

RUN mkdir -p /sd-models

# Add SDXL models and VAE
# These need to already have been downloaded:
# wget https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors
# wget https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/resolve/main/sd_xl_refiner_1.0.safetensors
# wget https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/resolve/main/sdxl_vae.safetensors
COPY sd_xl_base_1.0.safetensors /sd-models/sd_xl_base_1.0.safetensors
COPY sd_xl_refiner_1.0.safetensors /sd-models/sd_xl_refiner_1.0.safetensors
COPY sdxl_vae.safetensors /sd-models/sdxl_vae.safetensors

# Clone the git repo of the Stable Diffusion Web UI by Automatic1111
# and set version
WORKDIR /
ARG WEBUI_VERSION
RUN git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git && \
cd /stable-diffusion-webui && \
RUN git clone https://github.com/lllyasviel/stable-diffusion-webui-forge.git && \
cd /stable-diffusion-webui-forge && \
git checkout tags/${WEBUI_VERSION}

WORKDIR /stable-diffusion-webui
WORKDIR /stable-diffusion-webui-forge
RUN python3 -m venv --system-site-packages /venv && \
source /venv/bin/activate && \
pip3 install --no-cache-dir torch==${TORCH_VERSION} torchvision torchaudio --index-url ${INDEX_URL} && \
Expand All @@ -106,69 +96,38 @@ RUN source /venv/bin/activate && \
python3 -m install-automatic --skip-torch-cuda-test && \
deactivate

# Cache the Stable Diffusion Models
# SDXL models result in OOM kills with 8GB system memory, need 30GB+ to cache these
RUN source /venv/bin/activate && \
python3 cache-sd-model.py --no-half-vae --no-half --xformers --use-cpu=all --ckpt /sd-models/sd_xl_base_1.0.safetensors && \
python3 cache-sd-model.py --no-half-vae --no-half --xformers --use-cpu=all --ckpt /sd-models/sd_xl_refiner_1.0.safetensors && \
deactivate

# Clone the Automatic1111 Extensions
RUN git clone --depth=1 https://github.com/Mikubill/sd-webui-controlnet.git extensions/sd-webui-controlnet && \
git clone --depth=1 https://github.com/deforum-art/sd-webui-deforum.git extensions/deforum && \
git clone --depth=1 https://github.com/ashleykleynhans/a1111-sd-webui-locon.git extensions/a1111-sd-webui-locon && \
git clone --depth=1 https://github.com/Gourieff/sd-webui-reactor.git extensions/sd-webui-reactor && \
RUN git clone --depth=1 https://github.com/deforum-art/sd-forge-deforum.git extensions/deforum && \
git clone --depth=1 https://github.com/zanllp/sd-webui-infinite-image-browsing.git extensions/infinite-image-browsing && \
git clone --depth=1 https://github.com/Uminosachi/sd-webui-inpaint-anything.git extensions/inpaint-anything && \
git clone --depth=1 https://github.com/Bing-su/adetailer.git extensions/adetailer && \
git clone --depth=1 https://github.com/civitai/sd_civitai_extension.git extensions/sd_civitai_extension && \
git clone --depth=1 https://github.com/BlafKing/sd-civitai-browser-plus.git extensions/sd-civitai-browser-plus

# Install dependencies for Deforum, ControlNet, ReActor, Infinite Image Browsing,
# After Detailer, and CivitAI Browser+ extensions
# Install dependencies for Deforum Forge, Infinite Image Browsing, CivitAI Browser+ extensions
RUN source /venv/bin/activate && \
pip3 install basicsr && \
cd /stable-diffusion-webui/extensions/sd-webui-controlnet && \
pip3 install -r requirements.txt && \
cd /stable-diffusion-webui/extensions/deforum && \
pip3 install -r requirements.txt && \
cd /stable-diffusion-webui/extensions/sd-webui-reactor && \
cd /stable-diffusion-webui-forge/extensions/deforum && \
pip3 install -r requirements.txt && \
pip3 install onnxruntime-gpu && \
cd /stable-diffusion-webui/extensions/infinite-image-browsing && \
cd /stable-diffusion-webui-forge/extensions/infinite-image-browsing && \
pip3 install -r requirements.txt && \
cd /stable-diffusion-webui/extensions/adetailer && \
python3 -m install && \
cd /stable-diffusion-webui/extensions/sd_civitai_extension && \
cd /stable-diffusion-webui-forge/extensions/sd_civitai_extension && \
pip3 install -r requirements.txt && \
deactivate

# Install dependencies for inpaint anything extension
RUN source /venv/bin/activate && \
pip3 install segment_anything lama_cleaner && \
deactivate

# Install dependencies for Civitai Browser+ extension
RUN source /venv/bin/activate && \
cd /stable-diffusion-webui/extensions/sd-civitai-browser-plus && \
cd /stable-diffusion-webui-forge/extensions/sd-civitai-browser-plus && \
pip3 install send2trash ZipUnicode fake-useragent && \
deactivate

# Add inswapper model for the ReActor extension
RUN mkdir -p /stable-diffusion-webui/models/insightface && \
cd /stable-diffusion-webui/models/insightface && \
wget https://github.com/facefusion/facefusion-assets/releases/download/models/inswapper_128.onnx

# Configure ReActor to use the GPU instead of the CPU
RUN echo "CUDA" > /stable-diffusion-webui/extensions/sd-webui-reactor/last_device.txt

# Install Jupyter, gdown and OhMyRunPod
RUN pip3 install -U --no-cache-dir jupyterlab \
jupyterlab_widgets \
ipykernel \
ipywidgets \
gdown \
OhMyRunPod
jupyterlab_widgets \
ipykernel \
ipywidgets \
gdown \
OhMyRunPod

# Install RunPod File Uploader
RUN curl -sSL https://github.com/kodxana/RunPod-FilleUploader/raw/main/scripts/installer.sh -o installer.sh && \
Expand All @@ -192,15 +151,14 @@ RUN curl -s https://packagecloud.io/install/repositories/ookla/speedtest-cli/scr
apt install speedtest

# Install CivitAI Model Downloader
RUN git clone --depth=1 https://github.com/ashleykleynhans/civitai-downloader.git && \
mv civitai-downloader/download.sh /usr/local/bin/download-model && \
chmod +x /usr/local/bin/download-model
COPY scripts/civitai-download.sh /usr/local/bin/download-model
RUN chmod +x /usr/local/bin/download-model

# Copy Stable Diffusion Web UI config files
COPY a1111/relauncher.py a1111/webui-user.sh a1111/config.json a1111/ui-config.json /stable-diffusion-webui/
COPY a1111/relauncher.py a1111/webui-user.sh a1111/config.json a1111/ui-config.json /stable-diffusion-webui-forge/

# ADD SDXL styles.csv
ADD https://raw.githubusercontent.com/Douleb/SDXL-750-Styles-GPT4-/main/styles.csv /stable-diffusion-webui/styles.csv
ADD https://raw.githubusercontent.com/Douleb/SDXL-750-Styles-GPT4-/main/styles.csv /stable-diffusion-webui-forge/styles.csv

# Remove existing SSH host keys
RUN rm -f /etc/ssh/ssh_host_*
Expand All @@ -217,10 +175,16 @@ ENV TEMPLATE_VERSION=${RELEASE}
ARG VENV_PATH
ENV VENV_PATH=${VENV_PATH}

ENV ENABLE_SDXL_TURBO="false"

# Copy the scripts
WORKDIR /
COPY --chmod=755 scripts/* ./

# Copy the checksums
RUN mkdir -p ./checksums
COPY checksums/* ./checksums

# Start the container
SHELL ["/bin/bash", "--login", "-c"]
CMD [ "/start.sh" ]
Loading
Loading