Skip to content

GitLab

Commit 73939beb authored by Samuel Tesfai's avatar Samuel Tesfai
Browse files

Merge github.com:mit-han-lab/nunchaku into migrate_tinychat

parents 91c7b53c bfd9aa3a
Hide whitespace changes
Inline Side-by-side
Showing with 459 additions and 101 deletions
README.md
View file @ 73939beb
...@@ -4,6 +4,7 @@ Nunchaku is an inference engine designed for 4-bit diffusion models, as demonstr ...@@ -4,6 +4,7 @@ Nunchaku is an inference engine designed for 4-bit diffusion models, as demonstr
### [Paper](http://arxiv.org/abs/2411.05007) | [Project](https://hanlab.mit.edu/projects/svdquant) | [Blog](https://hanlab.mit.edu/blog/svdquant) | [Demo](https://svdquant.mit.edu) ### [Paper](http://arxiv.org/abs/2411.05007) | [Project](https://hanlab.mit.edu/projects/svdquant) | [Blog](https://hanlab.mit.edu/blog/svdquant) | [Demo](https://svdquant.mit.edu)
- **[2025-02-14]** 🔥 [LoRA conversion script](nunchaku/convert_lora.py) is now available!
- **[2025-02-11]** 🎉 **[SVDQuant](http://arxiv.org/abs/2411.05007) has been selected as a ICLR 2025 Spotlight! FLUX.1-tools Gradio demos are now available!** Check [here](#gradio-demos) for the usage details! Our new [depth-to-image demo](https://svdquant.mit.edu/flux1-depth-dev/) is also online—try it out! - **[2025-02-11]** 🎉 **[SVDQuant](http://arxiv.org/abs/2411.05007) has been selected as a ICLR 2025 Spotlight! FLUX.1-tools Gradio demos are now available!** Check [here](#gradio-demos) for the usage details! Our new [depth-to-image demo](https://svdquant.mit.edu/flux1-depth-dev/) is also online—try it out!
- **[2025-02-04]** **🚀 4-bit [FLUX.1-tools](https://blackforestlabs.ai/flux-1-tools/) is here!** Enjoy a **2-3× speedup** over the original models. Check out the [examples](./examples) for usage. **ComfyUI integration is coming soon!** - **[2025-02-04]** **🚀 4-bit [FLUX.1-tools](https://blackforestlabs.ai/flux-1-tools/) is here!** Enjoy a **2-3× speedup** over the original models. Check out the [examples](./examples) for usage. **ComfyUI integration is coming soon!**
- **[2025-01-23]** 🚀 **4-bit [SANA](https://nvlabs.github.io/Sana/) support is here!** Experience a 2-3× speedup compared to the 16-bit model. Check out the [usage example](./examples/sana_1600m_pag.py) and the [deployment guide](app/sana/t2i) for more details. Explore our live demo at [svdquant.mit.edu](https://svdquant.mit.edu)! - **[2025-01-23]** 🚀 **4-bit [SANA](https://nvlabs.github.io/Sana/) support is here!** Experience a 2-3× speedup compared to the 16-bit model. Check out the [usage example](./examples/sana_1600m_pag.py) and the [deployment guide](app/sana/t2i) for more details. Explore our live demo at [svdquant.mit.edu](https://svdquant.mit.edu)!
...@@ -43,7 +44,7 @@ SVDQuant is a post-training quantization technique for 4-bit weights and activat ...@@ -43,7 +44,7 @@ SVDQuant is a post-training quantization technique for 4-bit weights and activat
* Ensure your CUDA version is **≥ 12.2 on Linux** and **≥ 12.6 on Windows**. * Ensure your CUDA version is **≥ 12.2 on Linux** and **≥ 12.6 on Windows**.
* For Windows user, please refer to [this issue](https://github.com/mit-han-lab/nunchaku/issues/6) for the instruction. Please upgrade your MSVC compiler to the latest version. * For Windows users, please refer to [this issue](https://github.com/mit-han-lab/nunchaku/issues/6) for the instruction. Please upgrade your MSVC compiler to the latest version.
* We currently support only NVIDIA GPUs with architectures sm_86 (Ampere: RTX 3090, A6000), sm_89 (Ada: RTX 4090), and sm_80 (A100). See [this issue](https://github.com/mit-han-lab/nunchaku/issues/1) for more details. * We currently support only NVIDIA GPUs with architectures sm_86 (Ampere: RTX 3090, A6000), sm_89 (Ada: RTX 4090), and sm_80 (A100). See [this issue](https://github.com/mit-han-lab/nunchaku/issues/1) for more details.
...@@ -104,6 +105,7 @@ Please refer to [comfyui/README.md](comfyui/README.md) for the usage in [ComfyUI ...@@ -104,6 +105,7 @@ Please refer to [comfyui/README.md](comfyui/README.md) for the usage in [ComfyUI
* Sketch-to-Image ([pix2pix-Turbo](https://github.com/GaParmar/img2img-turbo)): see [`app/flux.1/sketch`](app/flux.1/sketch). * Sketch-to-Image ([pix2pix-Turbo](https://github.com/GaParmar/img2img-turbo)): see [`app/flux.1/sketch`](app/flux.1/sketch).
* Depth/Canny-to-Image ([FLUX.1-tools](https://blackforestlabs.ai/flux-1-tools/)): see [`app/flux.1/depth_canny`](app/flux.1/depth_canny). * Depth/Canny-to-Image ([FLUX.1-tools](https://blackforestlabs.ai/flux-1-tools/)): see [`app/flux.1/depth_canny`](app/flux.1/depth_canny).
* Inpainting ([FLUX.1-Fill-dev](https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev)): see [`app/flux.1/fill`](app/flux.1/fill). * Inpainting ([FLUX.1-Fill-dev](https://huggingface.co/black-forest-labs/FLUX.1-Depth-dev)): see [`app/flux.1/fill`](app/flux.1/fill).
* Redux ([FLUX.1-Redux-dev](https://huggingface.co/black-forest-labs/FLUX.1-Redux-dev)): see [`app/flux.1/redux`](app/flux.1/redux).
* SANA: * SANA:
* Text-to-image: see [`app/sana/t2i`](app/sana/t2i). * Text-to-image: see [`app/sana/t2i`](app/sana/t2i).
...@@ -115,7 +117,7 @@ Please refer to [app/flux/t2i/README.md](app/flux/t2i/README.md) for instruction ...@@ -115,7 +117,7 @@ Please refer to [app/flux/t2i/README.md](app/flux/t2i/README.md) for instruction
- [ ] Easy installation - [ ] Easy installation
- [x] Comfy UI node - [x] Comfy UI node
- [ ] Customized LoRA conversion instructions - [x] Customized LoRA conversion instructions
- [ ] Customized model quantization instructions - [ ] Customized model quantization instructions
- [x] FLUX.1 tools support - [x] FLUX.1 tools support
- [ ] Modularization - [ ] Modularization
...@@ -153,4 +155,4 @@ We thank MIT-IBM Watson AI Lab, MIT and Amazon Science Hub, MIT AI Hardware Prog ...@@ -153,4 +155,4 @@ We thank MIT-IBM Watson AI Lab, MIT and Amazon Science Hub, MIT AI Hardware Prog
We use [img2img-turbo](https://github.com/GaParmar/img2img-turbo) to train the sketch-to-image LoRA. Our text-to-image and sketch-to-image UI is built upon [playground-v.25](https://huggingface.co/spaces/playgroundai/playground-v2.5/blob/main/app.py) and [img2img-turbo](https://github.com/GaParmar/img2img-turbo/blob/main/gradio_sketch2image.py), respectively. Our safety checker is borrowed from [hart](https://github.com/mit-han-lab/hart). We use [img2img-turbo](https://github.com/GaParmar/img2img-turbo) to train the sketch-to-image LoRA. Our text-to-image and sketch-to-image UI is built upon [playground-v.25](https://huggingface.co/spaces/playgroundai/playground-v2.5/blob/main/app.py) and [img2img-turbo](https://github.com/GaParmar/img2img-turbo/blob/main/gradio_sketch2image.py), respectively. Our safety checker is borrowed from [hart](https://github.com/mit-han-lab/hart).
Nunchaku is also inspired by many open-source libraries, including (but not limited to) [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [vLLM](https://github.com/vllm-project/vllm), [QServe](https://github.com/mit-han-lab/qserve), [AWQ](https://github.com/mit-han-lab/llm-awq), [FlashAttention-2](https://github.com/Dao-AILab/flash-attention), and [Atom](https://github.com/efeslab/Atom). Nunchaku is also inspired by many open-source libraries, including (but not limited to) [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM), [vLLM](https://github.com/vllm-project/vllm), [QServe](https://github.com/mit-han-lab/qserve), [AWQ](https://github.com/mit-han-lab/llm-awq), [FlashAttention-2](https://github.com/Dao-AILab/flash-attention), and [Atom](https://github.com/efeslab/Atom).
\ No newline at end of file
app/flux.1/depth_canny/assets/description.html
View file @ 73939beb
...@@ -10,36 +10,36 @@ ...@@ -10,36 +10,36 @@
SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
</h2> </h2>
<h3> <h3>
<a href='https://lmxyy.me'>Muyang Li*</a>, <a href='https://lmxyy.me' target="_blank">Muyang Li*</a>,
<a href='https://yujunlin.com'>Yujun Lin*</a>, <a href='https://yujunlin.com' target="_blank">Yujun Lin*</a>,
<a href='https://hanlab.mit.edu/team/zhekai-zhang'>Zhekai Zhang*</a>, <a href='https://hanlab.mit.edu/team/zhekai-zhang' target="_blank">Zhekai Zhang*</a>,
<a href='https://www.tianle.website/#/'>Tianle Cai</a>, <a href='https://www.tianle.website/#/' target="_blank">Tianle Cai</a>,
<a href='https://xiuyuli.com'>Xiuyu Li</a>, <a href='https://xiuyuli.com' target="_blank">Xiuyu Li</a>,
<br> <br>
<a href='https://github.com/JerryGJX'>Junxian Guo</a>, <a href='https://github.com/JerryGJX' target="_blank">Junxian Guo</a>,
<a href='https://xieenze.github.io'>Enze Xie</a>, <a href='https://xieenze.github.io' target="_blank">Enze Xie</a>,
<a href='https://cs.stanford.edu/~chenlin/'>Chenlin Meng</a>, <a href='https://cs.stanford.edu/~chenlin/' target="_blank">Chenlin Meng</a>,
<a href='https://www.cs.cmu.edu/~junyanz/'>Jun-Yan Zhu</a>, <a href='https://www.cs.cmu.edu/~junyanz/' target="_blank">Jun-Yan Zhu</a>,
and <a href='https://hanlab.mit.edu/songhan'>Song Han</a> and <a href='https://hanlab.mit.edu/songhan' target="_blank">Song Han</a>
</h3> </h3>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> <div style="display: flex; justify-content: center; align-items: center; text-align: center;">
<a href="https://arxiv.org/abs/2411.05007">[Paper]</a> <a href="https://arxiv.org/abs/2411.05007" target="_blank">[Paper]</a>
&nbsp; &nbsp;
<a href='https://github.com/mit-han-lab/nunchaku'> <a href='https://github.com/mit-han-lab/nunchaku' target="_blank">
[Code] [Code]
</a> </a>
&nbsp; &nbsp;
<a href='https://hanlab.mit.edu/projects/svdquant'> <a href='https://hanlab.mit.edu/projects/svdquant' target="_blank">
[Website] [Website]
</a> </a>
&nbsp; &nbsp;
<a href='https://hanlab.mit.edu/blog/svdquant'> <a href='https://hanlab.mit.edu/blog/svdquant' target="_blank">
[Blog] [Blog]
</a> </a>
</div> </div>
<h4>Quantization Library: <h4>Quantization Library:
<a href='https://github.com/mit-han-lab/deepcompressor'>DeepCompressor</a>&nbsp; <a href='https://github.com/mit-han-lab/deepcompressor' target="_blank">DeepCompressor</a>&nbsp;
Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku'>Nunchaku</a>&nbsp; Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku' target="_blank">Nunchaku</a>&nbsp;
</h4> </h4>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> <div style="display: flex; justify-content: center; align-items: center; text-align: center;">
{device_info} {device_info}
......
app/flux.1/depth_canny/assets/style.css
View file @ 73939beb
@import url('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css'); @import url('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css');
.gradio-container{max-width: 1200px !important} .gradio-container {
h1{text-align:center} max-width: 1200px !important;
margin: auto; /* Centers the element horizontally */
}
h1 {
text-align: center
}
.wrap.svelte-p4aq0j.svelte-p4aq0j { .wrap.svelte-p4aq0j.svelte-p4aq0j {
display: none; display: none;
...@@ -22,8 +28,13 @@ h1{text-align:center} ...@@ -22,8 +28,13 @@ h1{text-align:center}
#accessibility { #accessibility {
text-align: center; /* Center-aligns the text */ text-align: center; /* Center-aligns the text */
margin: auto; /* Centers the element horizontally */ margin: auto; /* Centers the element horizontally */
}
#random_seed {
height: 71px;
} }
#random_seed {height: 71px;} #run_button {
#run_button {height: 87px;} height: 87px;
\ No newline at end of file }
\ No newline at end of file
app/flux.1/depth_canny/run_gradio.py
View file @ 73939beb
...@@ -137,8 +137,8 @@ with gr.Blocks(css_paths="assets/style.css", title=f"SVDQuant Flux.1-{model_name ...@@ -137,8 +137,8 @@ with gr.Blocks(css_paths="assets/style.css", title=f"SVDQuant Flux.1-{model_name
def get_header_str(): def get_header_str():
if args.count_use: if args.count_use:
if os.path.exists("use_count.txt"): if os.path.exists(f"{args.model}-use_count.txt"):
with open("use_count.txt", "r") as f: with open(f"{args.model}-use_count.txt", "r") as f:
count = int(f.read()) count = int(f.read())
else: else:
count = 0 count = 0
......
app/flux.1/depth_canny/vars.py
View file @ 73939beb
...@@ -65,7 +65,7 @@ EXAMPLES = { ...@@ -65,7 +65,7 @@ EXAMPLES = {
], ],
[ [
"https://huggingface.co/mit-han-lab/svdq-int4-flux.1-fill-dev/resolve/main/example.png", "https://huggingface.co/mit-han-lab/svdq-int4-flux.1-fill-dev/resolve/main/example.png",
"A wooden basked of several individual cartons of strawberries.", "A wooden basket of several individual cartons of strawberries.",
DEFAULT_STYLE_NAME, DEFAULT_STYLE_NAME,
STYLES[DEFAULT_STYLE_NAME], STYLES[DEFAULT_STYLE_NAME],
30, 30,
......
app/flux.1/fill/assets/description.html
View file @ 73939beb
...@@ -10,36 +10,36 @@ ...@@ -10,36 +10,36 @@
SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
</h2> </h2>
<h3> <h3>
<a href='https://lmxyy.me'>Muyang Li*</a>, <a href='https://lmxyy.me' target="_blank">Muyang Li*</a>,
<a href='https://yujunlin.com'>Yujun Lin*</a>, <a href='https://yujunlin.com' target="_blank">Yujun Lin*</a>,
<a href='https://hanlab.mit.edu/team/zhekai-zhang'>Zhekai Zhang*</a>, <a href='https://hanlab.mit.edu/team/zhekai-zhang' target="_blank">Zhekai Zhang*</a>,
<a href='https://www.tianle.website/#/'>Tianle Cai</a>, <a href='https://www.tianle.website/#/' target="_blank">Tianle Cai</a>,
<a href='https://xiuyuli.com'>Xiuyu Li</a>, <a href='https://xiuyuli.com' target="_blank">Xiuyu Li</a>,
<br> <br>
<a href='https://github.com/JerryGJX'>Junxian Guo</a>, <a href='https://github.com/JerryGJX' target="_blank">Junxian Guo</a>,
<a href='https://xieenze.github.io'>Enze Xie</a>, <a href='https://xieenze.github.io' target="_blank">Enze Xie</a>,
<a href='https://cs.stanford.edu/~chenlin/'>Chenlin Meng</a>, <a href='https://cs.stanford.edu/~chenlin/' target="_blank">Chenlin Meng</a>,
<a href='https://www.cs.cmu.edu/~junyanz/'>Jun-Yan Zhu</a>, <a href='https://www.cs.cmu.edu/~junyanz/' target="_blank">Jun-Yan Zhu</a>,
and <a href='https://hanlab.mit.edu/songhan'>Song Han</a> and <a href='https://hanlab.mit.edu/songhan' target="_blank">Song Han</a>
</h3> </h3>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> <div style="display: flex; justify-content: center; align-items: center; text-align: center;">
<a href="https://arxiv.org/abs/2411.05007">[Paper]</a> <a href="https://arxiv.org/abs/2411.05007" target="_blank">[Paper]</a>
&nbsp; &nbsp;
<a href='https://github.com/mit-han-lab/nunchaku'> <a href='https://github.com/mit-han-lab/nunchaku' target="_blank">
[Code] [Code]
</a> </a>
&nbsp; &nbsp;
<a href='https://hanlab.mit.edu/projects/svdquant'> <a href='https://hanlab.mit.edu/projects/svdquant' target="_blank">
[Website] [Website]
</a> </a>
&nbsp; &nbsp;
<a href='https://hanlab.mit.edu/blog/svdquant'> <a href='https://hanlab.mit.edu/blog/svdquant' target="_blank">
[Blog] [Blog]
</a> </a>
</div> </div>
<h4>Quantization Library: <h4>Quantization Library:
<a href='https://github.com/mit-han-lab/deepcompressor'>DeepCompressor</a>&nbsp; <a href='https://github.com/mit-han-lab/deepcompressor' target="_blank">DeepCompressor</a>&nbsp;
Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku'>Nunchaku</a>&nbsp; Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku' target="_blank">Nunchaku</a>&nbsp;
</h4> </h4>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> <div style="display: flex; justify-content: center; align-items: center; text-align: center;">
{device_info} {device_info}
......
app/flux.1/fill/assets/style.css
View file @ 73939beb
@import url('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css'); @import url('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css');
.gradio-container{max-width: 1200px !important} .gradio-container {
h1{text-align:center} max-width: 1200px !important;
margin: auto; /* Centers the element horizontally */
}
h1 {
text-align: center
}
.wrap.svelte-p4aq0j.svelte-p4aq0j { .wrap.svelte-p4aq0j.svelte-p4aq0j {
display: none; display: none;
...@@ -22,8 +28,13 @@ h1{text-align:center} ...@@ -22,8 +28,13 @@ h1{text-align:center}
#accessibility { #accessibility {
text-align: center; /* Center-aligns the text */ text-align: center; /* Center-aligns the text */
margin: auto; /* Centers the element horizontally */ margin: auto; /* Centers the element horizontally */
}
#random_seed {
height: 71px;
} }
#random_seed {height: 71px;} #run_button {
#run_button {height: 87px;} height: 87px;
\ No newline at end of file }
\ No newline at end of file
app/flux.1/redux/README.md 0 → 100644
View file @ 73939beb
# Nunchaku INT4 FLUX.1 Redux Demo
![demo](./assets/demo.png)
This interactive Gradio application allows you to interactively generate image variations. The base model is [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev). We use [FLUX.1-Redux-dev](https://huggingface.co/black-forest-labs/FLUX.1-Redux-dev) to preprocess the image before inputting it into Flux.1-dev. To launch the application, run:
```shell
python run_gradio.py
```
* By default, we use our INT4 model. Use `-p bf16` to switch to the BF16 model.
\ No newline at end of file
app/flux.1/redux/assets/description.html 0 → 100644
View file @ 73939beb
<div style="display: flex; justify-content: center; align-items: center; text-align: center;">
<div>
<h1>
<img src="https://github.com/mit-han-lab/nunchaku/raw/refs/heads/main/assets/logo.svg"
alt="logo"
style="height: 40px; width: auto; display: block; margin: auto;"/>
INT4 FLUX.1-redux-dev Demo
</h1>
<h2>
SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
</h2>
<h3>
<a href='https://lmxyy.me'>Muyang Li*</a>,
<a href='https://yujunlin.com'>Yujun Lin*</a>,
<a href='https://hanlab.mit.edu/team/zhekai-zhang'>Zhekai Zhang*</a>,
<a href='https://www.tianle.website/#/'>Tianle Cai</a>,
<a href='https://xiuyuli.com'>Xiuyu Li</a>,
<br>
<a href='https://github.com/JerryGJX'>Junxian Guo</a>,
<a href='https://xieenze.github.io'>Enze Xie</a>,
<a href='https://cs.stanford.edu/~chenlin/'>Chenlin Meng</a>,
<a href='https://www.cs.cmu.edu/~junyanz/'>Jun-Yan Zhu</a>,
and <a href='https://hanlab.mit.edu/songhan'>Song Han</a>
</h3>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;">
<a href="https://arxiv.org/abs/2411.05007">[Paper]</a>
&nbsp;
<a href='https://github.com/mit-han-lab/nunchaku'>
[Code]
</a>
&nbsp;
<a href='https://hanlab.mit.edu/projects/svdquant'>
[Website]
</a>
&nbsp;
<a href='https://hanlab.mit.edu/blog/svdquant'>
[Blog]
</a>
</div>
<h4>Quantization Library:
<a href='https://github.com/mit-han-lab/deepcompressor'>DeepCompressor</a>&nbsp;
Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku'>Nunchaku</a>&nbsp;
</h4>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;">
{device_info}
</div>
{count_info}
</div>
</div>
\ No newline at end of file
app/flux.1/redux/assets/style.css 0 → 100644
View file @ 73939beb
@import url('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css');
.gradio-container{max-width: 1200px !important}
h1{text-align:center}
.wrap.svelte-p4aq0j.svelte-p4aq0j {
display: none;
}
#column_input, #column_output {
width: 500px;
display: flex;
align-items: center;
}
#input_header, #output_header {
display: flex;
justify-content: center;
align-items: center;
width: 400px;
}
#accessibility {
text-align: center; /* Center-aligns the text */
margin: auto; /* Centers the element horizontally */
}
#random_seed {height: 71px;}
#run_button {height: 87px;}
\ No newline at end of file
app/flux.1/redux/run_gradio.py 0 → 100644
View file @ 73939beb
# Changed from https://github.com/GaParmar/img2img-turbo/blob/main/gradio_sketch2image.py
import os
import random
import time
from datetime import datetime
import GPUtil
import torch
from diffusers import FluxPipeline, FluxPriorReduxPipeline
from PIL import Image
from nunchaku.models.transformer_flux import NunchakuFluxTransformer2dModel
from utils import get_args
from vars import DEFAULT_GUIDANCE, DEFAULT_INFERENCE_STEP, EXAMPLES, MAX_SEED
# import gradio last to avoid conflicts with other imports
import gradio as gr
args = get_args()
pipe_prior_redux = FluxPriorReduxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-Redux-dev", torch_dtype=torch.bfloat16
).to("cuda")
if args.precision == "bf16":
pipeline = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipeline = pipeline.to("cuda")
pipeline.precision = "bf16"
else:
assert args.precision == "int4"
pipeline_init_kwargs = {}
transformer = NunchakuFluxTransformer2dModel.from_pretrained("mit-han-lab/svdq-int4-flux.1-dev")
pipeline = FluxPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
text_encoder=None,
text_encoder_2=None,
transformer=transformer,
torch_dtype=torch.bfloat16,
)
pipeline = pipeline.to("cuda")
pipeline.precision = "int4"
def run(image, num_inference_steps: int, guidance_scale: float, seed: int) -> tuple[Image, str]:
pipe_prior_output = pipe_prior_redux(image["composite"])
start_time = time.time()
result_image = pipeline(
num_inference_steps=num_inference_steps,
guidance_scale=guidance_scale,
generator=torch.Generator().manual_seed(seed),
**pipe_prior_output,
).images[0]
latency = time.time() - start_time
if latency < 1:
latency = latency * 1000
latency_str = f"{latency:.2f}ms"
else:
latency_str = f"{latency:.2f}s"
torch.cuda.empty_cache()
if args.count_use:
if os.path.exists("use_count.txt"):
with open("use_count.txt", "r") as f:
count = int(f.read())
else:
count = 0
count += 1
current_time = datetime.now()
print(f"{current_time}: {count}")
with open("use_count.txt", "w") as f:
f.write(str(count))
with open("use_record.txt", "a") as f:
f.write(f"{current_time}: {count}\n")
return result_image, latency_str
with gr.Blocks(css_paths="assets/style.css", title=f"SVDQuant Flux.1-redux-dev Demo") as demo:
with open("assets/description.html", "r") as f:
DESCRIPTION = f.read()
gpus = GPUtil.getGPUs()
if len(gpus) > 0:
gpu = gpus[0]
memory = gpu.memoryTotal / 1024
device_info = f"Running on {gpu.name} with {memory:.0f} GiB memory."
else:
device_info = "Running on CPU 🥶 This demo does not work on CPU."
def get_header_str():
if args.count_use:
if os.path.exists("use_count.txt"):
with open("use_count.txt", "r") as f:
count = int(f.read())
else:
count = 0
count_info = (
f"<div style='display: flex; justify-content: center; align-items: center; text-align: center;'>"
f"<span style='font-size: 18px; font-weight: bold;'>Total inference runs: </span>"
f"<span style='font-size: 18px; color:red; font-weight: bold;'>&nbsp;{count}</span></div>"
)
else:
count_info = ""
header_str = DESCRIPTION.format(device_info=device_info, count_info=count_info)
return header_str
header = gr.HTML(get_header_str())
demo.load(fn=get_header_str, outputs=header)
with gr.Row(elem_id="main_row"):
with gr.Column(elem_id="column_input"):
gr.Markdown("## INPUT", elem_id="input_header")
with gr.Group():
canvas = gr.ImageEditor(
height=640,
image_mode="RGB",
sources=["upload", "clipboard"],
type="pil",
label="Input",
show_label=False,
show_download_button=True,
interactive=True,
transforms=[],
canvas_size=(1024, 1024),
scale=1,
format="png",
layers=False,
)
with gr.Row():
run_button = gr.Button("Run", elem_id="run_button")
with gr.Row():
seed = gr.Slider(
label="Seed",
show_label=True,
minimum=0,
maximum=MAX_SEED,
value=233,
step=1,
scale=4,
)
randomize_seed = gr.Button("Random Seed", scale=1, min_width=50, elem_id="random_seed")
with gr.Accordion("Advanced options", open=False):
with gr.Group():
num_inference_steps = gr.Slider(
label="Inference Steps",
minimum=10,
maximum=50,
step=1,
value=DEFAULT_INFERENCE_STEP,
)
guidance_scale = gr.Slider(
label="Guidance Scale",
minimum=1,
maximum=10,
step=0.5,
value=DEFAULT_GUIDANCE,
)
with gr.Column(elem_id="column_output"):
gr.Markdown("## OUTPUT", elem_id="output_header")
with gr.Group():
result = gr.Image(
format="png",
height=640,
image_mode="RGB",
type="pil",
label="Result",
show_label=False,
show_download_button=True,
interactive=False,
elem_id="output_image",
)
latency_result = gr.Text(label="Inference Latency", show_label=True)
gr.Markdown("### Instructions")
gr.Markdown("**1**. Upload or paste an image")
gr.Markdown(
"**2**. Adjust the effect of sketch guidance and inference steps using sliders under Advanced options"
)
gr.Markdown("**3**. Try different seeds to generate different results")
run_inputs = [canvas, num_inference_steps, guidance_scale, seed]
run_outputs = [result, latency_result]
gr.Examples(examples=EXAMPLES, inputs=run_inputs, outputs=run_outputs, fn=run)
randomize_seed.click(
lambda: random.randint(0, MAX_SEED),
inputs=[],
outputs=seed,
api_name=False,
queue=False,
).then(run, inputs=run_inputs, outputs=run_outputs, api_name=False)
gr.on(triggers=[run_button.click], fn=run, inputs=run_inputs, outputs=run_outputs, api_name=False)
gr.Markdown("MIT Accessibility: https://accessibility.mit.edu/", elem_id="accessibility")
if __name__ == "__main__":
demo.queue().launch(debug=True, share=True, root_path=args.gradio_root_path)
app/flux.1/redux/utils.py 0 → 100644
View file @ 73939beb
import argparse
def get_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument(
"-p",
"--precision",
type=str,
default="int4",
choices=["int4", "bf16"],
help="Which precisions to use",
)
parser.add_argument(
"--count-use", action="store_true", help="Whether to count the number of uses"
)
parser.add_argument("--gradio-root-path", type=str, default="")
args = parser.parse_args()
return args
app/flux.1/redux/vars.py 0 → 100644
View file @ 73939beb
MAX_SEED = 1000000000
DEFAULT_GUIDANCE = 2.5
DEFAULT_INFERENCE_STEP = 50
EXAMPLES = [
[
"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png",
DEFAULT_INFERENCE_STEP,
DEFAULT_GUIDANCE,
1,
],
[
"https://huggingface.co/mit-han-lab/svdq-int4-flux.1-fill-dev/resolve/main/example.png",
DEFAULT_INFERENCE_STEP,
DEFAULT_GUIDANCE,
1,
],
]
app/flux.1/sketch/assets/description.html
View file @ 73939beb
...@@ -10,37 +10,37 @@ ...@@ -10,37 +10,37 @@
SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
</h2> </h2>
<h3> <h3>
<a href='https://lmxyy.me'>Muyang Li*</a>, <a href='https://lmxyy.me' target="_blank">Muyang Li*</a>,
<a href='https://yujunlin.com'>Yujun Lin*</a>, <a href='https://yujunlin.com' target="_blank">Yujun Lin*</a>,
<a href='https://hanlab.mit.edu/team/zhekai-zhang'>Zhekai Zhang*</a>, <a href='https://hanlab.mit.edu/team/zhekai-zhang' target="_blank">Zhekai Zhang*</a>,
<a href='https://www.tianle.website/#/'>Tianle Cai</a>, <a href='https://www.tianle.website/#/' target="_blank">Tianle Cai</a>,
<a href='https://xiuyuli.com'>Xiuyu Li</a>, <a href='https://xiuyuli.com' target="_blank">Xiuyu Li</a>,
<br> <br>
<a href='https://github.com/JerryGJX'>Junxian Guo</a>, <a href='https://github.com/JerryGJX' target="_blank">Junxian Guo</a>,
<a href='https://xieenze.github.io'>Enze Xie</a>, <a href='https://xieenze.github.io' target="_blank">Enze Xie</a>,
<a href='https://cs.stanford.edu/~chenlin/'>Chenlin Meng</a>, <a href='https://cs.stanford.edu/~chenlin/' target="_blank">Chenlin Meng</a>,
<a href='https://www.cs.cmu.edu/~junyanz/'>Jun-Yan Zhu</a>, <a href='https://www.cs.cmu.edu/~junyanz/' target="_blank">Jun-Yan Zhu</a>,
and <a href='https://hanlab.mit.edu/songhan'>Song Han</a> and <a href='https://hanlab.mit.edu/songhan' target="_blank">Song Han</a>
</h3> </h3>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> <div style="display: flex; justify-content: center; align-items: center; text-align: center;">
<a href="https://arxiv.org/abs/2411.05007">[Paper]</a> <a href="https://arxiv.org/abs/2411.05007" target="_blank">[Paper]</a>
&nbsp; &nbsp;
<a href='https://github.com/mit-han-lab/nunchaku'> <a href='https://github.com/mit-han-lab/nunchaku' target="_blank">
[Code] [Code]
</a> </a>
&nbsp; &nbsp;
<a href='https://hanlab.mit.edu/projects/svdquant'> <a href='https://hanlab.mit.edu/projects/svdquant' target="_blank">
[Website] [Website]
</a> </a>
&nbsp; &nbsp;
<a href='https://hanlab.mit.edu/blog/svdquant'> <a href='https://hanlab.mit.edu/blog/svdquant' target="_blank">
[Blog] [Blog]
</a> </a>
</div> </div>
<h4>Quantization Library: <h4>Quantization Library:
<a href='https://github.com/mit-han-lab/deepcompressor'>DeepCompressor</a>&nbsp; <a href='https://github.com/mit-han-lab/deepcompressor' target="_blank">DeepCompressor</a>&nbsp;
Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku'>Nunchaku</a>&nbsp; Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku' target="_blank">Nunchaku</a>&nbsp;
Image Control: <a href="https://github.com/GaParmar/img2img-turbo">img2img-turbo</a> Image Control: <a href="https://github.com/GaParmar/img2img-turbo" target="_blank">img2img-turbo</a>
</h4> </h4>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> <div style="display: flex; justify-content: center; align-items: center; text-align: center;">
{device_info} {device_info}
......
app/flux.1/sketch/assets/style.css
View file @ 73939beb
@import url('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css'); @import url('https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css');
.gradio-container{max-width: 1200px !important} .gradio-container {
h1{text-align:center} max-width: 1200px !important;
margin: auto; /* Centers the element horizontally */
}
h1 {
text-align: center
}
.wrap.svelte-p4aq0j.svelte-p4aq0j { .wrap.svelte-p4aq0j.svelte-p4aq0j {
display: none; display: none;
...@@ -22,8 +28,13 @@ h1{text-align:center} ...@@ -22,8 +28,13 @@ h1{text-align:center}
#accessibility { #accessibility {
text-align: center; /* Center-aligns the text */ text-align: center; /* Center-aligns the text */
margin: auto; /* Centers the element horizontally */ margin: auto; /* Centers the element horizontally */
}
#random_seed {
height: 71px;
} }
#random_seed {height: 71px;} #run_button {
#run_button {height: 87px;} height: 87px;
\ No newline at end of file }
\ No newline at end of file
app/flux.1/sketch/flux_pix2pix_pipeline.py
View file @ 73939beb
...@@ -139,10 +139,8 @@ class FluxPix2pixTurboPipeline(FluxPipeline): ...@@ -139,10 +139,8 @@ class FluxPix2pixTurboPipeline(FluxPipeline):
erosion_kernel = torch.ones(1, 1, kernel_size, kernel_size, dtype=self.dtype, device=device) erosion_kernel = torch.ones(1, 1, kernel_size, kernel_size, dtype=self.dtype, device=device)
self.erosion_kernel = erosion_kernel self.erosion_kernel = erosion_kernel
torchvision.utils.save_image(image_t[0], "before.png")
image_t = nn.functional.conv2d(image_t[:, :1], erosion_kernel, padding=kernel_size // 2) > kernel_size**2 - 0.1 image_t = nn.functional.conv2d(image_t[:, :1], erosion_kernel, padding=kernel_size // 2) > kernel_size**2 - 0.1
image_t = torch.concat([image_t, image_t, image_t], dim=1).to(self.dtype) image_t = torch.concat([image_t, image_t, image_t], dim=1).to(self.dtype)
torchvision.utils.save_image(image_t[0], "after.png")
image_t = (image_t - 0.5) * 2 image_t = (image_t - 0.5) * 2
......
app/flux.1/sketch/run_gradio.py
View file @ 73939beb
# Changed from https://github.com/GaParmar/img2img-turbo/blob/main/gradio_sketch2image.py # Changed from https://github.com/GaParmar/img2img-turbo/blob/main/gradio_sketch2image.py
import logging
import os import os
import random import random
import tempfile import tempfile
...@@ -29,7 +28,7 @@ if args.precision == "bf16": ...@@ -29,7 +28,7 @@ if args.precision == "bf16":
pipeline = pipeline.to("cuda") pipeline = pipeline.to("cuda")
pipeline.precision = "bf16" pipeline.precision = "bf16"
pipeline.load_control_module( pipeline.load_control_module(
"mit-han-lab/svdquant-models", "flux.1-pix2pix-turbo-sketch2image.safetensors", alpha=DEFAULT_SKETCH_GUIDANCE "mit-han-lab/svdq-flux.1-schnell-pix2pix-turbo", "sketch.safetensors", alpha=DEFAULT_SKETCH_GUIDANCE
) )
else: else:
assert args.precision == "int4" assert args.precision == "int4"
...@@ -48,9 +47,9 @@ else: ...@@ -48,9 +47,9 @@ else:
pipeline = pipeline.to("cuda") pipeline = pipeline.to("cuda")
pipeline.precision = "int4" pipeline.precision = "int4"
pipeline.load_control_module( pipeline.load_control_module(
"mit-han-lab/svdquant-models", "mit-han-lab/svdq-flux.1-schnell-pix2pix-turbo",
"flux.1-pix2pix-turbo-sketch2image.safetensors", "sketch.safetensors",
svdq_lora_path="mit-han-lab/svdquant-models/svdq-flux.1-pix2pix-turbo-sketch2image.safetensors", svdq_lora_path="mit-han-lab/svdq-flux.1-schnell-pix2pix-turbo/svdq-int4-sketch.safetensors",
alpha=DEFAULT_SKETCH_GUIDANCE, alpha=DEFAULT_SKETCH_GUIDANCE,
) )
safety_checker = SafetyChecker("cuda", disabled=args.no_safety_checker) safety_checker = SafetyChecker("cuda", disabled=args.no_safety_checker)
...@@ -223,13 +222,7 @@ with gr.Blocks(css_paths="assets/style.css", title=f"SVDQuant Sketch-to-Image De ...@@ -223,13 +222,7 @@ with gr.Blocks(css_paths="assets/style.css", title=f"SVDQuant Sketch-to-Image De
queue=False, queue=False,
).then(run, inputs=run_inputs, outputs=run_outputs, api_name=False) ).then(run, inputs=run_inputs, outputs=run_outputs, api_name=False)
style.change( style.change(lambda x: STYLES[x], inputs=[style], outputs=[prompt_template], api_name=False, queue=False)
lambda x: STYLES[x],
inputs=[style],
outputs=[prompt_template],
api_name=False,
queue=False,
)
gr.on( gr.on(
triggers=[prompt.submit, run_button.click, canvas.change], triggers=[prompt.submit, run_button.click, canvas.change],
fn=run, fn=run,
......
app/flux.1/t2i/assets/description.html
View file @ 73939beb
...@@ -10,37 +10,37 @@ ...@@ -10,37 +10,37 @@
SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models
</h2> </h2>
<h3> <h3>
<a href='https://lmxyy.me'>Muyang Li*</a>, <a href='https://lmxyy.me' target="_blank">Muyang Li*</a>,
<a href='https://yujunlin.com'>Yujun Lin*</a>, <a href='https://yujunlin.com' target="_blank">Yujun Lin*</a>,
<a href='https://hanlab.mit.edu/team/zhekai-zhang'>Zhekai Zhang*</a>, <a href='https://hanlab.mit.edu/team/zhekai-zhang' target="_blank">Zhekai Zhang*</a>,
<a href='https://www.tianle.website/#/'>Tianle Cai</a>, <a href='https://www.tianle.website/#/' target="_blank">Tianle Cai</a>,
<a href='https://xiuyuli.com'>Xiuyu Li</a>, <a href='https://xiuyuli.com' target="_blank">Xiuyu Li</a>,
<br> <br>
<a href='https://github.com/JerryGJX'>Junxian Guo</a>, <a href='https://github.com/JerryGJX' target="_blank">Junxian Guo</a>,
<a href='https://xieenze.github.io'>Enze Xie</a>, <a href='https://xieenze.github.io' target="_blank">Enze Xie</a>,
<a href='https://cs.stanford.edu/~chenlin/'>Chenlin Meng</a>, <a href='https://cs.stanford.edu/~chenlin/' target="_blank">Chenlin Meng</a>,
<a href='https://www.cs.cmu.edu/~junyanz/'>Jun-Yan Zhu</a>, <a href='https://www.cs.cmu.edu/~junyanz/' target="_blank">Jun-Yan Zhu</a>,
and <a href='https://hanlab.mit.edu/songhan'>Song Han</a> and <a href='https://hanlab.mit.edu/songhan' target="_blank">Song Han</a>
</h3> </h3>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> <div style="display: flex; justify-content: center; align-items: center; text-align: center;">
<a href="https://arxiv.org/abs/2411.05007">[Paper]</a> <a href="https://arxiv.org/abs/2411.05007" target="_blank">[Paper]</a>
&nbsp; &nbsp;
<a href='https://github.com/mit-han-lab/nunchaku'> <a href='https://github.com/mit-han-lab/nunchaku' target="_blank">
[Code] [Code]
</a> </a>
&nbsp; &nbsp;
<a href='https://hanlab.mit.edu/projects/svdquant'> <a href='https://hanlab.mit.edu/projects/svdquant' target="_blank">
[Website] [Website]
</a> </a>
&nbsp; &nbsp;
<a href='https://hanlab.mit.edu/blog/svdquant'> <a href='https://hanlab.mit.edu/blog/svdquant' target="_blank">
[Blog] [Blog]
</a> </a>
</div> </div>
<h4>Quantization Library: <h4>Quantization Library:
<a href='https://github.com/mit-han-lab/deepcompressor'>DeepCompressor</a> <a href='https://github.com/mit-han-lab/deepcompressor' target="_blank">DeepCompressor</a>
&nbsp; &nbsp;
Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku'>Nunchaku</a> Inference Engine: <a href='https://github.com/mit-han-lab/nunchaku' target="_blank">Nunchaku</a>
</h4> </h4>
<div style="display: flex; justify-content: center; align-items: center; text-align: center;"> <div style="display: flex; justify-content: center; align-items: center; text-align: center;">
{device_info} {device_info}
......
app/flux.1/t2i/assets/frame1.css
View file @ 73939beb
.gradio-container{max-width: 560px !important} .gradio-container {
max-width: 640px !important;
margin: auto; /* Centers the element horizontally */
}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment