Merge https://github.com/mit-han-lab/nunchaku into migrate_tinychat

README.md

@@ -4,7 +4,8 @@ Nunchaku is an inference engine designed for 4-bit diffusion models, as demonstr
 
 ### [Paper](http://arxiv.org/abs/2411.05007) | [Project](https://hanlab.mit.edu/projects/svdquant) | [Blog](https://hanlab.mit.edu/blog/svdquant) | [Demo](https://svdquant.mit.edu)
 
-- **[2025-02-14]** 🔥 [LoRA conversion script](nunchaku/convert_lora.py) is now available!
+- **[2025-02-18]** 🔥 [**Customized LoRA conversion**](#Customized-LoRA) and [**model quantization**](#Customized-Model-Quantization) instructions are now available! **[ComfyUI](./comfyui)** workflows now support **customized LoRA**, along with **FLUX.1-Tools**!
+- **[2025-02-14]** 🔥 **[LoRA conversion script](nunchaku/convert_lora.py)** is now available! [ComfyUI FLUX.1-tools workflows](./comfyui) is released!
 - **[2025-02-11]** 🎉 **[SVDQuant](http://arxiv.org/abs/2411.05007) has been selected as a ICLR 2025 Spotlight! FLUX.1-tools Gradio demos are now available!** Check [here](#gradio-demos) for the usage details! Our new [depth-to-image demo](https://svdquant.mit.edu/flux1-depth-dev/) is also online—try it out!
 - **[2025-02-04]** **🚀 4-bit [FLUX.1-tools](https://blackforestlabs.ai/flux-1-tools/) is here!** Enjoy a **2-3× speedup** over the original models. Check out the [examples](./examples) for usage. **ComfyUI integration is coming soon!**
 - **[2025-01-23]** 🚀 **4-bit [SANA](https://nvlabs.github.io/Sana/) support is here!** Experience a 2-3× speedup compared to the 16-bit model. Check out the [usage example](./examples/sana_1600m_pag.py) and the [deployment guide](app/sana/t2i) for more details. Explore our live demo at [svdquant.mit.edu](https://svdquant.mit.edu)!
@@ -54,8 +55,8 @@ SVDQuant is a post-training quantization technique for 4-bit weights and activat
 	conda create -n nunchaku python=3.11
 	conda activate nunchaku
 	pip install torch torchvision torchaudio
-	pip install diffusers ninja wheel transformers accelerate sentencepiece protobuf
-	pip install huggingface_hub peft opencv-python einops gradio spaces GPUtil
+	pip install ninja wheel diffusers transformers accelerate sentencepiece protobuf huggingface_hub
+	pip install peft opencv-python gradio spaces GPUtil  # For gradio demos
 	```
 	
 2. Install `nunchaku` package:
@@ -74,6 +75,8 @@ SVDQuant is a post-training quantization technique for 4-bit weights and activat
 	pip install -e . --no-build-isolation
 	```
 
+[Optional] You can verify your installation by running `python -m nunchaku.test`. This will execute our 4-bit FLUX.1-schnell model, which may take some time to download.
+
 ## Usage Example
 
 In [examples](examples), we provide minimal scripts for running INT4 [FLUX.1](https://github.com/black-forest-labs/flux) and [SANA](https://github.com/NVlabs/Sana) models with Nunchaku. For example, the [script](examples/flux.1-dev.py) for [FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev) is as follows:
@@ -94,6 +97,73 @@ image.save("flux.1-dev.png")
 
 Specifically, `nunchaku` shares the same APIs as [diffusers](https://github.com/huggingface/diffusers) and can be used in a similar way.
 
+## Customized LoRA
+
+![lora](./assets/lora.jpg)
+
+[SVDQuant](http://arxiv.org/abs/2411.05007) seamlessly integrates with off-the-shelf LoRAs without requiring requantization. To convert your LoRA safetensors to our format, use the following command:
+
+```shell
+python -m nunchaku.lora.flux.convert \
+  --quant-path mit-han-lab/svdq-int4-flux.1-dev/transformer_blocks.safetensors \
+  --lora-path aleksa-codes/flux-ghibsky-illustration/lora.safetensors \
+  --lora-format diffusers \
+  --output-root ./nunchaku_loras \
+  --lora-name svdq-int4-flux.1-dev-ghibsky
+```
+
+Argument Details:
+
+- `--quant-path`: The path to the quantized base model. It can be a local path or a remote Hugging Face model. For example, you can use [`mit-han-lab/svdq-int4-flux.1-dev/transformer_blocks.safetensors`](https://huggingface.co/mit-han-lab/svdq-int4-flux.1-dev/blob/main/transformer_blocks.safetensors) for FLUX.1-dev.
+
+- `--lora-path`: The path to your LoRA safetensors, which can also be a local or remote Hugging Face model.
+
+- `--lora-format`: Specifies the LoRA format. Supported formats include:
+  - `diffusers` (e.g., [aleksa-codes/flux-ghibsky-illustration](https://huggingface.co/aleksa-codes/flux-ghibsky-illustration))
+  - `comfyui` (e.g., [Shakker-Labs/FLUX.1-dev-LoRA-Children-Simple-Sketch](https://huggingface.co/Shakker-Labs/FLUX.1-dev-LoRA-Children-Simple-Sketch))
+  - `xlab` (e.g., [XLabs-AI/flux-RealismLora](https://huggingface.co/XLabs-AI/flux-RealismLora))
+  
+- `--output-root`: Specifies the output directory for the converted LoRA.
+
+- `--lora-name`: Sets the name of the converted LoRA file (without `.safetensors` extension).
+
+After converting your LoRA, you can use your converted weight with:
+
+```python
+transformer.update_lora_params(path_to_your_converted_lora)
+transformer.set_lora_strength(lora_strength)
+```
+
+`path_to_your_lora` can also be a remote HuggingFace path. In [examples/flux.1-dev-lora.py](examples/flux.1-dev-lora.py), we provide a minimal example script for running [Ghibsky](https://huggingface.co/aleksa-codes/flux-ghibsky-illustration) LoRA with SVDQuant's INT4 FLUX.1-dev:
+
+```python
+import torch
+from diffusers import FluxPipeline
+
+from nunchaku.models.transformer_flux import NunchakuFluxTransformer2dModel
+
+transformer = NunchakuFluxTransformer2dModel.from_pretrained("mit-han-lab/svdq-int4-flux.1-dev")
+pipeline = FluxPipeline.from_pretrained(
+    "black-forest-labs/FLUX.1-dev", transformer=transformer, torch_dtype=torch.bfloat16
+).to("cuda")
+
+### LoRA Related Code ###
+transformer.update_lora_params(
+    "mit-han-lab/svdquant-lora-collection/svdq-int4-flux.1-dev-ghibsky.safetensors"
+)  # Path to your converted LoRA safetensors, can also be a remote HuggingFace path
+transformer.set_lora_strength(1)  # Your LoRA strength here
+### End of LoRA Related Code ###
+
+image = pipeline(
+    "GHIBSKY style, cozy mountain cabin covered in snow, with smoke curling from the chimney and a warm, inviting light spilling through the windows",
+    num_inference_steps=25,
+    guidance_scale=3.5,
+).images[0]
+image.save("flux.1-dev-ghibsky.png")
+```
+
+**For ComfyUI users, we have implemented a node to convert the LoRA weights on the fly. All you need to do is specify the correct LoRA format. Please refer to Please refer to [comfyui/README.md](comfyui/README.md) for more details.**
+
 ## ComfyUI
 
 Please refer to [comfyui/README.md](comfyui/README.md) for the usage in [ComfyUI](https://github.com/comfyanonymous/ComfyUI).
@@ -109,6 +179,10 @@ Please refer to [comfyui/README.md](comfyui/README.md) for the usage in [ComfyUI
 * SANA:
   * Text-to-image: see [`app/sana/t2i`](app/sana/t2i).
 
+## Customized Model Quantization
+
+Please refer to [mit-han-lab/deepcompressor](https://github.com/mit-han-lab/deepcompressor/tree/main/examples/diffusion).
+
 ## Benchmark
 
 Please refer to [app/flux/t2i/README.md](app/flux/t2i/README.md) for instructions on reproducing our paper's quality results and benchmarking inference latency on FLUX.1 models.
@@ -118,7 +192,7 @@ Please refer to [app/flux/t2i/README.md](app/flux/t2i/README.md) for instruction
 - [ ] Easy installation
 - [x] Comfy UI node
 - [x] Customized LoRA conversion instructions
-- [ ] Customized model quantization instructions
+- [x] Customized model quantization instructions
 - [x] FLUX.1 tools support
 - [ ] Modularization
 - [ ] IP-Adapter integration

comfyui/LICENCE.txt

+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [2024] [MIT HAN Lab]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
\ No newline at end of file

comfyui/README.md

@@ -3,30 +3,41 @@
 ![comfyui](../assets/comfyui.jpg)
 ## Installation
 
-1. Install `nunchaku` following [README.md](https://github.com/mit-han-lab/nunchaku?tab=readme-ov-file#installation). 
-2. Install dependencies needed to run custom ComfyUI nodes:
-  ```shell
-  pip install git+https://github.com/asomoza/image_gen_aux.git
-  ```
-3. Set up the dependencies for [ComfyUI](https://github.com/comfyanonymous/ComfyUI/tree/master) with the following commands:
-```shell
-git clone https://github.com/comfyanonymous/ComfyUI.git
-cd ComfyUI
-pip install -r requirements.txt
-```
-4. Install [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager) with the following commands then restart ComfyUI:
+Please first install `nunchaku` following the instructions in [README.md](https://github.com/mit-han-lab/nunchaku?tab=readme-ov-file#installation).
+
+### ComfyUI-CLI
+
 ```shell
-cd ComfyUI/custom_nodes
-git clone https://github.com/ltdrdata/ComfyUI-Manager comfyui-manager
+comfy node registry-install svdquant
 ```
 
-## Usage
+### ComfyUI-Manager (Experimental)
+
+1. Install [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager) with the following commands then restart ComfyUI:
+
+   ```shell
+   cd ComfyUI/custom_nodes
+   git clone https://github.com/ltdrdata/ComfyUI-Manager comfyui-manager
+   ```
+
+2. Open the Manager, search `svdquant` in the Custom Nodes Manager and then install it.
 
-1. **Set Up ComfyUI and SVDQuant**:
 
-  * Navigate to the root directory of ComfyUI and link (or copy) the [`nunchaku/comfyui`](./) folder to `custom_nodes/svdquant`.
-  * Place the SVDQuant workflow configurations from [`workflows`](./workflows) into `user/default/workflows`.
-  * For example
+### Manual Installation
+1. Install dependencies needed to run custom ComfyUI nodes:
+
+   ```shell
+   pip install git+https://github.com/asomoza/image_gen_aux.git
+   ```
+2. Set up the dependencies for [ComfyUI](https://github.com/comfyanonymous/ComfyUI/tree/master) with the following commands:
+
+   ```shell
+   git clone https://github.com/comfyanonymous/ComfyUI.git
+   cd ComfyUI
+   pip install -r requirements.txt
+   ```
+
+3. Navigate to the root directory of ComfyUI and link (or copy) the [`nunchaku/comfyui`](./) folder to `custom_nodes/svdquant`. For example:
 
    ```shell
    # Clone repositories (skip if already cloned)
@@ -34,22 +45,34 @@ git clone https://github.com/ltdrdata/ComfyUI-Manager comfyui-manager
    git clone https://github.com/mit-han-lab/nunchaku.git
    cd ComfyUI
    
-    # Copy workflow configurations
-    mkdir -p user/default/workflows
-    cp ../nunchaku/comfyui/workflows/* user/default/workflows/
-    
    # Add SVDQuant nodes
    cd custom_nodes
    ln -s ../../nunchaku/comfyui svdquant
    ```
+
+## Usage
+
+1. **Set Up ComfyUI and SVDQuant**:
+
+     * SVDQuant workflows can be found at [`workflows`](./workflows). You can place them in `user/default/workflows` in ComfyUI root directory to load them. For example:
+
+       ```shell
+       cd ComfyUI
+       
+       # Copy workflow configurations
+       mkdir -p user/default/workflows
+       cp ../nunchaku/comfyui/workflows/* user/default/workflows/
+       ```
+
      * Install missing nodes (e.g., comfyui-inpainteasy) following [this tutorial](https://github.com/ltdrdata/ComfyUI-Manager?tab=readme-ov-file#support-of-missing-nodes-installation).
 
 
+
 2. **Download Required Models**: Follow [this tutorial](https://comfyanonymous.github.io/ComfyUI_examples/flux/) and download the required models into the appropriate directories using the commands below:
 
    ```shell
-   huggingface-cli download comfyanonymous/flux_text_encoders clip_l.safetensors --local-dir models/clip
-   huggingface-cli download comfyanonymous/flux_text_encoders t5xxl_fp16.safetensors --local-dir models/clip
+   huggingface-cli download comfyanonymous/flux_text_encoders clip_l.safetensors --local-dir models/text_encoders
+   huggingface-cli download comfyanonymous/flux_text_encoders t5xxl_fp16.safetensors --local-dir models/text_encoders
    huggingface-cli download black-forest-labs/FLUX.1-schnell ae.safetensors --local-dir models/vae
    ```
 
@@ -59,7 +82,7 @@ git clone https://github.com/ltdrdata/ComfyUI-Manager comfyui-manager
    python main.py
    ```
 
-4. **Select the SVDQuant Workflow**: Choose one of the SVDQuant workflows (`flux.1-dev-svdquant.json`, `flux.1-schnell-svdquant.json`, `flux.1-depth-svdquant.json`, `flux.1-canny-svdquant.json` or `flux.1-fill-svdquant.json`) to get started. For the flux.1 fill workflow, you can use the built-in MaskEditor tool to add mask on top of an image.
+4. **Select the SVDQuant Workflow**: Choose one of the SVDQuant workflows (workflows that start with `svdq-`) to get started. For the FLUX.1-Fill workflow, you can use the built-in MaskEditor tool to add mask on top of an image.
 
 ## SVDQuant Nodes
 
@@ -75,10 +98,18 @@ git clone https://github.com/ltdrdata/ComfyUI-Manager comfyui-manager
 
   * `device_id`: Indicates the GPU ID for running the model.
 
-* **SVDQuant LoRA Loader**: A node for loading LoRA modules for SVDQuant diffusion models.
+* **SVDQuant FLUX LoRA Loader**: A node for loading LoRA modules for SVDQuant FLUX models.
+
+  * Place your LoRA checkpoints in the `models/loras` directory. These will appear as selectable options under `lora_name`. Meanwhile, the [example Ghibsky LoRA](https://huggingface.co/aleksa-codes/flux-ghibsky-illustration) is included and will automatically download from our Hugging Face repository when used.
+  * `lora_format` specifies the LoRA format. Supported formats include:
+
+    - `diffusers` (e.g., [aleksa-codes/flux-ghibsky-illustration](https://huggingface.co/aleksa-codes/flux-ghibsky-illustration))
+    - `comfyui` (e.g., [Shakker-Labs/FLUX.1-dev-LoRA-Children-Simple-Sketch](https://huggingface.co/Shakker-Labs/FLUX.1-dev-LoRA-Children-Simple-Sketch))
+    - `xlab` (e.g., [XLabs-AI/flux-RealismLora](https://huggingface.co/XLabs-AI/flux-RealismLora))
+    - `svdquant` (e.g., [mit-han-lab/svdquant-lora-collection](https://huggingface.co/mit-han-lab/svdquant-lora-collection)).
 
-  * Place your LoRA checkpoints in the `models/loras` directory. These will appear as selectable options under `lora_name`. **Ensure your LoRA checkpoints conform to the SVDQuant format. **A LoRA conversion script will be released soon. Meanwhile, example LoRAs are included and will automatically download from our Hugging Face repository when used.
-  * **Note**: Currently, only **one LoRA** can be loaded at a time.
+  * `base_model_name` specifies the path to the quantized base model. If `lora_format` is already set to `svdquant`, this option has no use. You can set it to the same value as `model_path` in the above **SVDQuant Flux DiT Loader**.
+  * **Note**: Currently, **only one LoRA** can be loaded at a time.
 
 * **SVDQuant Text Encoder Loader**: A node for loading the text encoders.
 
@@ -106,3 +137,14 @@ git clone https://github.com/ltdrdata/ComfyUI-Manager comfyui-manager
       ```
   
        After downloading, specify the corresponding folder name as the `int4_model`.
+  
+
+
+* **FLUX.1 Depth Preprocessor**: A node for loading the depth estimation model and output the depth map. `model_path` specifies the model location. If set to [`LiheYoung/depth-anything-large-hf`](https://huggingface.co/LiheYoung/depth-anything-large-hf), the model will be automatically downloaded from the Hugging Face repository. Alternatively, you can manually download the repository at `models/checkpoints` by running the following command example:
+
+  ```shell
+  huggingface-cli download LiheYoung/depth-anything-large-hf --local-dir models/checkpoints/depth-anything-large-hf
+  ```
+
+  
+

comfyui/__init__.py

 # only import if running as a custom node
-try:
-    import comfy.utils
-except ImportError:
-    pass
-else:
-    from .nodes import NODE_CLASS_MAPPINGS
 
-    NODE_DISPLAY_NAME_MAPPINGS = {k: v.TITLE for k, v in NODE_CLASS_MAPPINGS.items()}
-    __all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
+from .nodes.lora import SVDQuantFluxLoraLoader
+from .nodes.models import SVDQuantFluxDiTLoader, SVDQuantTextEncoderLoader
+from .nodes.preprocessors import FluxDepthPreprocessor
+
+NODE_CLASS_MAPPINGS = {
+    "SVDQuantFluxDiTLoader": SVDQuantFluxDiTLoader,
+    "SVDQuantTextEncoderLoader": SVDQuantTextEncoderLoader,
+    "SVDQuantFluxLoraLoader": SVDQuantFluxLoraLoader,
+    "SVDQuantDepthPreprocessor": FluxDepthPreprocessor,
+}
+NODE_DISPLAY_NAME_MAPPINGS = {k: v.TITLE for k, v in NODE_CLASS_MAPPINGS.items()}
+__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]

comfyui/nodes/lora/__init__.py

+from .flux import SVDQuantFluxLoraLoader

comfyui/nodes/lora/flux.py

+import os
+import tempfile
+
+import folder_paths
+from safetensors.torch import save_file
+
+from nunchaku.lora.flux.comfyui_converter import comfyui2diffusers
+from nunchaku.lora.flux.diffusers_converter import convert_to_nunchaku_flux_lowrank_dict
+from nunchaku.lora.flux.xlab_converter import xlab2diffusers
+
+
+class SVDQuantFluxLoraLoader:
+    def __init__(self):
+        self.cur_lora_name = "None"
+
+    @classmethod
+    def INPUT_TYPES(s):
+        lora_name_list = [
+            "None",
+            *folder_paths.get_filename_list("loras"),
+            "aleksa-codes/flux-ghibsky-illustration/lora.safetensors",
+        ]
+
+        base_model_paths = [
+            "mit-han-lab/svdq-int4-flux.1-dev",
+            "mit-han-lab/svdq-int4-flux.1-schnell",
+            "mit-han-lab/svdq-int4-flux.1-canny-dev",
+            "mit-han-lab/svdq-int4-flux.1-depth-dev",
+            "mit-han-lab/svdq-int4-flux.1-fill-dev",
+        ]
+        prefix = os.path.join(folder_paths.models_dir, "diffusion_models")
+        local_base_model_folders = os.listdir(prefix)
+        local_base_model_folders = sorted(
+            [
+                folder
+                for folder in local_base_model_folders
+                if not folder.startswith(".") and os.path.isdir(os.path.join(prefix, folder))
+            ]
+        )
+        base_model_paths = local_base_model_folders + base_model_paths
+
+        return {
+            "required": {
+                "model": ("MODEL", {"tooltip": "The diffusion model the LoRA will be applied to."}),
+                "lora_name": (lora_name_list, {"tooltip": "The name of the LoRA."}),
+                "lora_format": (["comfyui", "diffusers", "svdquant", "xlab"], {"tooltip": "The format of the LoRA."}),
+                "base_model_name": (
+                    base_model_paths,
+                    {
+                        "tooltip": "If the lora format is SVDQuant, this field has no use. Otherwise, the base model's state dictionary is required for converting the LoRA weights to SVDQuant."
+                    },
+                ),
+                "lora_strength": (
+                    "FLOAT",
+                    {
+                        "default": 1.0,
+                        "min": -100.0,
+                        "max": 100.0,
+                        "step": 0.01,
+                        "tooltip": "How strongly to modify the diffusion model. This value can be negative.",
+                    },
+                ),
+            }
+        }
+
+    RETURN_TYPES = ("MODEL",)
+    OUTPUT_TOOLTIPS = ("The modified diffusion model.",)
+    FUNCTION = "load_lora"
+    TITLE = "SVDQuant FLUX.1 LoRA Loader"
+
+    CATEGORY = "SVDQuant"
+    DESCRIPTION = (
+        "LoRAs are used to modify the diffusion model, "
+        "altering the way in which latents are denoised such as applying styles. "
+        "Currently, only one LoRA nodes can be applied."
+    )
+
+    def load_lora(self, model, lora_name: str, lora_format: str, base_model_name: str, lora_strength: float):
+        if self.cur_lora_name == lora_name:
+            if self.cur_lora_name == "None":
+                pass  # Do nothing since the lora is None
+            else:
+                model.model.diffusion_model.model.set_lora_strength(lora_strength)
+        else:
+            if lora_name == "None":
+                model.model.diffusion_model.model.set_lora_strength(0)
+            else:
+                try:
+                    lora_path = folder_paths.get_full_path_or_raise("loras", lora_name)
+                except FileNotFoundError:
+                    lora_path = lora_name
+                if lora_format != "svdquant":
+                    if lora_format == "comfyui":
+                        input_lora = comfyui2diffusers(lora_path)
+                    elif lora_format == "xlab":
+                        input_lora = xlab2diffusers(lora_path)
+                    elif lora_format == "diffusers":
+                        input_lora = lora_path
+                    else:
+                        raise ValueError(f"Invalid LoRA format {lora_format}.")
+                    prefix = os.path.join(folder_paths.models_dir, "diffusion_models")
+                    base_model_path = os.path.join(prefix, base_model_name, "transformer_blocks.safetensors")
+                    if not os.path.exists(base_model_path):
+                        # download from huggingface
+                        base_model_path = os.path.join(base_model_name, "transformer_blocks.safetensors")
+                    state_dict = convert_to_nunchaku_flux_lowrank_dict(base_model_path, input_lora)
+
+                    with tempfile.NamedTemporaryFile(suffix=".safetensors", delete=True) as tmp_file:
+                        save_file(state_dict, tmp_file.name)
+                        model.model.diffusion_model.model.update_lora_params(tmp_file.name)
+                else:
+                    model.model.diffusion_model.model.update_lora_params(lora_path)
+                model.model.diffusion_model.model.set_lora_strength(lora_strength)
+            self.cur_lora_name = lora_name
+
+        return (model,)

comfyui/nodes/models/__init__.py

+from .flux import SVDQuantFluxDiTLoader
+from .text_encoder import SVDQuantTextEncoderLoader

comfyui/nodes.py → comfyui/nodes/models/flux.py

 import os
-import types
 
-import comfy.model_base
 import comfy.model_patcher
-import comfy.sd
 import folder_paths
 import GPUtil
 import torch
-import numpy as np
 from comfy.ldm.common_dit import pad_to_patch_size
 from comfy.supported_models import Flux, FluxSchnell
 from diffusers import FluxTransformer2DModel
 from einops import rearrange, repeat
 from torch import nn
-from transformers import T5EncoderModel
-from image_gen_aux import DepthPreprocessor
 
 from nunchaku.models.transformer_flux import NunchakuFluxTransformer2dModel
 
+
 class ComfyUIFluxForwardWrapper(nn.Module):
     def __init__(self, model: NunchakuFluxTransformer2dModel, config):
         super(ComfyUIFluxForwardWrapper, self).__init__()
@@ -41,9 +36,7 @@ class ComfyUIFluxForwardWrapper(nn.Module):
         patch_size = self.config["patch_size"]
         x = pad_to_patch_size(x, (patch_size, patch_size))
 
-        img = rearrange(
-            x, "b c (h ph) (w pw) -> b (h w) (c ph pw)", ph=patch_size, pw=patch_size
-        )
+        img = rearrange(x, "b c (h ph) (w pw) -> b (h w) (c ph pw)", ph=patch_size, pw=patch_size)
 
         h_len = (h + (patch_size // 2)) // patch_size
         w_len = (w + (patch_size // 2)) // patch_size
@@ -67,15 +60,14 @@ class ComfyUIFluxForwardWrapper(nn.Module):
             guidance=guidance if self.config["guidance_embed"] else None,
         ).sample
 
-        out = rearrange(
-            out, "b (h w) (c ph pw) -> b c (h ph) (w pw)", h=h_len, w=w_len, ph=2, pw=2
-        )[:, :, :h, :w]
+        out = rearrange(out, "b (h w) (c ph pw) -> b c (h ph) (w pw)", h=h_len, w=w_len, ph=2, pw=2)[:, :, :h, :w]
         return out
 
 
 class SVDQuantFluxDiTLoader:
     @classmethod
     def INPUT_TYPES(s):
+        # folder_paths.get_filename_list("loras"),
         model_paths = [
             "mit-han-lab/svdq-int4-flux.1-schnell",
             "mit-han-lab/svdq-int4-flux.1-dev",
@@ -83,31 +75,23 @@ class SVDQuantFluxDiTLoader:
             "mit-han-lab/svdq-int4-flux.1-depth-dev",
             "mit-han-lab/svdq-int4-flux.1-fill-dev",
         ]
-        prefix = "models/diffusion_models"
+        prefix = os.path.join(folder_paths.models_dir, "diffusion_models")
         local_folders = os.listdir(prefix)
         local_folders = sorted(
             [
                 folder
                 for folder in local_folders
-                if not folder.startswith(".")
-                and os.path.isdir(os.path.join(prefix, folder))
+                if not folder.startswith(".") and os.path.isdir(os.path.join(prefix, folder))
             ]
         )
-        model_paths.extend(local_folders)
+        model_paths = local_folders + model_paths
         ngpus = len(GPUtil.getGPUs())
         return {
             "required": {
                 "model_path": (model_paths,),
                 "device_id": (
                     "INT",
-                    {
-                        "default": 0,
-                        "min": 0,
-                        "max": ngpus,
-                        "step": 1,
-                        "display": "number",
-                        "lazy": True,
-                    },
+                    {"default": 0, "min": 0, "max": ngpus, "step": 1, "display": "number", "lazy": True},
                 ),
             }
         }
@@ -117,18 +101,14 @@ class SVDQuantFluxDiTLoader:
     CATEGORY = "SVDQuant"
     TITLE = "SVDQuant Flux DiT Loader"
 
-    def load_model(
-        self, model_path: str, device_id: int, **kwargs
-    ) -> tuple[FluxTransformer2DModel]:
+    def load_model(self, model_path: str, device_id: int, **kwargs) -> tuple[FluxTransformer2DModel]:
         device = f"cuda:{device_id}"
-        prefix = "models/diffusion_models"
+        prefix = os.path.join(folder_paths.models_dir, "diffusion_models")
         if os.path.exists(os.path.join(prefix, model_path)):
             model_path = os.path.join(prefix, model_path)
         else:
             model_path = model_path
-        transformer = NunchakuFluxTransformer2dModel.from_pretrained(model_path).to(
-            device
-        )
+        transformer = NunchakuFluxTransformer2dModel.from_pretrained(model_path).to(device)
         dit_config = {
             "image_model": "flux",
             "patch_size": 2,
@@ -158,9 +138,6 @@ class SVDQuantFluxDiTLoader:
             dit_config["in_channels"] = 64
             model_config = Flux(dit_config)
         else:
-            assert (
-                model_path == "mit-han-lab/svdq-int4-flux.1-dev"
-            ), f"model {model_path} not supported"
             dit_config["in_channels"] = 16
             model_config = Flux(dit_config)
 
@@ -168,239 +145,6 @@ class SVDQuantFluxDiTLoader:
         model_config.custom_operations = None
 
         model = model_config.get_model({})
-        model.diffusion_model = ComfyUIFluxForwardWrapper(
-            transformer, config=dit_config
-        )
+        model.diffusion_model = ComfyUIFluxForwardWrapper(transformer, config=dit_config)
         model = comfy.model_patcher.ModelPatcher(model, device, device_id)
         return (model,)
-
-
-def svdquant_t5_forward(
-    self: T5EncoderModel,
-    input_ids: torch.LongTensor,
-    attention_mask,
-    intermediate_output=None,
-    final_layer_norm_intermediate=True,
-    dtype: str | torch.dtype = torch.bfloat16,
-):
-    assert attention_mask is None
-    assert intermediate_output is None
-    assert final_layer_norm_intermediate
-    outputs = self.encoder(input_ids, attention_mask=attention_mask)
-    hidden_states = outputs["last_hidden_state"]
-    hidden_states = hidden_states.to(dtype=dtype)
-    return hidden_states, None
-
-
-class SVDQuantTextEncoderLoader:
-    @classmethod
-    def INPUT_TYPES(s):
-        model_paths = ["mit-han-lab/svdq-flux.1-t5"]
-        prefix = "models/text_encoders"
-        local_folders = os.listdir(prefix)
-        local_folders = sorted(
-            [
-                folder
-                for folder in local_folders
-                if not folder.startswith(".")
-                and os.path.isdir(os.path.join(prefix, folder))
-            ]
-        )
-        model_paths.extend(local_folders)
-        return {
-            "required": {
-                "model_type": (["flux"],),
-                "text_encoder1": (folder_paths.get_filename_list("text_encoders"),),
-                "text_encoder2": (folder_paths.get_filename_list("text_encoders"),),
-                "t5_min_length": (
-                    "INT",
-                    {
-                        "default": 512,
-                        "min": 256,
-                        "max": 1024,
-                        "step": 128,
-                        "display": "number",
-                        "lazy": True,
-                    },
-                ),
-                "t5_precision": (["BF16", "INT4"],),
-                "int4_model": (model_paths, {"tooltip": "The name of the INT4 model."}),
-            }
-        }
-
-    RETURN_TYPES = ("CLIP",)
-    FUNCTION = "load_text_encoder"
-
-    CATEGORY = "SVDQuant"
-
-    TITLE = "SVDQuant Text Encoder Loader"
-
-    def load_text_encoder(
-        self,
-        model_type: str,
-        text_encoder1: str,
-        text_encoder2: str,
-        t5_min_length: int,
-        t5_precision: str,
-        int4_model: str,
-    ):
-        text_encoder_path1 = folder_paths.get_full_path_or_raise(
-            "text_encoders", text_encoder1
-        )
-        text_encoder_path2 = folder_paths.get_full_path_or_raise(
-            "text_encoders", text_encoder2
-        )
-        if model_type == "flux":
-            clip_type = comfy.sd.CLIPType.FLUX
-        else:
-            raise ValueError(f"Unknown type {model_type}")
-
-        clip = comfy.sd.load_clip(
-            ckpt_paths=[text_encoder_path1, text_encoder_path2],
-            embedding_directory=folder_paths.get_folder_paths("embeddings"),
-            clip_type=clip_type,
-        )
-
-        if model_type == "flux":
-            clip.tokenizer.t5xxl.min_length = t5_min_length
-
-        if t5_precision == "INT4":
-            from nunchaku.models.text_encoder import NunchakuT5EncoderModel
-
-            transformer = clip.cond_stage_model.t5xxl.transformer
-            param = next(transformer.parameters())
-            dtype = param.dtype
-            device = param.device
-
-            prefix = "models/text_encoders"
-            if os.path.exists(os.path.join(prefix, int4_model)):
-                model_path = os.path.join(prefix, int4_model)
-            else:
-                model_path = int4_model
-            transformer = NunchakuT5EncoderModel.from_pretrained(model_path)
-            transformer.forward = types.MethodType(svdquant_t5_forward, transformer)
-            clip.cond_stage_model.t5xxl.transformer = (
-                transformer.to(device=device, dtype=dtype)
-                if device.type == "cuda"
-                else transformer
-            )
-
-        return (clip,)
-
-
-class SVDQuantLoraLoader:
-    def __init__(self):
-        self.cur_lora_name = "None"
-
-    @classmethod
-    def INPUT_TYPES(s):
-        hf_lora_names = ["anime", "ghibsky", "realism", "yarn", "sketch"]
-        lora_name_list = [
-            "None",
-            *folder_paths.get_filename_list("loras"),
-            *[
-                f"mit-han-lab/svdquant-models/svdq-flux.1-dev-lora-{n}.safetensors"
-                for n in hf_lora_names
-            ],
-        ]
-        return {
-            "required": {
-                "model": (
-                    "MODEL",
-                    {"tooltip": "The diffusion model the LoRA will be applied to."},
-                ),
-                "lora_name": (lora_name_list, {"tooltip": "The name of the LoRA."}),
-                "lora_strength": (
-                    "FLOAT",
-                    {
-                        "default": 1.0,
-                        "min": -100.0,
-                        "max": 100.0,
-                        "step": 0.01,
-                        "tooltip": "How strongly to modify the diffusion model. This value can be negative.",
-                    },
-                ),
-            }
-        }
-
-    RETURN_TYPES = ("MODEL",)
-    OUTPUT_TOOLTIPS = ("The modified diffusion model.",)
-    FUNCTION = "load_lora"
-    TITLE = "SVDQuant LoRA Loader"
-
-    CATEGORY = "SVDQuant"
-    DESCRIPTION = (
-        "LoRAs are used to modify the diffusion model, "
-        "altering the way in which latents are denoised such as applying styles. "
-        "Currently, only one LoRA nodes can be applied."
-    )
-
-    def load_lora(self, model, lora_name: str, lora_strength: float):
-        if self.cur_lora_name == lora_name:
-            if self.cur_lora_name == "None":
-                pass  # Do nothing since the lora is None
-            else:
-                model.model.diffusion_model.model.set_lora_strength(lora_strength)
-        else:
-            if lora_name == "None":
-                model.model.diffusion_model.model.set_lora_strength(0)
-            else:
-                try:
-                    lora_path = folder_paths.get_full_path_or_raise("loras", lora_name)
-                except FileNotFoundError:
-                    lora_path = lora_name
-
-                model.model.diffusion_model.model.update_lora_params(lora_path)
-                model.model.diffusion_model.model.set_lora_strength(lora_strength)
-            self.cur_lora_name = lora_name
-
-        return (model,)
-
-
-class DepthPreprocesser:
-    @classmethod
-    def INPUT_TYPES(s):
-        model_paths = ["LiheYoung/depth-anything-large-hf"]
-        prefix = "models/checkpoints"
-        local_folders = os.listdir(prefix)
-        local_folders = sorted(
-            [
-                folder
-                for folder in local_folders
-                if not folder.startswith(".")
-                and os.path.isdir(os.path.join(prefix, folder))
-            ]
-        )
-        model_paths.extend(local_folders)
-        return {
-            "required": {
-                "image": ("IMAGE", {}),
-                "model_path": (
-                    model_paths,
-                    {"tooltip": "Name of the depth preprocesser model."},
-                ),
-            }
-        }
-
-    RETURN_TYPES = ("IMAGE",)
-    FUNCTION = "depth_preprocess"
-    CATEGORY = "Flux.1"
-    TITLE = "Flux.1 Depth Preprocessor"
-
-    def depth_preprocess(self, image, model_path):
-        prefix = "models/checkpoints"
-        if os.path.exists(os.path.join(prefix, model_path)):
-            model_path = os.path.join(prefix, model_path)
-        processor = DepthPreprocessor.from_pretrained(model_path)
-        np_image = np.asarray(image)
-        np_result = np.array(processor(np_image)[0].convert("RGB"))
-        out_tensor = torch.from_numpy(np_result.astype(np.float32) / 255.0).unsqueeze(0)
-        return (out_tensor,)
-
-
-NODE_CLASS_MAPPINGS = {
-    "SVDQuantFluxDiTLoader": SVDQuantFluxDiTLoader,
-    "SVDQuantTextEncoderLoader": SVDQuantTextEncoderLoader,
-    "SVDQuantLoRALoader": SVDQuantLoraLoader,
-    "DepthPreprocesser": DepthPreprocesser
-}

comfyui/nodes/models/text_encoder.py

+import os
+import types
+
+import comfy.sd
+import folder_paths
+import torch
+from transformers import T5EncoderModel
+
+
+def svdquant_t5_forward(
+    self: T5EncoderModel,
+    input_ids: torch.LongTensor,
+    attention_mask,
+    intermediate_output=None,
+    final_layer_norm_intermediate=True,
+    dtype: str | torch.dtype = torch.bfloat16,
+):
+    assert attention_mask is None
+    assert intermediate_output is None
+    assert final_layer_norm_intermediate
+    outputs = self.encoder(input_ids, attention_mask=attention_mask)
+    hidden_states = outputs["last_hidden_state"]
+    hidden_states = hidden_states.to(dtype=dtype)
+    return hidden_states, None
+
+
+class SVDQuantTextEncoderLoader:
+    @classmethod
+    def INPUT_TYPES(s):
+        model_paths = ["mit-han-lab/svdq-flux.1-t5"]
+        prefix = os.path.join(folder_paths.models_dir, "text_encoders")
+        local_folders = os.listdir(prefix)
+        local_folders = sorted(
+            [
+                folder
+                for folder in local_folders
+                if not folder.startswith(".") and os.path.isdir(os.path.join(prefix, folder))
+            ]
+        )
+        model_paths.extend(local_folders)
+        return {
+            "required": {
+                "model_type": (["flux"],),
+                "text_encoder1": (folder_paths.get_filename_list("text_encoders"),),
+                "text_encoder2": (folder_paths.get_filename_list("text_encoders"),),
+                "t5_min_length": (
+                    "INT",
+                    {
+                        "default": 512,
+                        "min": 256,
+                        "max": 1024,
+                        "step": 128,
+                        "display": "number",
+                        "lazy": True,
+                    },
+                ),
+                "t5_precision": (["BF16", "INT4"],),
+                "int4_model": (model_paths, {"tooltip": "The name of the INT4 model."}),
+            }
+        }
+
+    RETURN_TYPES = ("CLIP",)
+    FUNCTION = "load_text_encoder"
+
+    CATEGORY = "SVDQuant"
+
+    TITLE = "SVDQuant Text Encoder Loader"
+
+    def load_text_encoder(
+        self,
+        model_type: str,
+        text_encoder1: str,
+        text_encoder2: str,
+        t5_min_length: int,
+        t5_precision: str,
+        int4_model: str,
+    ):
+        text_encoder_path1 = folder_paths.get_full_path_or_raise("text_encoders", text_encoder1)
+        text_encoder_path2 = folder_paths.get_full_path_or_raise("text_encoders", text_encoder2)
+        if model_type == "flux":
+            clip_type = comfy.sd.CLIPType.FLUX
+        else:
+            raise ValueError(f"Unknown type {model_type}")
+
+        clip = comfy.sd.load_clip(
+            ckpt_paths=[text_encoder_path1, text_encoder_path2],
+            embedding_directory=folder_paths.get_folder_paths("embeddings"),
+            clip_type=clip_type,
+        )
+
+        if model_type == "flux":
+            clip.tokenizer.t5xxl.min_length = t5_min_length
+
+        if t5_precision == "INT4":
+            from nunchaku.models.text_encoder import NunchakuT5EncoderModel
+
+            transformer = clip.cond_stage_model.t5xxl.transformer
+            param = next(transformer.parameters())
+            dtype = param.dtype
+            device = param.device
+
+            prefix = "models/text_encoders"
+            if os.path.exists(os.path.join(prefix, int4_model)):
+                model_path = os.path.join(prefix, int4_model)
+            else:
+                model_path = int4_model
+            transformer = NunchakuT5EncoderModel.from_pretrained(model_path)
+            transformer.forward = types.MethodType(svdquant_t5_forward, transformer)
+            clip.cond_stage_model.t5xxl.transformer = (
+                transformer.to(device=device, dtype=dtype) if device.type == "cuda" else transformer
+            )
+
+        return (clip,)

comfyui/nodes/preprocessors/__init__.py

+from .depth import FluxDepthPreprocessor

comfyui/nodes/preprocessors/depth.py

+import os
+
+import folder_paths
+import numpy as np
+import torch
+from image_gen_aux import DepthPreprocessor
+
+
+class FluxDepthPreprocessor:
+    @classmethod
+    def INPUT_TYPES(s):
+        model_paths = ["LiheYoung/depth-anything-large-hf"]
+        prefix = os.path.join(folder_paths.models_dir, "checkpoints")
+        local_folders = os.listdir(prefix)
+        local_folders = sorted(
+            [
+                folder
+                for folder in local_folders
+                if not folder.startswith(".") and os.path.isdir(os.path.join(prefix, folder))
+            ]
+        )
+        model_paths = local_folders + model_paths
+        return {
+            "required": {
+                "image": ("IMAGE", {}),
+                "model_path": (
+                    model_paths,
+                    {"tooltip": "Name of the depth preprocessor model."},
+                ),
+            }
+        }
+
+    RETURN_TYPES = ("IMAGE",)
+    FUNCTION = "depth_preprocess"
+    CATEGORY = "SVDQuant"
+    TITLE = "FLUX.1 Depth Preprocessor"
+
+    def depth_preprocess(self, image, model_path):
+        prefix = os.path.join(folder_paths.models_dir, "checkpoints")
+        if os.path.exists(os.path.join(prefix, model_path)):
+            model_path = os.path.join(prefix, model_path)
+        processor = DepthPreprocessor.from_pretrained(model_path)
+        np_image = np.asarray(image)
+        np_result = np.array(processor(np_image)[0].convert("RGB"))
+        out_tensor = torch.from_numpy(np_result.astype(np.float32) / 255.0).unsqueeze(0)
+        return (out_tensor,)

comfyui/pyproject.toml

+[project]
+name = "svdquant"
+description = "SVDQuant ComfyUI Node. SVDQuant is a new post-training training quantization paradigm for diffusion models, which quantize both the weights and activations of FLUX.1 to 4 bits, achieving 3.5× memory and 8.7× latency reduction on a 16GB laptop 4090 GPU."
+version = "0.1.1"
+license = { file = "LICENSE.txt" }
+dependencies = []
+requires-python = ">=3.11, <3.13"
+
+[project.urls]
+Repository = "https://github.com/mit-han-lab/nunchaku"
+#  Used by Comfy Registry https://comfyregistry.org
+
+[tool.comfy]
+PublisherId = "lmxyy1999"
+DisplayName = "svdquant"
+Icon = "https://raw.githubusercontent.com/mit-han-lab/nunchaku/bfd9aa3ae84f51a26414f1600d725a0098472820/assets/logo.svg"

comfyui/requirements.txt

+GPUtil
+diffusers>=0.32.2
+accelerate
+sentencepiece
+protobuf
+huggingface_hub
+git+https://github.com/asomoza/image_gen_aux.git
\ No newline at end of file

comfyui/workflows/flux.1-canny-svdquant.json → comfyui/workflows/svdq-flux.1-canny.json

@@ -57,7 +57,7 @@
         "Node name for S&R": "KSampler"
       },
       "widgets_values": [
-        646386750200194,
+        896617285614695,
         "randomize",
         20,
         1,
@@ -323,51 +323,6 @@
         "default"
       ]
     },
-    {
-      "id": 7,
-      "type": "CLIPTextEncode",
-      "pos": [
-        307,
-        282
-      ],
-      "size": [
-        425.27801513671875,
-        180.6060791015625
-      ],
-      "flags": {
-        "collapsed": true
-      },
-      "order": 5,
-      "mode": 0,
-      "inputs": [
-        {
-          "name": "clip",
-          "type": "CLIP",
-          "link": 63,
-          "label": "clip"
-        }
-      ],
-      "outputs": [
-        {
-          "name": "CONDITIONING",
-          "type": "CONDITIONING",
-          "links": [
-            68
-          ],
-          "slot_index": 0,
-          "label": "CONDITIONING"
-        }
-      ],
-      "title": "CLIP Text Encode (Negative Prompt)",
-      "properties": {
-        "Node name for S&R": "CLIPTextEncode"
-      },
-      "widgets_values": [
-        ""
-      ],
-      "color": "#322",
-      "bgcolor": "#533"
-    },
     {
       "id": 23,
       "type": "CLIPTextEncode",
@@ -483,36 +438,44 @@
       ]
     },
     {
-      "id": 36,
-      "type": "SVDQuantFluxDiTLoader",
+      "id": 38,
+      "type": "ImageScale",
       "pos": [
-        865.4989624023438,
-        -95.86973571777344
+        379.69903564453125,
+        565.2651977539062
       ],
       "size": [
         315,
-        82
+        130
       ],
       "flags": {},
-      "order": 2,
+      "order": 6,
       "mode": 0,
-      "inputs": [],
+      "inputs": [
+        {
+          "name": "image",
+          "type": "IMAGE",
+          "link": 75
+        }
+      ],
       "outputs": [
         {
-          "name": "MODEL",
-          "type": "MODEL",
+          "name": "IMAGE",
+          "type": "IMAGE",
           "links": [
-            71
+            76
           ],
           "slot_index": 0
         }
       ],
       "properties": {
-        "Node name for S&R": "SVDQuantFluxDiTLoader"
+        "Node name for S&R": "ImageScale"
       },
       "widgets_values": [
-        "mit-han-lab/svdq-int4-flux.1-canny-dev",
-        0
+        "nearest-exact",
+        1024,
+        1024,
+        "center"
       ]
     },
     {
@@ -527,7 +490,7 @@
         314.0000305175781
       ],
       "flags": {},
-      "order": 3,
+      "order": 2,
       "mode": 0,
       "inputs": [],
       "outputs": [
@@ -558,44 +521,81 @@
       ]
     },
     {
-      "id": 38,
-      "type": "ImageScale",
+      "id": 7,
+      "type": "CLIPTextEncode",
       "pos": [
-        379.69903564453125,
-        565.2651977539062
+        323.8695068359375,
+        387.9589538574219
       ],
       "size": [
-        315,
-        130
+        425.27801513671875,
+        180.6060791015625
       ],
-      "flags": {},
-      "order": 6,
+      "flags": {
+        "collapsed": true
+      },
+      "order": 5,
       "mode": 0,
       "inputs": [
         {
-          "name": "image",
-          "type": "IMAGE",
-          "link": 75
+          "name": "clip",
+          "type": "CLIP",
+          "link": 63,
+          "label": "clip"
         }
       ],
       "outputs": [
         {
-          "name": "IMAGE",
-          "type": "IMAGE",
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
           "links": [
-            76
+            68
+          ],
+          "slot_index": 0,
+          "label": "CONDITIONING"
+        }
+      ],
+      "title": "CLIP Text Encode (Negative Prompt)",
+      "properties": {
+        "Node name for S&R": "CLIPTextEncode"
+      },
+      "widgets_values": [
+        ""
+      ],
+      "color": "#322",
+      "bgcolor": "#533"
+    },
+    {
+      "id": 36,
+      "type": "SVDQuantFluxDiTLoader",
+      "pos": [
+        823.9686279296875,
+        -126.4416275024414
+      ],
+      "size": [
+        395.6002197265625,
+        105.77959442138672
+      ],
+      "flags": {},
+      "order": 3,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "MODEL",
+          "type": "MODEL",
+          "links": [
+            71
           ],
           "slot_index": 0
         }
       ],
       "properties": {
-        "Node name for S&R": "ImageScale"
+        "Node name for S&R": "SVDQuantFluxDiTLoader"
       },
       "widgets_values": [
-        "nearest-exact",
-        1024,
-        1024,
-        "center"
+        "mit-han-lab/svdq-int4-flux.1-canny-dev",
+        0
       ]
     }
   ],
@@ -743,9 +743,12 @@
     "ds": {
       "scale": 0.895430243255241,
       "offset": [
-        203.26101803057463,
-        215.36536277004458
+        838.4305404853558,
+        332.05158795287764
       ]
+    },
+    "node_versions": {
+      "comfy-core": "0.3.14"
     }
   },
   "version": 0.4

comfyui/workflows/flux.1-depth-svdquant.json → comfyui/workflows/svdq-flux.1-depth.json

 {
-  "last_node_id": 44,
-  "last_link_id": 85,
+  "last_node_id": 45,
+  "last_link_id": 88,
   "nodes": [
     {
       "id": 7,
@@ -124,40 +124,6 @@
         10
       ]
     },
-    {
-      "id": 32,
-      "type": "VAELoader",
-      "pos": [
-        656,
-        165
-      ],
-      "size": [
-        315,
-        58
-      ],
-      "flags": {},
-      "order": 1,
-      "mode": 0,
-      "inputs": [],
-      "outputs": [
-        {
-          "name": "VAE",
-          "type": "VAE",
-          "links": [
-            60,
-            69
-          ],
-          "slot_index": 0,
-          "label": "VAE"
-        }
-      ],
-      "properties": {
-        "Node name for S&R": "VAELoader"
-      },
-      "widgets_values": [
-        "ae.safetensors"
-      ]
-    },
     {
       "id": 3,
       "type": "KSampler",
@@ -213,7 +179,7 @@
         "Node name for S&R": "KSampler"
       },
       "widgets_values": [
-        718322679777603,
+        704308966490490,
         "randomize",
         20,
         1,
@@ -223,118 +189,144 @@
       ]
     },
     {
-      "id": 35,
-      "type": "InstructPixToPixConditioning",
+      "id": 39,
+      "type": "SVDQuantFluxDiTLoader",
       "pos": [
-        1008,
-        118
+        707.80908203125,
+        -172.0343017578125
       ],
       "size": [
-        235.1999969482422,
-        86
+        315,
+        82
+      ],
+      "flags": {},
+      "order": 1,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "MODEL",
+          "type": "MODEL",
+          "links": [
+            78
+          ],
+          "slot_index": 0
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "SVDQuantFluxDiTLoader"
+      },
+      "widgets_values": [
+        "mit-han-lab/svdq-int4-flux.1-depth-dev",
+        0
+      ]
+    },
+    {
+      "id": 43,
+      "type": "PreviewImage",
+      "pos": [
+        1001.3873291015625,
+        432.09039306640625
+      ],
+      "size": [
+        571.5869140625,
+        625.5296020507812
       ],
       "flags": {},
       "order": 9,
       "mode": 0,
       "inputs": [
         {
-          "name": "positive",
-          "type": "CONDITIONING",
-          "link": 67,
-          "label": "positive"
+          "name": "images",
+          "type": "IMAGE",
+          "link": 87,
+          "label": "images"
+        }
+      ],
+      "outputs": [],
+      "properties": {
+        "Node name for S&R": "PreviewImage"
+      },
+      "widgets_values": []
     },
     {
-          "name": "negative",
-          "type": "CONDITIONING",
-          "link": 68,
-          "label": "negative"
+      "id": 8,
+      "type": "VAEDecode",
+      "pos": [
+        1620,
+        98
+      ],
+      "size": [
+        210,
+        46
+      ],
+      "flags": {},
+      "order": 12,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "samples",
+          "type": "LATENT",
+          "link": 7,
+          "label": "samples"
         },
         {
           "name": "vae",
           "type": "VAE",
-          "link": 69,
+          "link": 60,
           "label": "vae"
-        },
-        {
-          "name": "pixels",
-          "type": "IMAGE",
-          "link": 80,
-          "label": "pixels"
         }
       ],
       "outputs": [
         {
-          "name": "positive",
-          "type": "CONDITIONING",
+          "name": "IMAGE",
+          "type": "IMAGE",
           "links": [
-            64
+            85
           ],
           "slot_index": 0,
-          "label": "positive"
-        },
-        {
-          "name": "negative",
-          "type": "CONDITIONING",
-          "links": [
-            65
-          ],
-          "slot_index": 1,
-          "label": "negative"
-        },
-        {
-          "name": "latent",
-          "type": "LATENT",
-          "links": [
-            73
-          ],
-          "slot_index": 2,
-          "label": "latent"
+          "label": "IMAGE"
         }
       ],
       "properties": {
-        "Node name for S&R": "InstructPixToPixConditioning"
+        "Node name for S&R": "VAEDecode"
       },
       "widgets_values": []
     },
     {
-      "id": 39,
-      "type": "SVDQuantFluxDiTLoader",
+      "id": 44,
+      "type": "SaveImage",
       "pos": [
-        707.80908203125,
-        -172.0343017578125
+        1912.7984619140625,
+        109.0069580078125
       ],
       "size": [
-        315,
-        82
+        828.9535522460938,
+        893.8475341796875
       ],
       "flags": {},
-      "order": 2,
+      "order": 13,
       "mode": 0,
-      "inputs": [],
-      "outputs": [
+      "inputs": [
         {
-          "name": "MODEL",
-          "type": "MODEL",
-          "links": [
-            78
-          ],
-          "slot_index": 0
+          "name": "images",
+          "type": "IMAGE",
+          "link": 85,
+          "label": "images"
         }
       ],
-      "properties": {
-        "Node name for S&R": "SVDQuantFluxDiTLoader"
-      },
+      "outputs": [],
+      "properties": {},
       "widgets_values": [
-        "mit-han-lab/svdq-int4-flux.1-depth-dev",
-        0
+        "ComfyUI"
       ]
     },
     {
       "id": 42,
       "type": "ImageScale",
       "pos": [
-        378.3890686035156,
-        472.7001953125
+        174.98765563964844,
+        450.5818786621094
       ],
       "size": [
         315,
@@ -355,7 +347,7 @@
           "name": "IMAGE",
           "type": "IMAGE",
           "links": [
-            83
+            86
           ],
           "slot_index": 0
         }
@@ -370,48 +362,6 @@
         "center"
       ]
     },
-    {
-      "id": 17,
-      "type": "LoadImage",
-      "pos": [
-        30.604948043823242,
-        419.3930358886719
-      ],
-      "size": [
-        315,
-        314.0000305175781
-      ],
-      "flags": {},
-      "order": 3,
-      "mode": 0,
-      "inputs": [],
-      "outputs": [
-        {
-          "name": "IMAGE",
-          "type": "IMAGE",
-          "links": [
-            82
-          ],
-          "slot_index": 0,
-          "shape": 3,
-          "label": "IMAGE"
-        },
-        {
-          "name": "MASK",
-          "type": "MASK",
-          "links": null,
-          "shape": 3,
-          "label": "MASK"
-        }
-      ],
-      "properties": {
-        "Node name for S&R": "LoadImage"
-      },
-      "widgets_values": [
-        "robot.png",
-        "image"
-      ]
-    },
     {
       "id": 23,
       "type": "CLIPTextEncode",
@@ -450,104 +400,138 @@
         "Node name for S&R": "CLIPTextEncode"
       },
       "widgets_values": [
-        "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
+        "ethereal fantasy concept art of A logo of 'MIT HAN Lab'. magnificent, celestial, ethereal, painterly, epic, majestic, magical, fantasy art, cover art, dreamy"
       ],
       "color": "#232",
       "bgcolor": "#353"
     },
     {
-      "id": 43,
-      "type": "PreviewImage",
+      "id": 32,
+      "type": "VAELoader",
       "pos": [
-        1001.3873291015625,
-        432.09039306640625
+        630.5574951171875,
+        280.441650390625
       ],
       "size": [
-        571.5869140625,
-        625.5296020507812
+        315,
+        58
       ],
       "flags": {},
-      "order": 10,
+      "order": 2,
       "mode": 0,
-      "inputs": [
+      "inputs": [],
+      "outputs": [
         {
-          "name": "images",
-          "type": "IMAGE",
-          "link": 84,
-          "label": "images"
+          "name": "VAE",
+          "type": "VAE",
+          "links": [
+            60,
+            69
+          ],
+          "slot_index": 0,
+          "label": "VAE"
         }
       ],
-      "outputs": [],
       "properties": {
-        "Node name for S&R": "PreviewImage"
+        "Node name for S&R": "VAELoader"
       },
-      "widgets_values": []
+      "widgets_values": [
+        "ae.safetensors"
+      ]
     },
     {
-      "id": 40,
-      "type": "DepthPreprocesser",
+      "id": 35,
+      "type": "InstructPixToPixConditioning",
       "pos": [
-        639.0159301757812,
-        350.06134033203125
+        1008,
+        118
       ],
       "size": [
-        315,
-        58
+        235.1999969482422,
+        86
       ],
       "flags": {},
-      "order": 8,
+      "order": 10,
       "mode": 0,
       "inputs": [
         {
-          "name": "image",
+          "name": "positive",
+          "type": "CONDITIONING",
+          "link": 67,
+          "label": "positive"
+        },
+        {
+          "name": "negative",
+          "type": "CONDITIONING",
+          "link": 68,
+          "label": "negative"
+        },
+        {
+          "name": "vae",
+          "type": "VAE",
+          "link": 69,
+          "label": "vae"
+        },
+        {
+          "name": "pixels",
           "type": "IMAGE",
-          "link": 83
+          "link": 88,
+          "label": "pixels"
         }
       ],
       "outputs": [
         {
-          "name": "IMAGE",
-          "type": "IMAGE",
+          "name": "positive",
+          "type": "CONDITIONING",
           "links": [
-            80,
-            84
+            64
           ],
-          "slot_index": 0
+          "slot_index": 0,
+          "label": "positive"
+        },
+        {
+          "name": "negative",
+          "type": "CONDITIONING",
+          "links": [
+            65
+          ],
+          "slot_index": 1,
+          "label": "negative"
+        },
+        {
+          "name": "latent",
+          "type": "LATENT",
+          "links": [
+            73
+          ],
+          "slot_index": 2,
+          "label": "latent"
         }
       ],
       "properties": {
-        "Node name for S&R": "DepthPreprocesser"
+        "Node name for S&R": "InstructPixToPixConditioning"
       },
-      "widgets_values": [
-        "LiheYoung/depth-anything-large-hf"
-      ]
+      "widgets_values": []
     },
     {
-      "id": 8,
-      "type": "VAEDecode",
+      "id": 45,
+      "type": "SVDQuantDepthPreprocessor",
       "pos": [
-        1620,
-        98
+        620.8667602539062,
+        445.8307189941406
       ],
       "size": [
-        210,
-        46
+        315,
+        58
       ],
       "flags": {},
-      "order": 12,
+      "order": 8,
       "mode": 0,
       "inputs": [
         {
-          "name": "samples",
-          "type": "LATENT",
-          "link": 7,
-          "label": "samples"
-        },
-        {
-          "name": "vae",
-          "type": "VAE",
-          "link": 60,
-          "label": "vae"
+          "name": "image",
+          "type": "IMAGE",
+          "link": 86
         }
       ],
       "outputs": [
@@ -555,43 +539,59 @@
           "name": "IMAGE",
           "type": "IMAGE",
           "links": [
-            85
+            87,
+            88
           ],
-          "slot_index": 0,
-          "label": "IMAGE"
+          "slot_index": 0
         }
       ],
       "properties": {
-        "Node name for S&R": "VAEDecode"
+        "Node name for S&R": "FluxDepthPreprocessor"
       },
-      "widgets_values": []
+      "widgets_values": [
+        "LiheYoung/depth-anything-large-hf"
+      ]
     },
     {
-      "id": 44,
-      "type": "SaveImage",
+      "id": 17,
+      "type": "LoadImage",
       "pos": [
-        1912.7984619140625,
-        109.0069580078125
+        -152.99026489257812,
+        409.8635559082031
       ],
       "size": [
-        828.9535522460938,
-        893.8475341796875
+        315,
+        314.0000305175781
       ],
       "flags": {},
-      "order": 13,
+      "order": 3,
       "mode": 0,
-      "inputs": [
+      "inputs": [],
+      "outputs": [
         {
-          "name": "images",
+          "name": "IMAGE",
           "type": "IMAGE",
-          "link": 85,
-          "label": "images"
+          "links": [
+            82
+          ],
+          "slot_index": 0,
+          "shape": 3,
+          "label": "IMAGE"
+        },
+        {
+          "name": "MASK",
+          "type": "MASK",
+          "links": null,
+          "shape": 3,
+          "label": "MASK"
         }
       ],
-      "outputs": [],
-      "properties": {},
+      "properties": {
+        "Node name for S&R": "LoadImage"
+      },
       "widgets_values": [
-        "ComfyUI"
+        "logo_example.png",
+        "image"
       ]
     }
   ],
@@ -693,43 +693,43 @@
       "MODEL"
     ],
     [
-      80,
-      40,
+      82,
+      17,
+      0,
+      42,
       0,
-      35,
-      3,
       "IMAGE"
     ],
     [
-      82,
-      17,
+      85,
+      8,
       0,
-      42,
+      44,
       0,
       "IMAGE"
     ],
     [
-      83,
+      86,
       42,
       0,
-      40,
+      45,
       0,
       "IMAGE"
     ],
     [
-      84,
-      40,
+      87,
+      45,
       0,
       43,
       0,
       "IMAGE"
     ],
     [
-      85,
-      8,
-      0,
-      44,
+      88,
+      45,
       0,
+      35,
+      3,
       "IMAGE"
     ]
   ],
@@ -737,11 +737,14 @@
   "config": {},
   "extra": {
     "ds": {
-      "scale": 0.6115909044841502,
+      "scale": 0.8140274938684042,
       "offset": [
-        724.4911189218763,
-        518.3043483917891
+        1060.3416359459316,
+        529.8567933439979
       ]
+    },
+    "node_versions": {
+      "comfy-core": "0.3.14"
     }
   },
   "version": 0.4

comfyui/workflows/flux.1-fill-svdquant.json → comfyui/workflows/svdq-flux.1-fill.json

@@ -167,7 +167,7 @@
         "Node name for S&R": "KSampler"
       },
       "widgets_values": [
-        1078304127779394,
+        54184445162233,
         "randomize",
         20,
         1,
@@ -388,74 +388,6 @@
         0
       ]
     },
-    {
-      "id": 48,
-      "type": "Note",
-      "pos": [
-        466.9884033203125,
-        643.9080810546875
-      ],
-      "size": [
-        314.99755859375,
-        117.98363494873047
-      ],
-      "flags": {},
-      "order": 3,
-      "mode": 0,
-      "inputs": [],
-      "outputs": [],
-      "properties": {
-        "text": ""
-      },
-      "widgets_values": [
-        "To add mask for fill inpainting, right click on the uploaded image and select \"Open in MaskEditor\". Use the brush tool to add masking and click save to continue."
-      ],
-      "color": "#432",
-      "bgcolor": "#653"
-    },
-    {
-      "id": 17,
-      "type": "LoadImage",
-      "pos": [
-        126.66505432128906,
-        460.53631591796875
-      ],
-      "size": [
-        315,
-        314.0000305175781
-      ],
-      "flags": {},
-      "order": 4,
-      "mode": 0,
-      "inputs": [],
-      "outputs": [
-        {
-          "name": "IMAGE",
-          "type": "IMAGE",
-          "links": [
-            105
-          ],
-          "slot_index": 0,
-          "shape": 3
-        },
-        {
-          "name": "MASK",
-          "type": "MASK",
-          "links": [
-            106
-          ],
-          "slot_index": 1,
-          "shape": 3
-        }
-      ],
-      "properties": {
-        "Node name for S&R": "LoadImage"
-      },
-      "widgets_values": [
-        "clipspace/clipspace-mask-123191.png [input]",
-        "image"
-      ]
-    },
     {
       "id": 58,
       "type": "ImageAndMaskResizeNode",
@@ -547,10 +479,80 @@
         "Node name for S&R": "CLIPTextEncode"
       },
       "widgets_values": [
-        "A robot with a closed eye pink face is giving a presentation"
+        "A wooden basket of a cat."
       ],
       "color": "#232",
       "bgcolor": "#353"
+    },
+    {
+      "id": 48,
+      "type": "Note",
+      "pos": [
+        210.1423797607422,
+        572.7574462890625
+      ],
+      "size": [
+        266.4635925292969,
+        132.3040771484375
+      ],
+      "flags": {},
+      "order": 3,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [],
+      "properties": {
+        "text": ""
+      },
+      "widgets_values": [
+        "To add mask for fill inpainting, right click on the uploaded image and select \"Open in MaskEditor\". Use the brush tool to add masking and click save to continue."
+      ],
+      "color": "#432",
+      "bgcolor": "#653"
+    },
+    {
+      "id": 17,
+      "type": "LoadImage",
+      "pos": [
+        -225.73123168945312,
+        316.9361267089844
+      ],
+      "size": [
+        423.5578308105469,
+        437.250732421875
+      ],
+      "flags": {
+        "collapsed": false
+      },
+      "order": 4,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "IMAGE",
+          "type": "IMAGE",
+          "links": [
+            105
+          ],
+          "slot_index": 0,
+          "shape": 3
+        },
+        {
+          "name": "MASK",
+          "type": "MASK",
+          "links": [
+            106
+          ],
+          "slot_index": 1,
+          "shape": 3
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LoadImage"
+      },
+      "widgets_values": [
+        "clipspace/clipspace-mask-8389612.599999994.png [input]",
+        "image"
+      ]
     }
   ],
   "links": [
@@ -697,8 +699,8 @@
     "ds": {
       "scale": 0.8390545288824038,
       "offset": [
-        361.8437326514503,
-        242.3368651567008
+        815.2093059315082,
+        185.9955477896796
       ]
     },
     "node_versions": {

examples/flux.1-dev-lora.py

+import torch
+from diffusers import FluxPipeline
+
+from nunchaku.models.transformer_flux import NunchakuFluxTransformer2dModel
+
+transformer = NunchakuFluxTransformer2dModel.from_pretrained("mit-han-lab/svdq-int4-flux.1-dev")
+pipeline = FluxPipeline.from_pretrained(
+    "black-forest-labs/FLUX.1-dev", transformer=transformer, torch_dtype=torch.bfloat16
+).to("cuda")
+
+### LoRA Related Code ###
+transformer.update_lora_params(
+    "mit-han-lab/svdquant-lora-collection/svdq-int4-flux.1-dev-ghibsky.safetensors"
+)  # Path to your converted LoRA safetensors, can also be a remote HuggingFace path
+transformer.set_lora_strength(1)  # Your LoRA strength here
+### End of LoRA Related Code ###
+
+image = pipeline(
+    "GHIBSKY style, cozy mountain cabin covered in snow, with smoke curling from the chimney and a warm, inviting light spilling through the windows",
+    num_inference_steps=25,
+    guidance_scale=3.5,
+).images[0]
+image.save("flux.1-dev-ghibsky.png")