Unverified Commit 9a2eaed0 authored by Sayak Paul's avatar Sayak Paul Committed by GitHub
Browse files

[LoRA] support lightx2v lora in wan (#12040)

* support lightx2v lora in wan

* add docsa.

* reviewer feedback

* empty
parent 0c71189a
...@@ -29,6 +29,7 @@ ...@@ -29,6 +29,7 @@
You can find all the original Wan2.1 checkpoints under the [Wan-AI](https://huggingface.co/Wan-AI) organization. You can find all the original Wan2.1 checkpoints under the [Wan-AI](https://huggingface.co/Wan-AI) organization.
The following Wan models are supported in Diffusers: The following Wan models are supported in Diffusers:
- [Wan 2.1 T2V 1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers) - [Wan 2.1 T2V 1.3B](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers)
- [Wan 2.1 T2V 14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B-Diffusers) - [Wan 2.1 T2V 14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B-Diffusers)
- [Wan 2.1 I2V 14B - 480P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P-Diffusers) - [Wan 2.1 I2V 14B - 480P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P-Diffusers)
...@@ -36,6 +37,9 @@ The following Wan models are supported in Diffusers: ...@@ -36,6 +37,9 @@ The following Wan models are supported in Diffusers:
- [Wan 2.1 FLF2V 14B - 720P](https://huggingface.co/Wan-AI/Wan2.1-FLF2V-14B-720P-diffusers) - [Wan 2.1 FLF2V 14B - 720P](https://huggingface.co/Wan-AI/Wan2.1-FLF2V-14B-720P-diffusers)
- [Wan 2.1 VACE 1.3B](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B-diffusers) - [Wan 2.1 VACE 1.3B](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B-diffusers)
- [Wan 2.1 VACE 14B](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers) - [Wan 2.1 VACE 14B](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B-diffusers)
- [Wan 2.2 T2V 14B](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B-Diffusers)
- [Wan 2.2 I2V 14B](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B-Diffusers)
- [Wan 2.2 TI2V 5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B-Diffusers)
> [!TIP] > [!TIP]
> Click on the Wan2.1 models in the right sidebar for more examples of video generation. > Click on the Wan2.1 models in the right sidebar for more examples of video generation.
...@@ -327,6 +331,8 @@ The general rule of thumb to keep in mind when preparing inputs for the VACE pip ...@@ -327,6 +331,8 @@ The general rule of thumb to keep in mind when preparing inputs for the VACE pip
- Try lower `shift` values (`2.0` to `5.0`) for lower resolution videos and higher `shift` values (`7.0` to `12.0`) for higher resolution images. - Try lower `shift` values (`2.0` to `5.0`) for lower resolution videos and higher `shift` values (`7.0` to `12.0`) for higher resolution images.
- Wan 2.1 and 2.2 support using [LightX2V LoRAs](https://huggingface.co/Kijai/WanVideo_comfy/tree/main/Lightx2v) to speed up inference. Using them on Wan 2.2 is slightly more involed. Refer to [this code snippet](https://github.com/huggingface/diffusers/pull/12040#issuecomment-3144185272) to learn more.
## WanPipeline ## WanPipeline
[[autodoc]] WanPipeline [[autodoc]] WanPipeline
......
...@@ -1974,6 +1974,10 @@ def _convert_non_diffusers_wan_lora_to_diffusers(state_dict): ...@@ -1974,6 +1974,10 @@ def _convert_non_diffusers_wan_lora_to_diffusers(state_dict):
converted_key = f"condition_embedder.image_embedder.{img_ours}.lora_B.weight" converted_key = f"condition_embedder.image_embedder.{img_ours}.lora_B.weight"
if original_key in original_state_dict: if original_key in original_state_dict:
converted_state_dict[converted_key] = original_state_dict.pop(original_key) converted_state_dict[converted_key] = original_state_dict.pop(original_key)
bias_key_theirs = original_key.removesuffix(f".{lora_up_key}.weight") + ".diff_b"
if bias_key_theirs in original_state_dict:
bias_key = converted_key.removesuffix(".weight") + ".bias"
converted_state_dict[bias_key] = original_state_dict.pop(bias_key_theirs)
if len(original_state_dict) > 0: if len(original_state_dict) > 0:
diff = all(".diff" in k for k in original_state_dict) diff = all(".diff" in k for k in original_state_dict)
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment