model_structure.md 21.8 KB
Newer Older
gushiqiao's avatar
gushiqiao committed
1
# Model Format and Loading Guide
gushiqiao's avatar
gushiqiao committed
2
3
4

## 📖 Overview

gushiqiao's avatar
gushiqiao committed
5
LightX2V is a flexible video generation inference framework that supports multiple model sources and formats, providing users with rich options:
gushiqiao's avatar
gushiqiao committed
6

gushiqiao's avatar
gushiqiao committed
7
8
9
-**Wan Official Models**: Directly compatible with officially released complete models from Wan2.1 and Wan2.2
-**Single-File Models**: Supports single-file format models released by LightX2V (including quantized versions)
-**LoRA Models**: Supports loading distilled LoRAs released by LightX2V
gushiqiao's avatar
gushiqiao committed
10

gushiqiao's avatar
gushiqiao committed
11
This document provides detailed instructions on how to use various model formats, configuration parameters, and best practices.
gushiqiao's avatar
gushiqiao committed
12

gushiqiao's avatar
gushiqiao committed
13
---
gushiqiao's avatar
gushiqiao committed
14

gushiqiao's avatar
gushiqiao committed
15
## 🗂️ Format 1: Wan Official Models
gushiqiao's avatar
gushiqiao committed
16

gushiqiao's avatar
gushiqiao committed
17
18
19
### Model Repositories
- [Wan2.1 Collection](https://huggingface.co/collections/Wan-AI/wan21-68ac4ba85372ae5a8e282a1b)
- [Wan2.2 Collection](https://huggingface.co/collections/Wan-AI/wan22-68ac4ae80a8b477e79636fc8)
gushiqiao's avatar
gushiqiao committed
20

gushiqiao's avatar
gushiqiao committed
21
22
23
24
25
26
27
### Model Features
- **Official Guarantee**: Complete models officially released by Wan-AI with highest quality
- **Complete Components**: Includes all necessary components (DIT, T5, CLIP, VAE)
- **Original Precision**: Uses BF16/FP32 precision with no quantization loss
- **Strong Compatibility**: Fully compatible with Wan official toolchain

### Wan2.1 Official Models
gushiqiao's avatar
gushiqiao committed
28

gushiqiao's avatar
gushiqiao committed
29
30
31
#### Directory Structure

Using [Wan2.1-I2V-14B-720P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) as an example:
gushiqiao's avatar
gushiqiao committed
32
33

```
gushiqiao's avatar
gushiqiao committed
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
Wan2.1-I2V-14B-720P/
├── diffusion_pytorch_model-00001-of-00007.safetensors   # DIT model shard 1
├── diffusion_pytorch_model-00002-of-00007.safetensors   # DIT model shard 2
├── diffusion_pytorch_model-00003-of-00007.safetensors   # DIT model shard 3
├── diffusion_pytorch_model-00004-of-00007.safetensors   # DIT model shard 4
├── diffusion_pytorch_model-00005-of-00007.safetensors   # DIT model shard 5
├── diffusion_pytorch_model-00006-of-00007.safetensors   # DIT model shard 6
├── diffusion_pytorch_model-00007-of-00007.safetensors   # DIT model shard 7
├── diffusion_pytorch_model.safetensors.index.json       # Shard index file
├── models_t5_umt5-xxl-enc-bf16.pth                      # T5 text encoder
├── models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth  # CLIP encoder
├── Wan2.1_VAE.pth                                       # VAE encoder/decoder
├── config.json                                          # Model configuration
├── xlm-roberta-large/                                   # CLIP tokenizer
├── google/                                              # T5 tokenizer
├── assets/
└── examples/
gushiqiao's avatar
gushiqiao committed
51
52
```

gushiqiao's avatar
gushiqiao committed
53
#### Usage
gushiqiao's avatar
gushiqiao committed
54
55

```bash
gushiqiao's avatar
gushiqiao committed
56
57
58
# Download model
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P \
    --local-dir ./models/Wan2.1-I2V-14B-720P
gushiqiao's avatar
gushiqiao committed
59

gushiqiao's avatar
gushiqiao committed
60
61
62
# Configure launch script
model_path=./models/Wan2.1-I2V-14B-720P
lightx2v_path=/path/to/LightX2V
gushiqiao's avatar
gushiqiao committed
63

gushiqiao's avatar
gushiqiao committed
64
65
66
67
68
69
# Run inference
cd LightX2V/scripts
bash wan/run_wan_i2v.sh
```

### Wan2.2 Official Models
gushiqiao's avatar
gushiqiao committed
70

gushiqiao's avatar
gushiqiao committed
71
#### Directory Structure
gushiqiao's avatar
gushiqiao committed
72

gushiqiao's avatar
gushiqiao committed
73
Using [Wan2.2-I2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B) as an example:
gushiqiao's avatar
gushiqiao committed
74

gushiqiao's avatar
gushiqiao committed
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
```
Wan2.2-I2V-A14B/
├── high_noise_model/                                    # High-noise model directory
│   ├── diffusion_pytorch_model-00001-of-00009.safetensors
│   ├── diffusion_pytorch_model-00002-of-00009.safetensors
│   ├── ...
│   ├── diffusion_pytorch_model-00009-of-00009.safetensors
│   └── diffusion_pytorch_model.safetensors.index.json
├── low_noise_model/                                     # Low-noise model directory
│   ├── diffusion_pytorch_model-00001-of-00009.safetensors
│   ├── diffusion_pytorch_model-00002-of-00009.safetensors
│   ├── ...
│   ├── diffusion_pytorch_model-00009-of-00009.safetensors
│   └── diffusion_pytorch_model.safetensors.index.json
├── models_t5_umt5-xxl-enc-bf16.pth                      # T5 text encoder
├── Wan2.1_VAE.pth                                       # VAE encoder/decoder
├── configuration.json                                   # Model configuration
├── google/                                              # T5 tokenizer
├── assets/                                              # Example assets (optional)
└── examples/                                            # Example files (optional)
```
gushiqiao's avatar
gushiqiao committed
96

gushiqiao's avatar
gushiqiao committed
97
#### Usage
gushiqiao's avatar
gushiqiao committed
98
99

```bash
gushiqiao's avatar
gushiqiao committed
100
101
102
# Download model
huggingface-cli download Wan-AI/Wan2.2-I2V-A14B \
    --local-dir ./models/Wan2.2-I2V-A14B
gushiqiao's avatar
gushiqiao committed
103

gushiqiao's avatar
gushiqiao committed
104
105
106
# Configure launch script
model_path=./models/Wan2.2-I2V-A14B
lightx2v_path=/path/to/LightX2V
gushiqiao's avatar
gushiqiao committed
107

gushiqiao's avatar
gushiqiao committed
108
109
110
# Run inference
cd LightX2V/scripts
bash wan22/run_wan22_moe_i2v.sh
gushiqiao's avatar
gushiqiao committed
111
```
gushiqiao's avatar
gushiqiao committed
112

gushiqiao's avatar
gushiqiao committed
113
### Available Model List
gushiqiao's avatar
gushiqiao committed
114

gushiqiao's avatar
gushiqiao committed
115
#### Wan2.1 Official Model List
gushiqiao's avatar
gushiqiao committed
116

gushiqiao's avatar
gushiqiao committed
117
118
119
120
121
122
123
124
125
| Model Name | Download Link |
|---------|----------|
| Wan2.1-I2V-14B-720P | [Link](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) |
| Wan2.1-I2V-14B-480P | [Link](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) |
| Wan2.1-T2V-14B | [Link](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) |
| Wan2.1-T2V-1.3B | [Link](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) |
| Wan2.1-FLF2V-14B-720P | [Link](https://huggingface.co/Wan-AI/Wan2.1-FLF2V-14B-720P) |
| Wan2.1-VACE-14B | [Link](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B) |
| Wan2.1-VACE-1.3B | [Link](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B) |
gushiqiao's avatar
gushiqiao committed
126

gushiqiao's avatar
gushiqiao committed
127
#### Wan2.2 Official Model List
gushiqiao's avatar
gushiqiao committed
128

gushiqiao's avatar
gushiqiao committed
129
130
131
132
133
134
| Model Name | Download Link |
|---------|----------|
| Wan2.2-I2V-A14B | [Link](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B) |
| Wan2.2-T2V-A14B | [Link](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B) |
| Wan2.2-TI2V-5B | [Link](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) |
| Wan2.2-Animate-14B | [Link](https://huggingface.co/Wan-AI/Wan2.2-Animate-14B) |
gushiqiao's avatar
gushiqiao committed
135

gushiqiao's avatar
gushiqiao committed
136
### Usage Tips
gushiqiao's avatar
gushiqiao committed
137

gushiqiao's avatar
gushiqiao committed
138
139
140
141
142
143
144
> 💡 **Quantized Model Usage**: To use quantized models, refer to the [Model Conversion Script](https://github.com/ModelTC/LightX2V/blob/main/tools/convert/readme_zh.md) for conversion, or directly use pre-converted quantized models in Format 2 below
>
> 💡 **Memory Optimization**: For devices with RTX 4090 24GB or smaller memory, it's recommended to combine quantization techniques with CPU offload features:
> - Quantization Configuration: Refer to [Quantization Documentation](../method_tutorials/quantization.md)
> - CPU Offload: Refer to [Parameter Offload Documentation](../method_tutorials/offload.md)
> - Wan2.1 Configuration: Refer to [offload config files](https://github.com/ModelTC/LightX2V/tree/main/configs/offload)
> - Wan2.2 Configuration: Refer to [wan22 config files](https://github.com/ModelTC/LightX2V/tree/main/configs/wan22) with `4090` suffix
gushiqiao's avatar
gushiqiao committed
145

gushiqiao's avatar
gushiqiao committed
146
---
gushiqiao's avatar
gushiqiao committed
147

gushiqiao's avatar
gushiqiao committed
148
## 🗂️ Format 2: LightX2V Single-File Models (Recommended)
gushiqiao's avatar
gushiqiao committed
149

gushiqiao's avatar
gushiqiao committed
150
151
152
### Model Repositories
- [Wan2.1-LightX2V](https://huggingface.co/lightx2v/Wan2.1-Distill-Models)
- [Wan2.2-LightX2V](https://huggingface.co/lightx2v/Wan2.2-Distill-Models)
gushiqiao's avatar
gushiqiao committed
153

gushiqiao's avatar
gushiqiao committed
154
155
156
157
158
### Model Features
- **Single-File Management**: Single safetensors file, easy to manage and deploy
- **Multi-Precision Support**: Provides original precision, FP8, INT8, and other precision versions
- **Distillation Acceleration**: Supports 4-step fast inference
- **Tool Compatibility**: Compatible with ComfyUI and other tools
gushiqiao's avatar
gushiqiao committed
159

gushiqiao's avatar
gushiqiao committed
160
161
162
163
164
**Examples**:
- `wan2.1_i2v_720p_lightx2v_4step.safetensors` - 720P I2V original precision
- `wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors` - 720P I2V FP8 quantization
- `wan2.1_i2v_480p_int8_lightx2v_4step.safetensors` - 480P I2V INT8 quantization
- ...
gushiqiao's avatar
gushiqiao committed
165

gushiqiao's avatar
gushiqiao committed
166
### Wan2.1 Single-File Models
gushiqiao's avatar
gushiqiao committed
167

gushiqiao's avatar
gushiqiao committed
168
169
170
#### Scenario A: Download Single Model File

**Step 1: Select and Download Model**
gushiqiao's avatar
gushiqiao committed
171
172

```bash
gushiqiao's avatar
gushiqiao committed
173
174
# Create model directory
mkdir -p ./models/wan2.1_i2v_720p
gushiqiao's avatar
gushiqiao committed
175

gushiqiao's avatar
gushiqiao committed
176
177
178
179
180
# Download 720P I2V FP8 quantized model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models/wan2.1_i2v_720p \
    --include "wan2.1_i2v_720p_lightx2v_4step.safetensors"
```
gushiqiao's avatar
gushiqiao committed
181

gushiqiao's avatar
gushiqiao committed
182
**Step 2: Configure Launch Script**
gushiqiao's avatar
gushiqiao committed
183
184

```bash
gushiqiao's avatar
gushiqiao committed
185
186
187
188
189
190
191
# Set in launch script (point to directory containing model file)
model_path=./models/wan2.1_i2v_720p
lightx2v_path=/path/to/LightX2V

# Run script
cd LightX2V/scripts
bash wan/run_wan_i2v_distill_4step_cfg.sh
gushiqiao's avatar
gushiqiao committed
192
193
```

gushiqiao's avatar
gushiqiao committed
194
> 💡 **Tip**: When there's only one model file in the directory, LightX2V will automatically load it.
gushiqiao's avatar
gushiqiao committed
195

gushiqiao's avatar
gushiqiao committed
196
#### Scenario B: Download Multiple Model Files
gushiqiao's avatar
gushiqiao committed
197

gushiqiao's avatar
gushiqiao committed
198
When you download multiple models with different precisions to the same directory, you need to explicitly specify which model to use in the configuration file.
gushiqiao's avatar
gushiqiao committed
199

gushiqiao's avatar
gushiqiao committed
200
**Step 1: Download Multiple Models**
gushiqiao's avatar
gushiqiao committed
201
202

```bash
gushiqiao's avatar
gushiqiao committed
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
# Create model directory
mkdir -p ./models/wan2.1_i2v_720p_multi

# Download original precision model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models/wan2.1_i2v_720p_multi \
    --include "wan2.1_i2v_720p_lightx2v_4step.safetensors"

# Download FP8 quantized model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models/wan2.1_i2v_720p_multi \
    --include "wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors"

# Download INT8 quantized model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models/wan2.1_i2v_720p_multi \
    --include "wan2.1_i2v_720p_int8_lightx2v_4step.safetensors"
gushiqiao's avatar
gushiqiao committed
220
221
```

gushiqiao's avatar
gushiqiao committed
222
**Directory Structure**:
gushiqiao's avatar
gushiqiao committed
223

gushiqiao's avatar
gushiqiao committed
224
225
226
227
228
229
230
```
wan2.1_i2v_720p_multi/
├── wan2.1_i2v_720p_lightx2v_4step.safetensors                    # Original precision
├── wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors   # FP8 quantization
└── wan2.1_i2v_720p_int8_lightx2v_4step.safetensors              # INT8 quantization
└── t5/clip/vae/config.json/xlm-roberta-large/google and other components  # Manually organized
```
gushiqiao's avatar
gushiqiao committed
231

gushiqiao's avatar
gushiqiao committed
232
**Step 2: Specify Model in Configuration File**
gushiqiao's avatar
gushiqiao committed
233

gushiqiao's avatar
gushiqiao committed
234
Edit configuration file (e.g., `configs/distill/wan_i2v_distill_4step_cfg.json`):
gushiqiao's avatar
gushiqiao committed
235

gushiqiao's avatar
gushiqiao committed
236
237
238
239
```json
{
    // Use original precision model
    "dit_original_ckpt": "./models/wan2.1_i2v_720p_multi/wan2.1_i2v_720p_lightx2v_4step.safetensors",
gushiqiao's avatar
gushiqiao committed
240

gushiqiao's avatar
gushiqiao committed
241
242
243
244
    // Or use FP8 quantized model
    // "dit_quantized_ckpt": "./models/wan2.1_i2v_720p_multi/wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors",
    // "dit_quantized": true,
    // "dit_quant_scheme": "fp8-vllm",
gushiqiao's avatar
gushiqiao committed
245

gushiqiao's avatar
gushiqiao committed
246
247
248
249
    // Or use INT8 quantized model
    // "dit_quantized_ckpt": "./models/wan2.1_i2v_720p_multi/wan2.1_i2v_720p_int8_lightx2v_4step.safetensors",
    // "dit_quantized": true,
    // "dit_quant_scheme": "int8-vllm",
gushiqiao's avatar
gushiqiao committed
250

gushiqiao's avatar
gushiqiao committed
251
    // Other configurations...
gushiqiao's avatar
gushiqiao committed
252
253
}
```
gushiqiao's avatar
gushiqiao committed
254
### Usage Tips
gushiqiao's avatar
gushiqiao committed
255

gushiqiao's avatar
gushiqiao committed
256
257
258
> 💡 **Configuration Parameter Description**:
> - **dit_original_ckpt**: Used to specify the path to original precision models (BF16/FP32/FP16)
> - **dit_quantized_ckpt**: Used to specify the path to quantized models (FP8/INT8), must be used with `dit_quantized` and `dit_quant_scheme` parameters
gushiqiao's avatar
gushiqiao committed
259

gushiqiao's avatar
gushiqiao committed
260
**Step 3: Start Inference**
gushiqiao's avatar
gushiqiao committed
261

gushiqiao's avatar
gushiqiao committed
262
263
264
265
266
267
268
269
270
271
```bash
cd LightX2V/scripts
bash wan/run_wan_i2v_distill_4step_cfg.sh
```

### Wan2.2 Single-File Models

#### Directory Structure Requirements

When using Wan2.2 single-file models, you need to manually create a specific directory structure:
gushiqiao's avatar
gushiqiao committed
272

gushiqiao's avatar
gushiqiao committed
273
274
275
276
277
278
279
```
wan2.2_models/
├── high_noise_model/                                    # High-noise model directory (required)
│   └── wan2.2_i2v_A14b_high_noise_lightx2v_4step.safetensors  # High-noise model file
└── low_noise_model/                                     # Low-noise model directory (required)
│   └── wan2.2_i2v_A14b_low_noise_lightx2v_4step.safetensors  # Low-noise model file
└── t5/vae/config.json/xlm-roberta-large/google and other components  # Manually organized
gushiqiao's avatar
gushiqiao committed
280
281
```

gushiqiao's avatar
gushiqiao committed
282
#### Scenario A: Only One Model File Per Directory
gushiqiao's avatar
gushiqiao committed
283
284

```bash
gushiqiao's avatar
gushiqiao committed
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
# Create required subdirectories
mkdir -p ./models/wan2.2_models/high_noise_model
mkdir -p ./models/wan2.2_models/low_noise_model

# Download high-noise model to corresponding directory
huggingface-cli download lightx2v/Wan2.2-Distill-Models \
    --local-dir ./models/wan2.2_models/high_noise_model \
    --include "wan2.2_i2v_A14b_high_noise_scaled_fp8_e4m3_lightx2v_4step.safetensors"

# Download low-noise model to corresponding directory
huggingface-cli download lightx2v/Wan2.2-Distill-Models \
    --local-dir ./models/wan2.2_models/low_noise_model \
    --include "wan2.2_i2v_A14b_low_noise_scaled_fp8_e4m3_lightx2v_4step.safetensors"

# Configure launch script (point to parent directory)
model_path=./models/wan2.2_models
lightx2v_path=/path/to/LightX2V

# Run script
gushiqiao's avatar
gushiqiao committed
304
cd LightX2V/scripts
gushiqiao's avatar
gushiqiao committed
305
bash wan22/run_wan22_moe_i2v_distill.sh
gushiqiao's avatar
gushiqiao committed
306
307
```

gushiqiao's avatar
gushiqiao committed
308
309
310
> 💡 **Tip**: When there's only one model file in each subdirectory, LightX2V will automatically load it.

#### Scenario B: Multiple Model Files Per Directory
gushiqiao's avatar
gushiqiao committed
311

gushiqiao's avatar
gushiqiao committed
312
When you place multiple models with different precisions in both `high_noise_model/` and `low_noise_model/` directories, you need to explicitly specify them in the configuration file.
gushiqiao's avatar
gushiqiao committed
313
314

```bash
gushiqiao's avatar
gushiqiao committed
315
316
317
318
319
320
321
322
323
324
325
326
327
# Create directories
mkdir -p ./models/wan2.2_models_multi/high_noise_model
mkdir -p ./models/wan2.2_models_multi/low_noise_model

# Download multiple versions of high-noise model
huggingface-cli download lightx2v/Wan2.2-Distill-Models \
    --local-dir ./models/wan2.2_models_multi/high_noise_model \
    --include "wan2.2_i2v_A14b_high_noise_*.safetensors"

# Download multiple versions of low-noise model
huggingface-cli download lightx2v/Wan2.2-Distill-Models \
    --local-dir ./models/wan2.2_models_multi/low_noise_model \
    --include "wan2.2_i2v_A14b_low_noise_*.safetensors"
gushiqiao's avatar
gushiqiao committed
328
329
```

gushiqiao's avatar
gushiqiao committed
330
**Directory Structure**:
gushiqiao's avatar
gushiqiao committed
331

gushiqiao's avatar
gushiqiao committed
332
333
334
335
336
337
338
339
340
341
342
```
wan2.2_models_multi/
├── high_noise_model/
│   ├── wan2.2_i2v_A14b_high_noise_lightx2v_4step.safetensors        # Original precision
│   ├── wan2.2_i2v_A14b_high_noise_fp8_e4m3_lightx2v_4step.safetensors    # FP8 quantization
│   └── wan2.2_i2v_A14b_high_noise_int8_lightx2v_4step.safetensors   # INT8 quantization
└── low_noise_model/
    ├── wan2.2_i2v_A14b_low_noise_lightx2v_4step.safetensors         # Original precision
    ├── wan2.2_i2v_A14b_low_noise_fp8_e4m3_lightx2v_4step.safetensors     # FP8 quantization
    └── wan2.2_i2v_A14b_low_noise_int8_lightx2v_4step.safetensors    # INT8 quantization
```
gushiqiao's avatar
gushiqiao committed
343

gushiqiao's avatar
gushiqiao committed
344
**Configuration File Settings**:
gushiqiao's avatar
gushiqiao committed
345
346
347

```json
{
gushiqiao's avatar
gushiqiao committed
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
    // Use original precision model
    "high_noise_original_ckpt": "./models/wan2.2_models_multi/high_noise_model/wan2.2_i2v_A14b_high_noise_lightx2v_4step.safetensors",
    "low_noise_original_ckpt": "./models/wan2.2_models_multi/low_noise_model/wan2.2_i2v_A14b_low_noise_lightx2v_4step.safetensors",

    // Or use FP8 quantized model
    // "high_noise_quantized_ckpt": "./models/wan2.2_models_multi/high_noise_model/wan2.2_i2v_A14b_high_noise_fp8_e4m3_lightx2v_4step.safetensors",
    // "low_noise_quantized_ckpt": "./models/wan2.2_models_multi/low_noise_model/wan2.2_i2v_A14b_low_noise_fp8_e4m3_lightx2v_4step.safetensors",
    // "dit_quantized": true,
    // "dit_quant_scheme": "fp8-vllm"

    // Or use INT8 quantized model
    // "high_noise_quantized_ckpt": "./models/wan2.2_models_multi/high_noise_model/wan2.2_i2v_A14b_high_noise_int8_lightx2v_4step.safetensors",
    // "low_noise_quantized_ckpt": "./models/wan2.2_models_multi/low_noise_model/wan2.2_i2v_A14b_low_noise_int8_lightx2v_4step.safetensors",
    // "dit_quantized": true,
    // "dit_quant_scheme": "int8-vllm"
gushiqiao's avatar
gushiqiao committed
363
}
gushiqiao's avatar
gushiqiao committed
364
365
```

gushiqiao's avatar
gushiqiao committed
366
### Usage Tips
gushiqiao's avatar
gushiqiao committed
367

gushiqiao's avatar
gushiqiao committed
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
> 💡 **Configuration Parameter Description**:
> - **high_noise_original_ckpt** / **low_noise_original_ckpt**: Used to specify the path to original precision models (BF16/FP32/FP16)
> - **high_noise_quantized_ckpt** / **low_noise_quantized_ckpt**: Used to specify the path to quantized models (FP8/INT8), must be used with `dit_quantized` and `dit_quant_scheme` parameters


### Available Model List

#### Wan2.1 Single-File Model List

**Image-to-Video Models (I2V)**

| Filename | Precision | Description |
|--------|------|------|
| `wan2.1_i2v_480p_lightx2v_4step.safetensors` | BF16 | 4-step model original precision |
| `wan2.1_i2v_480p_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | 4-step model FP8 quantization |
| `wan2.1_i2v_480p_int8_lightx2v_4step.safetensors` | INT8 | 4-step model INT8 quantization |
| `wan2.1_i2v_480p_scaled_fp8_e4m3_lightx2v_4step_comfyui.safetensors` | FP8 | 4-step model ComfyUI format |
| `wan2.1_i2v_720p_lightx2v_4step.safetensors` | BF16 | 4-step model original precision |
| `wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | 4-step model FP8 quantization |
| `wan2.1_i2v_720p_int8_lightx2v_4step.safetensors` | INT8 | 4-step model INT8 quantization |
| `wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step_comfyui.safetensors` | FP8 | 4-step model ComfyUI format |

**Text-to-Video Models (T2V)**

| Filename | Precision | Description |
|--------|------|------|
| `wan2.1_t2v_14b_lightx2v_4step.safetensors` | BF16 | 4-step model original precision |
| `wan2.1_t2v_14b_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | 4-step model FP8 quantization |
| `wan2.1_t2v_14b_int8_lightx2v_4step.safetensors` | INT8 | 4-step model INT8 quantization |
| `wan2.1_t2v_14b_scaled_fp8_e4m3_lightx2v_4step_comfyui.safetensors` | FP8 | 4-step model ComfyUI format |

#### Wan2.2 Single-File Model List

**Image-to-Video Models (I2V) - A14B Series**

| Filename | Precision | Description |
|--------|------|------|
| `wan2.2_i2v_A14b_high_noise_lightx2v_4step.safetensors` | BF16 | High-noise model - 4-step original precision |
| `wan2.2_i2v_A14b_high_noise_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | High-noise model - 4-step FP8 quantization |
| `wan2.2_i2v_A14b_high_noise_int8_lightx2v_4step.safetensors` | INT8 | High-noise model - 4-step INT8 quantization |
| `wan2.2_i2v_A14b_low_noise_lightx2v_4step.safetensors` | BF16 | Low-noise model - 4-step original precision |
| `wan2.2_i2v_A14b_low_noise_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | Low-noise model - 4-step FP8 quantization |
| `wan2.2_i2v_A14b_low_noise_int8_lightx2v_4step.safetensors` | INT8 | Low-noise model - 4-step INT8 quantization |

> 💡 **Usage Tips**:
> - Wan2.2 models use a dual-noise architecture, requiring both high-noise and low-noise models to be downloaded
> - Refer to the "Wan2.2 Single-File Models" section above for detailed directory organization

---

## 🗂️ Format 3: LightX2V LoRA Models

LoRA (Low-Rank Adaptation) models provide a lightweight model fine-tuning solution that enables customization for specific effects without modifying the base model.

### Model Repositories

- **Wan2.1 LoRA Models**: [lightx2v/Wan2.1-Distill-Loras](https://huggingface.co/lightx2v/Wan2.1-Distill-Loras)
- **Wan2.2 LoRA Models**: [lightx2v/Wan2.2-Distill-Loras](https://huggingface.co/lightx2v/Wan2.2-Distill-Loras)

### Usage Methods

#### Method 1: Offline Merging

Merge LoRA weights offline into the base model to generate a new complete model file.

**Steps**:

Refer to the [Model Conversion Documentation](https://github.com/ModelTC/lightx2v/tree/main/tools/convert/readme_zh.md) for offline merging.

**Advantages**:
- ✅ No need to load LoRA during inference
- ✅ Better performance

**Disadvantages**:
- ❌ Requires additional storage space
- ❌ Switching different LoRAs requires re-merging

#### Method 2: Online Loading

Dynamically load LoRA weights during inference without modifying the base model.

**LoRA Application Principle**:

```python
# LoRA weight application formula
# W' = W + (alpha/rank) * B @ A
# Where: B = up_proj (out_features, rank)
#        A = down_proj (rank, in_features)

if weights_dict["alpha"] is not None:
    lora_alpha = weights_dict["alpha"] / lora_down.shape[0]
elif alpha is not None:
    lora_alpha = alpha / lora_down.shape[0]
else:
    lora_alpha = 1.0
gushiqiao's avatar
gushiqiao committed
463
```
gushiqiao's avatar
gushiqiao committed
464

gushiqiao's avatar
gushiqiao committed
465
466
467
**Configuration Method**:

**Wan2.1 LoRA Configuration**:
gushiqiao's avatar
gushiqiao committed
468

gushiqiao's avatar
gushiqiao committed
469
470
```json
{
gushiqiao's avatar
gushiqiao committed
471
472
473
474
475
476
477
  "lora_configs": [
    {
      "path": "wan2.1_i2v_lora_rank64_lightx2v_4step.safetensors",
      "strength": 1.0,
      "alpha": null
    }
  ]
gushiqiao's avatar
gushiqiao committed
478
479
}
```
gushiqiao's avatar
gushiqiao committed
480

gushiqiao's avatar
gushiqiao committed
481
482
483
**Wan2.2 LoRA Configuration**:

Since Wan2.2 uses a dual-model architecture (high-noise/low-noise), LoRA needs to be configured separately for both models:
gushiqiao's avatar
gushiqiao committed
484

gushiqiao's avatar
gushiqiao committed
485
486
```json
{
gushiqiao's avatar
gushiqiao committed
487
488
489
490
491
492
493
494
495
496
497
498
499
500
  "lora_configs": [
    {
      "name": "low_noise_model",
      "path": "wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step.safetensors",
      "strength": 1.0,
      "alpha": null
    },
    {
      "name": "high_noise_model",
      "path": "wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step.safetensors",
      "strength": 1.0,
      "alpha": null
    }
  ]
gushiqiao's avatar
gushiqiao committed
501
502
}
```
gushiqiao's avatar
gushiqiao committed
503

gushiqiao's avatar
gushiqiao committed
504
**Parameter Description**:
gushiqiao's avatar
gushiqiao committed
505

gushiqiao's avatar
gushiqiao committed
506
507
508
509
510
511
| Parameter | Description | Default |
|------|------|--------|
| `path` | LoRA model file path | Required |
| `strength` | LoRA strength coefficient, range [0.0, 1.0] | 1.0 |
| `alpha` | LoRA scaling factor, uses model's built-in value when `null`, defaults to 1 if no built-in value | null |
| `name` | (Wan2.2 only) Specifies which model to apply to | Required |
gushiqiao's avatar
gushiqiao committed
512

gushiqiao's avatar
gushiqiao committed
513
514
515
516
**Advantages**:
- ✅ Flexible switching between different LoRAs
- ✅ Saves storage space
- ✅ Can dynamically adjust LoRA strength
gushiqiao's avatar
gushiqiao committed
517

gushiqiao's avatar
gushiqiao committed
518
519
520
**Disadvantages**:
- ❌ Additional loading time during inference
- ❌ Slightly increases memory usage
gushiqiao's avatar
gushiqiao committed
521

gushiqiao's avatar
gushiqiao committed
522
---
gushiqiao's avatar
gushiqiao committed
523

gushiqiao's avatar
gushiqiao committed
524
## 📚 Related Resources
gushiqiao's avatar
gushiqiao committed
525

gushiqiao's avatar
gushiqiao committed
526
527
528
529
### Official Repositories
- [LightX2V GitHub](https://github.com/ModelTC/LightX2V)
- [LightX2V Single-File Model Repository](https://huggingface.co/lightx2v/Wan2.1-Distill-Models)
- [Wan-AI Official Model Repository](https://huggingface.co/Wan-AI)
gushiqiao's avatar
gushiqiao committed
530

gushiqiao's avatar
gushiqiao committed
531
### Model Download Links
gushiqiao's avatar
gushiqiao committed
532

gushiqiao's avatar
gushiqiao committed
533
534
**Wan2.1 Series**
- [Wan2.1 Collection](https://huggingface.co/collections/Wan-AI/wan21-68ac4ba85372ae5a8e282a1b)
gushiqiao's avatar
gushiqiao committed
535

gushiqiao's avatar
gushiqiao committed
536
537
**Wan2.2 Series**
- [Wan2.2 Collection](https://huggingface.co/collections/Wan-AI/wan22-68ac4ae80a8b477e79636fc8)
gushiqiao's avatar
gushiqiao committed
538

gushiqiao's avatar
gushiqiao committed
539
540
541
**LightX2V Single-File Models**
- [Wan2.1-Distill-Models](https://huggingface.co/lightx2v/Wan2.1-Distill-Models)
- [Wan2.2-Distill-Models](https://huggingface.co/lightx2v/Wan2.2-Distill-Models)
gushiqiao's avatar
gushiqiao committed
542

gushiqiao's avatar
gushiqiao committed
543
544
545
### Documentation Links
- [Quantization Documentation](../method_tutorials/quantization.md)
- [Parameter Offload Documentation](../method_tutorials/offload.md)
gushiqiao's avatar
gushiqiao committed
546
- [Configuration File Examples](https://github.com/ModelTC/LightX2V/tree/main/configs)
gushiqiao's avatar
gushiqiao committed
547
548
549

---

gushiqiao's avatar
gushiqiao committed
550
551
552
553
554
555
556
557
558
Through this document, you should be able to:

✅ Understand all model formats supported by LightX2V
✅ Select appropriate models and precisions based on your needs
✅ Correctly download and organize model files
✅ Configure launch parameters and successfully run inference
✅ Resolve common model loading issues

If you have other questions, feel free to ask in [GitHub Issues](https://github.com/ModelTC/LightX2V/issues).