model_structure.md 22.4 KB
Newer Older
gushiqiao's avatar
gushiqiao committed
1
# Model Format and Loading Guide
gushiqiao's avatar
gushiqiao committed
2
3
4

## 📖 Overview

gushiqiao's avatar
gushiqiao committed
5
LightX2V is a flexible video generation inference framework that supports multiple model sources and formats, providing users with rich options:
gushiqiao's avatar
gushiqiao committed
6

gushiqiao's avatar
gushiqiao committed
7
8
9
-**Wan Official Models**: Directly compatible with officially released complete models from Wan2.1 and Wan2.2
-**Single-File Models**: Supports single-file format models released by LightX2V (including quantized versions)
-**LoRA Models**: Supports loading distilled LoRAs released by LightX2V
gushiqiao's avatar
gushiqiao committed
10

gushiqiao's avatar
gushiqiao committed
11
This document provides detailed instructions on how to use various model formats, configuration parameters, and best practices.
gushiqiao's avatar
gushiqiao committed
12

gushiqiao's avatar
gushiqiao committed
13
---
gushiqiao's avatar
gushiqiao committed
14

gushiqiao's avatar
gushiqiao committed
15
## 🗂️ Format 1: Wan Official Models
gushiqiao's avatar
gushiqiao committed
16

gushiqiao's avatar
gushiqiao committed
17
18
19
### Model Repositories
- [Wan2.1 Collection](https://huggingface.co/collections/Wan-AI/wan21-68ac4ba85372ae5a8e282a1b)
- [Wan2.2 Collection](https://huggingface.co/collections/Wan-AI/wan22-68ac4ae80a8b477e79636fc8)
gushiqiao's avatar
gushiqiao committed
20

gushiqiao's avatar
gushiqiao committed
21
22
23
24
25
26
27
### Model Features
- **Official Guarantee**: Complete models officially released by Wan-AI with highest quality
- **Complete Components**: Includes all necessary components (DIT, T5, CLIP, VAE)
- **Original Precision**: Uses BF16/FP32 precision with no quantization loss
- **Strong Compatibility**: Fully compatible with Wan official toolchain

### Wan2.1 Official Models
gushiqiao's avatar
gushiqiao committed
28

gushiqiao's avatar
gushiqiao committed
29
30
31
#### Directory Structure

Using [Wan2.1-I2V-14B-720P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) as an example:
gushiqiao's avatar
gushiqiao committed
32
33

```
gushiqiao's avatar
gushiqiao committed
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
Wan2.1-I2V-14B-720P/
├── diffusion_pytorch_model-00001-of-00007.safetensors   # DIT model shard 1
├── diffusion_pytorch_model-00002-of-00007.safetensors   # DIT model shard 2
├── diffusion_pytorch_model-00003-of-00007.safetensors   # DIT model shard 3
├── diffusion_pytorch_model-00004-of-00007.safetensors   # DIT model shard 4
├── diffusion_pytorch_model-00005-of-00007.safetensors   # DIT model shard 5
├── diffusion_pytorch_model-00006-of-00007.safetensors   # DIT model shard 6
├── diffusion_pytorch_model-00007-of-00007.safetensors   # DIT model shard 7
├── diffusion_pytorch_model.safetensors.index.json       # Shard index file
├── models_t5_umt5-xxl-enc-bf16.pth                      # T5 text encoder
├── models_clip_open-clip-xlm-roberta-large-vit-huge-14.pth  # CLIP encoder
├── Wan2.1_VAE.pth                                       # VAE encoder/decoder
├── config.json                                          # Model configuration
├── xlm-roberta-large/                                   # CLIP tokenizer
├── google/                                              # T5 tokenizer
├── assets/
└── examples/
gushiqiao's avatar
gushiqiao committed
51
52
```

gushiqiao's avatar
gushiqiao committed
53
#### Usage
gushiqiao's avatar
gushiqiao committed
54
55

```bash
gushiqiao's avatar
gushiqiao committed
56
57
58
# Download model
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P \
    --local-dir ./models/Wan2.1-I2V-14B-720P
gushiqiao's avatar
gushiqiao committed
59

gushiqiao's avatar
gushiqiao committed
60
61
62
# Configure launch script
model_path=./models/Wan2.1-I2V-14B-720P
lightx2v_path=/path/to/LightX2V
gushiqiao's avatar
gushiqiao committed
63

gushiqiao's avatar
gushiqiao committed
64
65
66
67
68
69
# Run inference
cd LightX2V/scripts
bash wan/run_wan_i2v.sh
```

### Wan2.2 Official Models
gushiqiao's avatar
gushiqiao committed
70

gushiqiao's avatar
gushiqiao committed
71
#### Directory Structure
gushiqiao's avatar
gushiqiao committed
72

gushiqiao's avatar
gushiqiao committed
73
Using [Wan2.2-I2V-A14B](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B) as an example:
gushiqiao's avatar
gushiqiao committed
74

gushiqiao's avatar
gushiqiao committed
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
```
Wan2.2-I2V-A14B/
├── high_noise_model/                                    # High-noise model directory
│   ├── diffusion_pytorch_model-00001-of-00009.safetensors
│   ├── diffusion_pytorch_model-00002-of-00009.safetensors
│   ├── ...
│   ├── diffusion_pytorch_model-00009-of-00009.safetensors
│   └── diffusion_pytorch_model.safetensors.index.json
├── low_noise_model/                                     # Low-noise model directory
│   ├── diffusion_pytorch_model-00001-of-00009.safetensors
│   ├── diffusion_pytorch_model-00002-of-00009.safetensors
│   ├── ...
│   ├── diffusion_pytorch_model-00009-of-00009.safetensors
│   └── diffusion_pytorch_model.safetensors.index.json
├── models_t5_umt5-xxl-enc-bf16.pth                      # T5 text encoder
├── Wan2.1_VAE.pth                                       # VAE encoder/decoder
├── configuration.json                                   # Model configuration
├── google/                                              # T5 tokenizer
├── assets/                                              # Example assets (optional)
└── examples/                                            # Example files (optional)
```
gushiqiao's avatar
gushiqiao committed
96

gushiqiao's avatar
gushiqiao committed
97
#### Usage
gushiqiao's avatar
gushiqiao committed
98
99

```bash
gushiqiao's avatar
gushiqiao committed
100
101
102
# Download model
huggingface-cli download Wan-AI/Wan2.2-I2V-A14B \
    --local-dir ./models/Wan2.2-I2V-A14B
gushiqiao's avatar
gushiqiao committed
103

gushiqiao's avatar
gushiqiao committed
104
105
106
# Configure launch script
model_path=./models/Wan2.2-I2V-A14B
lightx2v_path=/path/to/LightX2V
gushiqiao's avatar
gushiqiao committed
107

gushiqiao's avatar
gushiqiao committed
108
109
110
# Run inference
cd LightX2V/scripts
bash wan22/run_wan22_moe_i2v.sh
gushiqiao's avatar
gushiqiao committed
111
```
gushiqiao's avatar
gushiqiao committed
112

gushiqiao's avatar
gushiqiao committed
113
### Available Model List
gushiqiao's avatar
gushiqiao committed
114

gushiqiao's avatar
gushiqiao committed
115
#### Wan2.1 Official Model List
gushiqiao's avatar
gushiqiao committed
116

gushiqiao's avatar
gushiqiao committed
117
118
119
120
121
122
123
124
125
| Model Name | Download Link |
|---------|----------|
| Wan2.1-I2V-14B-720P | [Link](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) |
| Wan2.1-I2V-14B-480P | [Link](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) |
| Wan2.1-T2V-14B | [Link](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) |
| Wan2.1-T2V-1.3B | [Link](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B) |
| Wan2.1-FLF2V-14B-720P | [Link](https://huggingface.co/Wan-AI/Wan2.1-FLF2V-14B-720P) |
| Wan2.1-VACE-14B | [Link](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B) |
| Wan2.1-VACE-1.3B | [Link](https://huggingface.co/Wan-AI/Wan2.1-VACE-1.3B) |
gushiqiao's avatar
gushiqiao committed
126

gushiqiao's avatar
gushiqiao committed
127
#### Wan2.2 Official Model List
gushiqiao's avatar
gushiqiao committed
128

gushiqiao's avatar
gushiqiao committed
129
130
131
132
133
134
| Model Name | Download Link |
|---------|----------|
| Wan2.2-I2V-A14B | [Link](https://huggingface.co/Wan-AI/Wan2.2-I2V-A14B) |
| Wan2.2-T2V-A14B | [Link](https://huggingface.co/Wan-AI/Wan2.2-T2V-A14B) |
| Wan2.2-TI2V-5B | [Link](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) |
| Wan2.2-Animate-14B | [Link](https://huggingface.co/Wan-AI/Wan2.2-Animate-14B) |
gushiqiao's avatar
gushiqiao committed
135

gushiqiao's avatar
gushiqiao committed
136
### Usage Tips
gushiqiao's avatar
gushiqiao committed
137

gushiqiao's avatar
gushiqiao committed
138
139
140
141
142
143
144
> 💡 **Quantized Model Usage**: To use quantized models, refer to the [Model Conversion Script](https://github.com/ModelTC/LightX2V/blob/main/tools/convert/readme_zh.md) for conversion, or directly use pre-converted quantized models in Format 2 below
>
> 💡 **Memory Optimization**: For devices with RTX 4090 24GB or smaller memory, it's recommended to combine quantization techniques with CPU offload features:
> - Quantization Configuration: Refer to [Quantization Documentation](../method_tutorials/quantization.md)
> - CPU Offload: Refer to [Parameter Offload Documentation](../method_tutorials/offload.md)
> - Wan2.1 Configuration: Refer to [offload config files](https://github.com/ModelTC/LightX2V/tree/main/configs/offload)
> - Wan2.2 Configuration: Refer to [wan22 config files](https://github.com/ModelTC/LightX2V/tree/main/configs/wan22) with `4090` suffix
gushiqiao's avatar
gushiqiao committed
145

gushiqiao's avatar
gushiqiao committed
146
---
gushiqiao's avatar
gushiqiao committed
147

gushiqiao's avatar
gushiqiao committed
148
## 🗂️ Format 2: LightX2V Single-File Models (Recommended)
gushiqiao's avatar
gushiqiao committed
149

gushiqiao's avatar
gushiqiao committed
150
151
152
### Model Repositories
- [Wan2.1-LightX2V](https://huggingface.co/lightx2v/Wan2.1-Distill-Models)
- [Wan2.2-LightX2V](https://huggingface.co/lightx2v/Wan2.2-Distill-Models)
gushiqiao's avatar
gushiqiao committed
153

gushiqiao's avatar
gushiqiao committed
154
155
156
157
158
### Model Features
- **Single-File Management**: Single safetensors file, easy to manage and deploy
- **Multi-Precision Support**: Provides original precision, FP8, INT8, and other precision versions
- **Distillation Acceleration**: Supports 4-step fast inference
- **Tool Compatibility**: Compatible with ComfyUI and other tools
gushiqiao's avatar
gushiqiao committed
159

gushiqiao's avatar
gushiqiao committed
160
161
162
163
164
**Examples**:
- `wan2.1_i2v_720p_lightx2v_4step.safetensors` - 720P I2V original precision
- `wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors` - 720P I2V FP8 quantization
- `wan2.1_i2v_480p_int8_lightx2v_4step.safetensors` - 480P I2V INT8 quantization
- ...
gushiqiao's avatar
gushiqiao committed
165

gushiqiao's avatar
gushiqiao committed
166
### Wan2.1 Single-File Models
gushiqiao's avatar
gushiqiao committed
167

gushiqiao's avatar
gushiqiao committed
168
169
170
#### Scenario A: Download Single Model File

**Step 1: Select and Download Model**
gushiqiao's avatar
gushiqiao committed
171
172

```bash
gushiqiao's avatar
gushiqiao committed
173
174
# Create model directory
mkdir -p ./models/wan2.1_i2v_720p
gushiqiao's avatar
gushiqiao committed
175

gushiqiao's avatar
gushiqiao committed
176
177
178
179
180
# Download 720P I2V FP8 quantized model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models/wan2.1_i2v_720p \
    --include "wan2.1_i2v_720p_lightx2v_4step.safetensors"
```
gushiqiao's avatar
gushiqiao committed
181

gushiqiao's avatar
gushiqiao committed
182
183
184
185
186
187
188
189
190
191
**Step 2: Manually Organize Other Components**

Directory structure as follows:
```
wan2.1_i2v_720p/
├── wan2.1_i2v_720p_lightx2v_4step.safetensors                    # Original precision
└── t5/clip/vae/config.json/xlm-roberta-large/google and other components       # Need manual organization
```

**Step 3: Configure Launch Script**
gushiqiao's avatar
gushiqiao committed
192
193

```bash
gushiqiao's avatar
gushiqiao committed
194
195
196
197
198
199
200
# Set in launch script (point to directory containing model file)
model_path=./models/wan2.1_i2v_720p
lightx2v_path=/path/to/LightX2V

# Run script
cd LightX2V/scripts
bash wan/run_wan_i2v_distill_4step_cfg.sh
gushiqiao's avatar
gushiqiao committed
201
202
```

gushiqiao's avatar
gushiqiao committed
203
> 💡 **Tip**: When there's only one model file in the directory, LightX2V will automatically load it.
gushiqiao's avatar
gushiqiao committed
204

gushiqiao's avatar
gushiqiao committed
205
#### Scenario B: Download Multiple Model Files
gushiqiao's avatar
gushiqiao committed
206

gushiqiao's avatar
gushiqiao committed
207
When you download multiple models with different precisions to the same directory, you need to explicitly specify which model to use in the configuration file.
gushiqiao's avatar
gushiqiao committed
208

gushiqiao's avatar
gushiqiao committed
209
**Step 1: Download Multiple Models**
gushiqiao's avatar
gushiqiao committed
210
211

```bash
gushiqiao's avatar
gushiqiao committed
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
# Create model directory
mkdir -p ./models/wan2.1_i2v_720p_multi

# Download original precision model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models/wan2.1_i2v_720p_multi \
    --include "wan2.1_i2v_720p_lightx2v_4step.safetensors"

# Download FP8 quantized model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models/wan2.1_i2v_720p_multi \
    --include "wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors"

# Download INT8 quantized model
huggingface-cli download lightx2v/Wan2.1-Distill-Models \
    --local-dir ./models/wan2.1_i2v_720p_multi \
    --include "wan2.1_i2v_720p_int8_lightx2v_4step.safetensors"
gushiqiao's avatar
gushiqiao committed
229
230
```

gushiqiao's avatar
gushiqiao committed
231
232
233
**Step 2: Manually Organize Other Components**

Directory structure as follows:
gushiqiao's avatar
gushiqiao committed
234

gushiqiao's avatar
gushiqiao committed
235
236
237
238
239
```
wan2.1_i2v_720p_multi/
├── wan2.1_i2v_720p_lightx2v_4step.safetensors                    # Original precision
├── wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors   # FP8 quantization
└── wan2.1_i2v_720p_int8_lightx2v_4step.safetensors              # INT8 quantization
gushiqiao's avatar
gushiqiao committed
240
└── t5/clip/vae/config.json/xlm-roberta-large/google and other components       # Need manual organization
gushiqiao's avatar
gushiqiao committed
241
```
gushiqiao's avatar
gushiqiao committed
242

gushiqiao's avatar
gushiqiao committed
243
**Step 3: Specify Model in Configuration File**
gushiqiao's avatar
gushiqiao committed
244

gushiqiao's avatar
gushiqiao committed
245
Edit configuration file (e.g., `configs/distill/wan_i2v_distill_4step_cfg.json`):
gushiqiao's avatar
gushiqiao committed
246

gushiqiao's avatar
gushiqiao committed
247
248
249
250
```json
{
    // Use original precision model
    "dit_original_ckpt": "./models/wan2.1_i2v_720p_multi/wan2.1_i2v_720p_lightx2v_4step.safetensors",
gushiqiao's avatar
gushiqiao committed
251

gushiqiao's avatar
gushiqiao committed
252
253
254
255
    // Or use FP8 quantized model
    // "dit_quantized_ckpt": "./models/wan2.1_i2v_720p_multi/wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors",
    // "dit_quantized": true,
    // "dit_quant_scheme": "fp8-vllm",
gushiqiao's avatar
gushiqiao committed
256

gushiqiao's avatar
gushiqiao committed
257
258
259
260
    // Or use INT8 quantized model
    // "dit_quantized_ckpt": "./models/wan2.1_i2v_720p_multi/wan2.1_i2v_720p_int8_lightx2v_4step.safetensors",
    // "dit_quantized": true,
    // "dit_quant_scheme": "int8-vllm",
gushiqiao's avatar
gushiqiao committed
261

gushiqiao's avatar
gushiqiao committed
262
    // Other configurations...
gushiqiao's avatar
gushiqiao committed
263
264
}
```
gushiqiao's avatar
gushiqiao committed
265
### Usage Tips
gushiqiao's avatar
gushiqiao committed
266

gushiqiao's avatar
gushiqiao committed
267
268
269
> 💡 **Configuration Parameter Description**:
> - **dit_original_ckpt**: Used to specify the path to original precision models (BF16/FP32/FP16)
> - **dit_quantized_ckpt**: Used to specify the path to quantized models (FP8/INT8), must be used with `dit_quantized` and `dit_quant_scheme` parameters
gushiqiao's avatar
gushiqiao committed
270

gushiqiao's avatar
gushiqiao committed
271
**Step 4: Start Inference**
gushiqiao's avatar
gushiqiao committed
272

gushiqiao's avatar
gushiqiao committed
273
274
275
276
277
```bash
cd LightX2V/scripts
bash wan/run_wan_i2v_distill_4step_cfg.sh
```

gushiqiao's avatar
gushiqiao committed
278
279
> 💡 **Tip**: Other components (T5, CLIP, VAE, tokenizer, etc.) need to be manually organized into the model directory

gushiqiao's avatar
gushiqiao committed
280
281
282
283
284
### Wan2.2 Single-File Models

#### Directory Structure Requirements

When using Wan2.2 single-file models, you need to manually create a specific directory structure:
gushiqiao's avatar
gushiqiao committed
285

gushiqiao's avatar
gushiqiao committed
286
287
288
```
wan2.2_models/
├── high_noise_model/                                    # High-noise model directory (required)
gushiqiao's avatar
gushiqiao committed
289
290
291
292
│   └── wan2.2_i2v_A14b_high_noise_lightx2v_4step.safetensors
├── low_noise_model/                                     # Low-noise model directory (required)
│   └── wan2.2_i2v_A14b_low_noise_lightx2v_4step.safetensors
└── t5/clip/vae/config.json/...                          # Other components (manually organized)
gushiqiao's avatar
gushiqiao committed
293
294
```

gushiqiao's avatar
gushiqiao committed
295
#### Scenario A: Only One Model File Per Directory
gushiqiao's avatar
gushiqiao committed
296
297

```bash
gushiqiao's avatar
gushiqiao committed
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
# Create required subdirectories
mkdir -p ./models/wan2.2_models/high_noise_model
mkdir -p ./models/wan2.2_models/low_noise_model

# Download high-noise model to corresponding directory
huggingface-cli download lightx2v/Wan2.2-Distill-Models \
    --local-dir ./models/wan2.2_models/high_noise_model \
    --include "wan2.2_i2v_A14b_high_noise_scaled_fp8_e4m3_lightx2v_4step.safetensors"

# Download low-noise model to corresponding directory
huggingface-cli download lightx2v/Wan2.2-Distill-Models \
    --local-dir ./models/wan2.2_models/low_noise_model \
    --include "wan2.2_i2v_A14b_low_noise_scaled_fp8_e4m3_lightx2v_4step.safetensors"

# Configure launch script (point to parent directory)
model_path=./models/wan2.2_models
lightx2v_path=/path/to/LightX2V

# Run script
gushiqiao's avatar
gushiqiao committed
317
cd LightX2V/scripts
gushiqiao's avatar
gushiqiao committed
318
bash wan22/run_wan22_moe_i2v_distill.sh
gushiqiao's avatar
gushiqiao committed
319
320
```

gushiqiao's avatar
gushiqiao committed
321
322
323
> 💡 **Tip**: When there's only one model file in each subdirectory, LightX2V will automatically load it.

#### Scenario B: Multiple Model Files Per Directory
gushiqiao's avatar
gushiqiao committed
324

gushiqiao's avatar
gushiqiao committed
325
When you place multiple models with different precisions in both `high_noise_model/` and `low_noise_model/` directories, you need to explicitly specify them in the configuration file.
gushiqiao's avatar
gushiqiao committed
326
327

```bash
gushiqiao's avatar
gushiqiao committed
328
329
330
331
332
333
334
335
336
337
338
339
340
# Create directories
mkdir -p ./models/wan2.2_models_multi/high_noise_model
mkdir -p ./models/wan2.2_models_multi/low_noise_model

# Download multiple versions of high-noise model
huggingface-cli download lightx2v/Wan2.2-Distill-Models \
    --local-dir ./models/wan2.2_models_multi/high_noise_model \
    --include "wan2.2_i2v_A14b_high_noise_*.safetensors"

# Download multiple versions of low-noise model
huggingface-cli download lightx2v/Wan2.2-Distill-Models \
    --local-dir ./models/wan2.2_models_multi/low_noise_model \
    --include "wan2.2_i2v_A14b_low_noise_*.safetensors"
gushiqiao's avatar
gushiqiao committed
341
342
```

gushiqiao's avatar
gushiqiao committed
343
**Directory Structure**:
gushiqiao's avatar
gushiqiao committed
344

gushiqiao's avatar
gushiqiao committed
345
346
347
348
349
350
351
```
wan2.2_models_multi/
├── high_noise_model/
│   ├── wan2.2_i2v_A14b_high_noise_lightx2v_4step.safetensors        # Original precision
│   ├── wan2.2_i2v_A14b_high_noise_fp8_e4m3_lightx2v_4step.safetensors    # FP8 quantization
│   └── wan2.2_i2v_A14b_high_noise_int8_lightx2v_4step.safetensors   # INT8 quantization
└── low_noise_model/
gushiqiao's avatar
gushiqiao committed
352
353
354
355
│    ├── wan2.2_i2v_A14b_low_noise_lightx2v_4step.safetensors         # Original precision
│    ├── wan2.2_i2v_A14b_low_noise_fp8_e4m3_lightx2v_4step.safetensors     # FP8 quantization
│    └── wan2.2_i2v_A14b_low_noise_int8_lightx2v_4step.safetensors    # INT8 quantization
└── t5/vae/config.json/xlm-roberta-large/google and other components       # Need manual organization
gushiqiao's avatar
gushiqiao committed
356
```
gushiqiao's avatar
gushiqiao committed
357

gushiqiao's avatar
gushiqiao committed
358
**Configuration File Settings**:
gushiqiao's avatar
gushiqiao committed
359
360
361

```json
{
gushiqiao's avatar
gushiqiao committed
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
    // Use original precision model
    "high_noise_original_ckpt": "./models/wan2.2_models_multi/high_noise_model/wan2.2_i2v_A14b_high_noise_lightx2v_4step.safetensors",
    "low_noise_original_ckpt": "./models/wan2.2_models_multi/low_noise_model/wan2.2_i2v_A14b_low_noise_lightx2v_4step.safetensors",

    // Or use FP8 quantized model
    // "high_noise_quantized_ckpt": "./models/wan2.2_models_multi/high_noise_model/wan2.2_i2v_A14b_high_noise_fp8_e4m3_lightx2v_4step.safetensors",
    // "low_noise_quantized_ckpt": "./models/wan2.2_models_multi/low_noise_model/wan2.2_i2v_A14b_low_noise_fp8_e4m3_lightx2v_4step.safetensors",
    // "dit_quantized": true,
    // "dit_quant_scheme": "fp8-vllm"

    // Or use INT8 quantized model
    // "high_noise_quantized_ckpt": "./models/wan2.2_models_multi/high_noise_model/wan2.2_i2v_A14b_high_noise_int8_lightx2v_4step.safetensors",
    // "low_noise_quantized_ckpt": "./models/wan2.2_models_multi/low_noise_model/wan2.2_i2v_A14b_low_noise_int8_lightx2v_4step.safetensors",
    // "dit_quantized": true,
    // "dit_quant_scheme": "int8-vllm"
gushiqiao's avatar
gushiqiao committed
377
}
gushiqiao's avatar
gushiqiao committed
378
379
```

gushiqiao's avatar
gushiqiao committed
380
### Usage Tips
gushiqiao's avatar
gushiqiao committed
381

gushiqiao's avatar
gushiqiao committed
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
> 💡 **Configuration Parameter Description**:
> - **high_noise_original_ckpt** / **low_noise_original_ckpt**: Used to specify the path to original precision models (BF16/FP32/FP16)
> - **high_noise_quantized_ckpt** / **low_noise_quantized_ckpt**: Used to specify the path to quantized models (FP8/INT8), must be used with `dit_quantized` and `dit_quant_scheme` parameters


### Available Model List

#### Wan2.1 Single-File Model List

**Image-to-Video Models (I2V)**

| Filename | Precision | Description |
|--------|------|------|
| `wan2.1_i2v_480p_lightx2v_4step.safetensors` | BF16 | 4-step model original precision |
| `wan2.1_i2v_480p_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | 4-step model FP8 quantization |
| `wan2.1_i2v_480p_int8_lightx2v_4step.safetensors` | INT8 | 4-step model INT8 quantization |
| `wan2.1_i2v_480p_scaled_fp8_e4m3_lightx2v_4step_comfyui.safetensors` | FP8 | 4-step model ComfyUI format |
| `wan2.1_i2v_720p_lightx2v_4step.safetensors` | BF16 | 4-step model original precision |
| `wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | 4-step model FP8 quantization |
| `wan2.1_i2v_720p_int8_lightx2v_4step.safetensors` | INT8 | 4-step model INT8 quantization |
| `wan2.1_i2v_720p_scaled_fp8_e4m3_lightx2v_4step_comfyui.safetensors` | FP8 | 4-step model ComfyUI format |

**Text-to-Video Models (T2V)**

| Filename | Precision | Description |
|--------|------|------|
| `wan2.1_t2v_14b_lightx2v_4step.safetensors` | BF16 | 4-step model original precision |
| `wan2.1_t2v_14b_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | 4-step model FP8 quantization |
| `wan2.1_t2v_14b_int8_lightx2v_4step.safetensors` | INT8 | 4-step model INT8 quantization |
| `wan2.1_t2v_14b_scaled_fp8_e4m3_lightx2v_4step_comfyui.safetensors` | FP8 | 4-step model ComfyUI format |

#### Wan2.2 Single-File Model List

**Image-to-Video Models (I2V) - A14B Series**

| Filename | Precision | Description |
|--------|------|------|
| `wan2.2_i2v_A14b_high_noise_lightx2v_4step.safetensors` | BF16 | High-noise model - 4-step original precision |
| `wan2.2_i2v_A14b_high_noise_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | High-noise model - 4-step FP8 quantization |
| `wan2.2_i2v_A14b_high_noise_int8_lightx2v_4step.safetensors` | INT8 | High-noise model - 4-step INT8 quantization |
| `wan2.2_i2v_A14b_low_noise_lightx2v_4step.safetensors` | BF16 | Low-noise model - 4-step original precision |
| `wan2.2_i2v_A14b_low_noise_scaled_fp8_e4m3_lightx2v_4step.safetensors` | FP8 | Low-noise model - 4-step FP8 quantization |
| `wan2.2_i2v_A14b_low_noise_int8_lightx2v_4step.safetensors` | INT8 | Low-noise model - 4-step INT8 quantization |

> 💡 **Usage Tips**:
> - Wan2.2 models use a dual-noise architecture, requiring both high-noise and low-noise models to be downloaded
> - Refer to the "Wan2.2 Single-File Models" section above for detailed directory organization

---

## 🗂️ Format 3: LightX2V LoRA Models

LoRA (Low-Rank Adaptation) models provide a lightweight model fine-tuning solution that enables customization for specific effects without modifying the base model.

### Model Repositories

- **Wan2.1 LoRA Models**: [lightx2v/Wan2.1-Distill-Loras](https://huggingface.co/lightx2v/Wan2.1-Distill-Loras)
- **Wan2.2 LoRA Models**: [lightx2v/Wan2.2-Distill-Loras](https://huggingface.co/lightx2v/Wan2.2-Distill-Loras)

### Usage Methods

#### Method 1: Offline Merging

Merge LoRA weights offline into the base model to generate a new complete model file.

**Steps**:

Refer to the [Model Conversion Documentation](https://github.com/ModelTC/lightx2v/tree/main/tools/convert/readme_zh.md) for offline merging.

**Advantages**:
- ✅ No need to load LoRA during inference
- ✅ Better performance

**Disadvantages**:
- ❌ Requires additional storage space
- ❌ Switching different LoRAs requires re-merging

#### Method 2: Online Loading

Dynamically load LoRA weights during inference without modifying the base model.

**LoRA Application Principle**:

```python
# LoRA weight application formula
gushiqiao's avatar
gushiqiao committed
467
468
# lora_scale = (alpha / rank)
# W' = W + lora_scale * B @ A
gushiqiao's avatar
gushiqiao committed
469
470
471
472
# Where: B = up_proj (out_features, rank)
#        A = down_proj (rank, in_features)

if weights_dict["alpha"] is not None:
gushiqiao's avatar
gushiqiao committed
473
    lora_scale = weights_dict["alpha"] / lora_down.shape[0]
gushiqiao's avatar
gushiqiao committed
474
elif alpha is not None:
gushiqiao's avatar
gushiqiao committed
475
    lora_scale = alpha / lora_down.shape[0]
gushiqiao's avatar
gushiqiao committed
476
else:
gushiqiao's avatar
gushiqiao committed
477
    lora_scale = 1.0
gushiqiao's avatar
gushiqiao committed
478
```
gushiqiao's avatar
gushiqiao committed
479

gushiqiao's avatar
gushiqiao committed
480
481
482
**Configuration Method**:

**Wan2.1 LoRA Configuration**:
gushiqiao's avatar
gushiqiao committed
483

gushiqiao's avatar
gushiqiao committed
484
485
```json
{
gushiqiao's avatar
gushiqiao committed
486
487
488
489
490
491
492
  "lora_configs": [
    {
      "path": "wan2.1_i2v_lora_rank64_lightx2v_4step.safetensors",
      "strength": 1.0,
      "alpha": null
    }
  ]
gushiqiao's avatar
gushiqiao committed
493
494
}
```
gushiqiao's avatar
gushiqiao committed
495

gushiqiao's avatar
gushiqiao committed
496
497
498
**Wan2.2 LoRA Configuration**:

Since Wan2.2 uses a dual-model architecture (high-noise/low-noise), LoRA needs to be configured separately for both models:
gushiqiao's avatar
gushiqiao committed
499

gushiqiao's avatar
gushiqiao committed
500
501
```json
{
gushiqiao's avatar
gushiqiao committed
502
503
504
505
506
507
508
509
510
511
512
513
514
515
  "lora_configs": [
    {
      "name": "low_noise_model",
      "path": "wan2.2_i2v_A14b_low_noise_lora_rank64_lightx2v_4step.safetensors",
      "strength": 1.0,
      "alpha": null
    },
    {
      "name": "high_noise_model",
      "path": "wan2.2_i2v_A14b_high_noise_lora_rank64_lightx2v_4step.safetensors",
      "strength": 1.0,
      "alpha": null
    }
  ]
gushiqiao's avatar
gushiqiao committed
516
517
}
```
gushiqiao's avatar
gushiqiao committed
518

gushiqiao's avatar
gushiqiao committed
519
**Parameter Description**:
gushiqiao's avatar
gushiqiao committed
520

gushiqiao's avatar
gushiqiao committed
521
522
523
524
| Parameter | Description | Default |
|------|------|--------|
| `path` | LoRA model file path | Required |
| `strength` | LoRA strength coefficient, range [0.0, 1.0] | 1.0 |
gushiqiao's avatar
gushiqiao committed
525
| `alpha` | LoRA scaling factor, uses model's built-in value when `null` | null |
gushiqiao's avatar
gushiqiao committed
526
| `name` | (Wan2.2 only) Specifies which model to apply to | Required |
gushiqiao's avatar
gushiqiao committed
527

gushiqiao's avatar
gushiqiao committed
528
529
530
531
**Advantages**:
- ✅ Flexible switching between different LoRAs
- ✅ Saves storage space
- ✅ Can dynamically adjust LoRA strength
gushiqiao's avatar
gushiqiao committed
532

gushiqiao's avatar
gushiqiao committed
533
534
535
**Disadvantages**:
- ❌ Additional loading time during inference
- ❌ Slightly increases memory usage
gushiqiao's avatar
gushiqiao committed
536

gushiqiao's avatar
gushiqiao committed
537
---
gushiqiao's avatar
gushiqiao committed
538

gushiqiao's avatar
gushiqiao committed
539
## 📚 Related Resources
gushiqiao's avatar
gushiqiao committed
540

gushiqiao's avatar
gushiqiao committed
541
542
543
544
### Official Repositories
- [LightX2V GitHub](https://github.com/ModelTC/LightX2V)
- [LightX2V Single-File Model Repository](https://huggingface.co/lightx2v/Wan2.1-Distill-Models)
- [Wan-AI Official Model Repository](https://huggingface.co/Wan-AI)
gushiqiao's avatar
gushiqiao committed
545

gushiqiao's avatar
gushiqiao committed
546
### Model Download Links
gushiqiao's avatar
gushiqiao committed
547

gushiqiao's avatar
gushiqiao committed
548
549
**Wan2.1 Series**
- [Wan2.1 Collection](https://huggingface.co/collections/Wan-AI/wan21-68ac4ba85372ae5a8e282a1b)
gushiqiao's avatar
gushiqiao committed
550

gushiqiao's avatar
gushiqiao committed
551
552
**Wan2.2 Series**
- [Wan2.2 Collection](https://huggingface.co/collections/Wan-AI/wan22-68ac4ae80a8b477e79636fc8)
gushiqiao's avatar
gushiqiao committed
553

gushiqiao's avatar
gushiqiao committed
554
555
556
**LightX2V Single-File Models**
- [Wan2.1-Distill-Models](https://huggingface.co/lightx2v/Wan2.1-Distill-Models)
- [Wan2.2-Distill-Models](https://huggingface.co/lightx2v/Wan2.2-Distill-Models)
gushiqiao's avatar
gushiqiao committed
557

gushiqiao's avatar
gushiqiao committed
558
559
560
### Documentation Links
- [Quantization Documentation](../method_tutorials/quantization.md)
- [Parameter Offload Documentation](../method_tutorials/offload.md)
gushiqiao's avatar
gushiqiao committed
561
- [Configuration File Examples](https://github.com/ModelTC/LightX2V/tree/main/configs)
gushiqiao's avatar
gushiqiao committed
562
563
564

---

gushiqiao's avatar
gushiqiao committed
565
566
567
568
569
570
571
572
573
Through this document, you should be able to:

✅ Understand all model formats supported by LightX2V
✅ Select appropriate models and precisions based on your needs
✅ Correctly download and organize model files
✅ Configure launch parameters and successfully run inference
✅ Resolve common model loading issues

If you have other questions, feel free to ask in [GitHub Issues](https://github.com/ModelTC/LightX2V/issues).