parallel.md 1.3 KB
Newer Older
helloyongyang's avatar
helloyongyang committed
1
# Parallel Inference
helloyongyang's avatar
helloyongyang committed
2

helloyongyang's avatar
helloyongyang committed
3
LightX2V supports distributed parallel inference, enabling the utilization of multiple GPUs for inference. The DiT component supports two parallel attention mechanisms: **Ulysses** and **Ring**, while also supporting **Cfg parallel inference**. Parallel inference significantly reduces inference time and alleviates memory overhead on each GPU.
helloyongyang's avatar
helloyongyang committed
4
5
6
7
8

## DiT Parallel Configuration

### 1. Ulysses Parallel

helloyongyang's avatar
helloyongyang committed
9
**Configuration method:**
helloyongyang's avatar
helloyongyang committed
10
```json
helloyongyang's avatar
helloyongyang committed
11
12
13
14
    "parallel": {
        "seq_p_size": 4,
        "seq_p_attn_type": "ulysses"
    }
helloyongyang's avatar
helloyongyang committed
15
16
17
18
```

### 2. Ring Parallel

helloyongyang's avatar
helloyongyang committed
19
**Configuration method:**
helloyongyang's avatar
helloyongyang committed
20
```json
helloyongyang's avatar
helloyongyang committed
21
22
23
24
    "parallel": {
        "seq_p_size": 4,
        "seq_p_attn_type": "ring"
    }
helloyongyang's avatar
helloyongyang committed
25
26
```

helloyongyang's avatar
helloyongyang committed
27
## Cfg Parallel Configuration
helloyongyang's avatar
helloyongyang committed
28

helloyongyang's avatar
helloyongyang committed
29
**Configuration method:**
helloyongyang's avatar
helloyongyang committed
30
```json
helloyongyang's avatar
helloyongyang committed
31
32
33
    "parallel": {
        "cfg_p_size": 2
    }
helloyongyang's avatar
helloyongyang committed
34
35
```

helloyongyang's avatar
helloyongyang committed
36
## Hybrid Parallel Configuration
helloyongyang's avatar
helloyongyang committed
37

helloyongyang's avatar
helloyongyang committed
38
39
40
41
42
43
44
45
**Configuration method:**
```json
    "parallel": {
        "seq_p_size": 4,
        "seq_p_attn_type": "ulysses",
        "cfg_p_size": 2
    }
```
helloyongyang's avatar
helloyongyang committed
46
47
48

## Usage

helloyongyang's avatar
helloyongyang committed
49
Parallel inference configuration files are available [here](https://github.com/ModelTC/lightx2v/tree/main/configs/dist_infer)
helloyongyang's avatar
helloyongyang committed
50

helloyongyang's avatar
helloyongyang committed
51
By specifying --config_json to a specific config file, you can test parallel inference.
helloyongyang's avatar
helloyongyang committed
52

helloyongyang's avatar
helloyongyang committed
53
[Here](https://github.com/ModelTC/lightx2v/tree/main/scripts/dist_infer) are some run scripts for your use.