"tests/test_zero/test_legacy/common.py" did not exist on "d0ae0f2215750a158dd66d4428e0ed6b5f5141a9"
README.md 914 Bytes
Newer Older
Ziyue Jiang's avatar
polish  
Ziyue Jiang committed
1
# Pipeline Parallelism Demo with GPT2
Ziyue Jiang's avatar
Ziyue Jiang committed
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

## Requirements

Before you can launch training, you need to install the following requirements.

### Install PyTorch

```bash
#conda
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch
#pip
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113
```

### Install [Colossal-AI v0.2.0](https://colossalai.org/download/) From Official Website

```bash
pip install colossalai==0.2.0+torch1.12cu11.3 -f https://release.colossalai.org
```

### Install transformers

```bash
pip install transformers
```

## Dataset

For simplicity, the input data is randonly generated here.

## Training

```bash
#Run the Pipeline Parallel on GPT with default setting and a dummy dataset.
Ziyue Jiang's avatar
polish  
Ziyue Jiang committed
36
#You can change the GPU number or microbatch number in the run.sh .
Ziyue Jiang's avatar
Ziyue Jiang committed
37
38
bash run.sh
```