README.md-old2

### 1.Benchmark下载地址

transformer：https://github.com/pytorch/fairseq.git

最新的githup上代码和现在存在区别，最好使用本地旧版本代码，否则可能出现HIP版本的torch不识别情况

### 2.1 数据集准备

```
环境准备：

pip3 install fastBPE sacremoses subword_nmt

数据集下载：

cd examples/translation/

bash prepare-wmt14en2de.sh --icml17

数据预处理：

DATA_DIR=`pwd`/data-bin/wmt14_en_de_joined_dict

TEXT=`pwd`/wmt14_en_de

fairseq-preprocess --source-lang en --target-lang de --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test --destdir $DATA_DIR --nwordssrc 32768 --nwordstgt 32768 --joined-dictionary --workers 20

相关参数说明：

--source-lang   source language

--target-lang   target language

--trainpref     rain file prefix (also used to build dictionaries)

--validpref     comma separated, valid file prefixes (words missing from train set are replaced with <unk>)

--testpref      comma separated, test file prefixes (words missing from train set are replaced with <unk>)

--destdir       destination dir, Default: “data-bin”

数据集路径：

DATA_PATH=`pwd`/data-bin/wmt14_en_de_joined_dict
```



#### 2.2.环境部署

##### 2.2.1.构建测试的虚拟环境

```
virtualenv -p python3 venv

source venv/bin/activate
```

##### 2.2.2.安装python3.6环境下的依赖包

```
pip3 install --upgrade pip

pip3 install typing

pip3 install sacremoses

pip3 install numpy

pip3 install torch-1.10.0a0+gitcc7c9c7-cp36-cp36m-linux_x86_64.whl

pip3 install apex-0.1-cp36-cp36m-linux_x86_64.whl 
```

##### 2.2.3.安装fairseq

```
git clone https://github.com/pytorch/fairseq.git

cd fairseq

pip3 install --editable ./

相关说明：可以先把setup.py里的torch、torch-audio的安装屏蔽掉
```

##### 2.2.4.环境变量里设置env.sh

（昆山平台依赖库已配置好，直接设置环境变量使用）

```
module unload compiler/rocm/2.9 

module load compiler/rocm/dtk-21.10.1 

export PYTORCH_DEP=/public/software/apps/DeepLearning/PyTorch

export C_INCLUDE_PATH=${PYTORCH_DEP}/gflags-2.1.2-build/include:${PYTORCH_DEP}/glog-build/include:$C_INCLUDE_PATH

export CPLUS_INCLUDE_PATH=${PYTORCH_DEP}/gflags-2.1.2-build/include:${PYTORCH_DEP}/glog-build/include:$CPLUS_INCLUDE_PATH

export LD_LIBRARY_PATH=${PYTORCH_DEP}/glog-build/lib/:${PYTORCH_DEP}/lmdb-0.9.24-build/lib/:${PYTORCH_DEP}/opencv-2.4.13.6-build/lib/:${PYTORCH_DEP}/openblas-0.3.7-build/lib/:${PYTORCH_DEP}/gflags-2.1.2-build/lib/:${PYTORCH_DEP}/lib/:${PYTORCH_DEP}/openmp-build/lib:$LD_LIBRARY_PATH

export NCCL_SOCKET_IFNAME=eno1

export HSA_FORCE_FINE_GRAIN_PCIE=1

export MIOPEN_FIND_MODE=3
```

### 3.transformer测试（昆山）

```
WORK_PATH=~/fairseq

source env.sh
```

#### 3.1.单卡测试（单精度）

##### 3.1.1.run_transformer_single.sh

```
export HIP_VISIBLE_DEVICES=0

python3 $WORK_PATH/train.py \

  $DATA_PATH \

   --arch transformer_wmt_en_de --share-decoder-input-output-embed \

   --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \

   --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 \

   --dropout 0.3 --weight-decay 0.0001 \

   --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \

   --max-tokens 2560 \
 
   --eval-bleu \

   --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' \

   --eval-bleu-detok moses \

   --eval-bleu-remove-bpe \

   --eval-bleu-print-samples \

   --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --max-epoch 1
```

##### 3.1.2.运行
```
./ run_transformer_single.sh
```

#### 3.2.四卡测试（单精度）

##### 3.2.1.single_process.sh

```
#!/bin/bash

export MIOPEN_DEBUG_DISABLE_FIND_DB=1

export NCCL_SOCKET_IFNAME=eno1

export HSA_USERPTR_FOR_PAGED_MEM=0

lrank=$OMPI_COMM_WORLD_LOCAL_RANK

comm_rank=$OMPI_COMM_WORLD_RANK

comm_size=$OMPI_COMM_WORLD_SIZE

TOKENS=2560

DATA_PATH=~/fairseq/examples/translation/data-bin/wmt14_en_de_joined_dict

APP="python3 ~/fairseq/train.py $DATA_PATH --arch transformer_wmt_en_de --share-decoder-input-output-embed --optimizer adam --adam-betas (0.9,0.98) --clip-norm 0.0 --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 --dropout 0.3 --weight-decay 0.0001 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-tokens $TOKENS --eval-bleu --eval-bleu-args {\"beam\":5,\"max_len_a\":1.2,\"max_len_b\":10} --eval-bleu-detok moses --eval-bleu-remove-bpe --eval-bleu-print-samples --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --distributed-rank ${comm_rank} --distributed-world-size ${comm_size} --device-id ${lrank} --local_rank ${lrank} --distributed-init-method tcp://${1}:34567 --distributed-no-spawn --max-epoch 1"

case ${lrank} in

[0])

  export HIP_VISIBLE_DEVICES=0,1,2,3

  export UCX_NET_DEVICES=mlx5_0:1

  export UCX_IB_PCI_BW=mlx5_0:50Gbs

  echo NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=0 --membind=0 ${APP}

  NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=0 --membind=0 ${APP}

  ;;

[1])

  export HIP_VISIBLE_DEVICES=0,1,2,3

  export UCX_NET_DEVICES=mlx5_1:1

  export UCX_IB_PCI_BW=mlx5_1:50Gbs

  echo NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=1 --membind=1 ${APP}

  NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=1 --membind=1 ${APP}

  ;;

[2])

  export HIP_VISIBLE_DEVICES=0,1,2,3

  export UCX_NET_DEVICES=mlx5_2:1

  export UCX_IB_PCI_BW=mlx5_2:50Gbs

  echo NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=2 --membind=2 ${APP}

  NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=2 --membind=2 ${APP}

  ;;

[3])

  export HIP_VISIBLE_DEVICES=0,1,2,3

  export UCX_NET_DEVICES=mlx5_3:1

  export UCX_IB_PCI_BW=mlx5_3:50Gbs

  echo NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=3 --membind=3 ${APP}

  NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=3 --membind=3 ${APP}

  ;;

esac
```

##### 3.2.2.run_transformer_4dcus.sh

```
#!/usr/bin/env bash

#SBATCH -J distribute

#SBATCH -p kshdtest

#SBATCH -N 1

#SBARCH -n 32

#SBATCH --ntasks-per-node=4

#SBATCH --cpus-per-task=8

#SBATCH --gres=dcu:4

set -x

source env.sh

hostfile=./$SLURM_JOB_ID

scontrol show hostnames $SLURM_JOB_NODELIST > ${hostfile}

for i in `cat $hostfile`

do

    echo ${i} slots=4 >> `pwd`/hostfile-$SLURM_JOB_ID

    ((num_node=${num_node}+1))

done

num_dcu=$((${num_node}\4))

echo $num_dcu

nodename=$(cat $hostfile |sed -n "1p")

echo $nodename

dist_url=`echo $nodename | awk '{print $1}'`

export HSA_USERPTR_FOR_PAGED_MEM=0

mpirun -np ${num_dcu} --hostfile hostfile-$SLURM_JOB_ID single_process.sh  $dist_url
```

##### 3.2.3.运行
```
sbatch run_transformer_4dcus.sh
```

##### 3.2.4.参数说明

- 上面的中single_process.sh需要关注--max-tokens；
-  通过--arch 设置要测试的网络，eg:transformer_wmt_en_de 等；
- 上述 run_transformer_4dcus.sh中mpirun 运行命令表示使用4张DCU加速卡训练。

#### 3.3.单卡测试（半精度）

##### 3.3.1.fp16_run_transformer.sh

```
export HIP_VISIBLE_DEVICES=2

python3 $WORK_PATH/train.py \

    $DATA_PATH \

    --arch transformer_wmt_en_de --share-decoder-input-output-embed \

    --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \

    --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 \

    --dropout 0.3 --weight-decay 0.0001 \

    --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \

    --max-tokens 2560 \

    --eval-bleu \

    --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' \

    --eval-bleu-detok moses \

    --eval-bleu-remove-bpe \

    --eval-bleu-print-samples \

    --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --max-epoch 1 --fp16
```

##### 3.3.2.运行

./ fp16_run_transformer.sh

##### 3.3.3.参数说明

--max-tokens 根据tokens设置batch size

--fp16  使用半精度训练

#### 3.4.四卡测试（半精度）

##### 3.4.1.fp16_single_process.sh

```
#!/bin/bash

export MIOPEN_DEBUG_DISABLE_FIND_DB=1

export NCCL_SOCKET_IFNAME=eno1

export HSA_USERPTR_FOR_PAGED_MEM=0

lrank=$OMPI_COMM_WORLD_LOCAL_RANK

comm_rank=$OMPI_COMM_WORLD_RANK

comm_size=$OMPI_COMM_WORLD_SIZE

TOKENS=2560

DATA_PATH=~/fairseq/examples/translation/data-bin/wmt14_en_de_joined_dict

APP="python3 ~/fairseq/train.py $DATA_PATH --arch transformer_wmt_en_de --share-decoder-input-output-embed --optimizer adam --adam-betas (0.9,0.98) --clip-norm 0.0 --lr 5e-4 --lr-scheduler inverse_sqrt --warmup-updates 4000 --dropout 0.3 --weight-decay 0.0001 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-tokens $TOKENS --eval-bleu --eval-bleu-args {\"beam\":5,\"max_len_a\":1.2,\"max_len_b\":10} --eval-bleu-detok moses --eval-bleu-remove-bpe --eval-bleu-print-samples --best-checkpoint-metric bleu --maximize-best-checkpoint-metric --distributed-rank ${comm_rank} --distributed-world-size ${comm_size} --device-id ${lrank} --local_rank ${lrank} --distributed-init-method tcp://${1}:34567 --distributed-no-spawn --max-epoch 1 --fp16"

case ${lrank} in

[0])

  export HIP_VISIBLE_DEVICES=0,1,2,3

  export UCX_NET_DEVICES=mlx5_0:1

  export UCX_IB_PCI_BW=mlx5_0:50Gbs

  echo NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=0 --membind=0 ${APP}

  NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=0 --membind=0 ${APP}

  ;;

[1])

  export HIP_VISIBLE_DEVICES=0,1,2,3

  export UCX_NET_DEVICES=mlx5_1:1

  export UCX_IB_PCI_BW=mlx5_1:50Gbs

  echo NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=1 --membind=1 ${APP}

  NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=1 --membind=1 ${APP}

  ;;

[2])

  export HIP_VISIBLE_DEVICES=0,1,2,3

  export UCX_NET_DEVICES=mlx5_2:1

  export UCX_IB_PCI_BW=mlx5_2:50Gbs

  echo NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=2 --membind=2 ${APP} 

  NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=2 --membind=2 ${APP}

  ;;

[3])

  export HIP_VISIBLE_DEVICES=0,1,2,3

  export UCX_NET_DEVICES=mlx5_3:1

  export UCX_IB_PCI_BW=mlx5_3:50Gbs

  echo NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=3 --membind=3 ${APP}

  NCCL_SOCKET_IFNAME=ib0 numactl --cpunodebind=3 --membind=3 ${APP}

  ;;

esac
```

##### 3.4.2.fp16_run_transformer_4dcus.sh

```
#!/usr/bin/env bash

#SBATCH -J distribute

#SBATCH -p kshdtest

#SBATCH -N 1

#SBARCH -n 32

#SBATCH --ntasks-per-node=4

#SBATCH --cpus-per-task=8

#SBATCH --gres=dcu:4

set -x

source env.sh

hostfile=./$SLURM_JOB_ID

scontrol show hostnames $SLURM_JOB_NODELIST > ${hostfile}

for i in `cat $hostfile`

do

    echo ${i} slots=4 >> `pwd`/hostfile-$SLURM_JOB_ID

    ((num_node=${num_node}+1))

done

num_dcu=$((${num_node}\4))

echo $num_dcu

nodename=$(cat $hostfile |sed -n "1p")

echo $nodename

dist_url=`echo $nodename | awk '{print $1}'`

export HSA_USERPTR_FOR_PAGED_MEM=0

mpirun -np ${num_dcu} --hostfile hostfile-$SLURM_JOB_ID single_process.sh  $dist_url
```

##### 3.4.3.运行

sbatch fp16_ run_transformer_4dcus.sh

##### 3.4.4.参数说明

- 上面的中single_process.sh需要关注--max-tokens；
-  通过--arch 设置要测试的网络，eg:transformer_wmt_en_de 等；
- 上述 run_transformer_4dcus.sh中mpirun 运行命令表示使用4张DCU加速卡训练。

#### 3.5. 部分问题说明

##### 3.5.1. format错误

报错信息如下：

```
  File "~/virturlenv-test/venv/lib/python3.6/site-packages/sacrebleu/metrics/bleu.py", line 103, in __init__
    self._verbose += f"ratio = {self.ratio:.3f} hyp_len = {self.sys_len:d} "
 ...

ValueError: Unknown format code 'd' for object of type 'float'
```

修改方法：修改报错提示中的bleu.py，103和104行的d改成.0f

```
#修改后
self._verbose += f"ratio = {self.ratio:.3f} hyp_len ={self.sys_len:.0f}"
self._verbose += f"ref_len = {slef.ref_len:.0f}"
```



##### 3.5.2 json格式解析错误

报错信息如下：

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

解析错误的地方，可以看出是引号嵌套问题

错误的打印

```
'eval_bleu_args': "'{beam:5,max_len_a:1.2,max_len_b:10}'
```

正确的打印

```
'eval_bleu_args': '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}'
```

修改为：

```
" --eval-bleu-args {\"beam\":5,\"max_len_a\":1.2,\"max_len_b\":10}"
```