README_old.md

# 简介

使用PyTorch框架计算Bert网络。

* BERT 的训练分为pre-train和fine-tune两种，pre-train训练分为两个phrase。

* BERT 的推理可基于不同数据集进行精度验证
* 数据生成、模型转换相关细节见  [README.md](http://10.0.100.3/dcutoolkit/deeplearing/dlexamples/-/blob/develop/PyTorch/NLP/BERT/scripts/README.md)

# 运行示例

目前提供基于wiki英文数据集 pre-train 两个阶段的训练和基于squad数据集fine-tune 训练的代码示例，

## pre-train phrase1

|参数名|解释|示例|
|:---:|:---:|:---:|
|PATH_PHRASE1|第一阶段训练数据集路径|/workspace/lower_case_1_seq_len_128_max_pred_20_masked_lm_prob_0.<br>15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10
|OUTPUT_DIR|输出路径|/workspace/results
|PATH_CONFIG|confing路径|/workspace/bert_large_uncased
|PATH_PHRASE2|第一阶段训练数据集路径|/workspace/lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.<br>15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10
<br>

### 单卡
```
export HIP_VISIBLE_DEVICES=0
python3 run_pretraining_v1.py  \
    --input_dir=${PATH_PHRASE1}    \
    --output_dir=${OUTPUT_DIR}/checkpoints1 \
    --config_file=${PATH_CONFIG}bert_config.json \
    --bert_model=bert-large-uncased \
    --train_batch_size=16 \
    --max_seq_length=128 \
    --max_predictions_per_seq=20 \
    --max_steps=100000 \
    --warmup_proportion=0.0 \
    --num_steps_per_checkpoint=20000 \
    --learning_rate=4.0e-4 \
    --seed=12439 \
    --gradient_accumulation_steps=1 \
    --allreduce_post_accumulation \
    --do_train \
    --json-summary dllogger.json
```

### 多卡

* 方法一
```
export HIP_VISIBLE_DEVICES=0,1,2,3
python3 run_pretraining_v1.py  \
    --input_dir=${PATH_PHRASE1}    \
    --output_dir=${OUTPUT_DIR}/checkpoints \
    --config_file=${PATH_CONFIG}bert_config.json \
    --bert_model=bert-large-uncased \
    --train_batch_size=16 \
    --max_seq_length=128 \
    --max_predictions_per_seq=20 \
    --max_steps=100000 \
    --warmup_proportion=0.0 \
    --num_steps_per_checkpoint=20000 \
    --learning_rate=4.0e-4 \
    --seed=12439 \
    --gradient_accumulation_steps=1 \
    --allreduce_post_accumulation \
    --do_train \
    --json-summary dllogger.json
```
* 方法二

hostfile:
```
node1 slots=4
node2 slots=4
```

```
#scripts/run_pretrain.sh 脚本默认每个节点四块卡
cd scripts; bash run_pretrain.sh
```


## pre-train phrase2

### 单卡
```
HIP_VISIBLE_DEVICES=0
python3 run_pretraining_v1.py
   --input_dir=${PATH_PHRASE2} \
   --output_dir=${OUTPUT_DIR}/checkpoints2 \
   --config_file=${PATH_CONFIG}bert_config.json \
   --bert_model=bert-large-uncased \
   --train_batch_size=4 \
   --max_seq_length=512 \
   --max_predictions_per_seq=80 \
   --max_steps=400000 \
   --warmup_proportion=0.128 \
   --num_steps_per_checkpoint=200000 \
   --learning_rate=4e-3 \
   --seed=12439 \
   --gradient_accumulation_steps=1 \
   --allreduce_post_accumulation \
   --do_train \
   --phase2 \
   --phase1_end_step=0 \
   --json-summary dllogger.json
```

### 多卡

* 方法一
```
export HIP_VISIBLE_DEVICES=0,1,2,3
python3 run_pretraining_v1.py
   --input_dir=${PATH_PHRASE2} \
   --output_dir=${OUTPUT_DIR}/checkpoints2 \
   --config_file=${PATH_CONFIG}bert_config.json \
   --bert_model=bert-large-uncased \
   --train_batch_size=4 \
   --max_seq_length=512 \
   --max_predictions_per_seq=80 \
   --max_steps=400000 \
   --warmup_proportion=0.128 \
   --num_steps_per_checkpoint=200000 \
   --learning_rate=4e-3 \
   --seed=12439 \
   --gradient_accumulation_steps=1 \
   --allreduce_post_accumulation \
   --do_train \
   --phase2 \
   --phase1_end_step=0 \
   --json-summary dllogger.json
```
* 方法二

hostfile:
```
node1 slots=4
node2 slots=4
```

```
#scripts/run_pretrain2.sh 脚本默认每个节点四块卡
cd scripts; bash run_pretrain2.sh
```



## fine-tune 训练

### 单卡
```
python3 run_squad_v1.py \
  --train_file squad/v1.1/train-v1.1.json \
  --init_checkpoint model.ckpt-28252.pt \
  --vocab_file vocab.txt \
  --output_dir SQuAD \
  --config_file bert_config.json \
  --bert_model=bert-large-uncased \
  --do_train \
  --train_batch_size 1 \
  --gpus_per_node 1 
```
### 多卡

hostfile:
```
node1 slots=4
node2 slots=4
```

```
#scripts/run_squad_1.sh 脚本默认每个节点四块卡
bash run_squad_1.sh
```



# 参考资料
[https://github.com/mlperf/training_results_v0.7/blob/master/NVIDIA/benchmarks/bert/implementations/pytorch](https://github.com/mlperf/training_results_v0.7/blob/master/NVIDIA/benchmarks/bert/implementations/pytorch)
[https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/LanguageModeling/BERT](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/LanguageModeling/BERT)