Initial commit

README.md

-# SpeechT5_pytorch
+# SpeechT5
 
-speecht5是微软推出的语音模型，支持文本到语音，语音到语音，语音到文本的多个模态的推理
+## 论文
\ No newline at end of file
+
+  - https://arxiv.org/abs/2110.07205
+
+
+## 开源代码
+
+  - https://github.com/microsoft/SpeechT5
+
+
+## 模型结构
+
+speechT5的核心是一个常规的**Transformer编码器-解码器**，为了使得同一个Transformer可以同时处理文本和语音数据，添加了**pre-nets**和**post-nets**，**pre-net**将输入的文本或语音转换为Transformer使用的隐藏表示；**post-net**从Transformer中获取输出并转换为文本或语音。
+<div align="center">
+    <img src="./images/model_architecture.png"/>
+</div>
+
+
+## 算法原理
+
+在预训练期间，同时使用所有的 per-nets 和 post-nets 。预训练后，整个编码器 - 解码器主干在单个任务上进行微调。这种经过微调的模型仅使用特定于给定任务的 per-nets 和 post-nets 。*例如：要将 SpeechT5 用于文本到语音转换，您需要将文本编码器 per-nets 交换为文本输入，将语音解码器 per-nets 和 post-nets 交换为语音输出。*
+<div align="center">
+    <img src="./images/principle_algorithm.png"/>
+</div>
+
+**注意: 即使微调模型一开始使用共享预训练模型的同一组权重，但最终版本最终还是完全不同。例如，您不能采用经过微调的 ASR 模型并换掉 per-nets 和 post-nets 来获得有效的 TTS 模型。SpeechT5 很灵活，但不是那么灵活。**
+
+
+## 环境配置
+
+### Docker (方法一)
+**注意修改路径参数**
+
+```
+docker pull image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.1.0-centos7.6-dtk24.04-py310
+
+docker run -it --network=host --ipc=host --name=your_container_name --shm-size=32G --device=/dev/kfd --device=/dev/mkfd --device=/dev/dri -v /opt/hyhal:/opt/hyhal:ro -v /path/your_code_data/:/path/your_code_data/ --group-add video --cap-add=SYS_PTRACE --security-opt seccomp=unconfined image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.1.0-centos7.6-dtk24.04-py310 /bin/bash
+
+cd /path/your_code_data/
+pip3 install -r requirements.txt
+```
+
+### Dockerfile (方法二)
+
+```
+cd ./docker
+docker build --no-cache -t speecht5 .
+docker run -it -v /path/your_code_data/:/path/your_code_data/ --shm-size=32G --privileged=true --device=/dev/kfd --device=/dev/dri/ --group-add video --name docker_name imageID bash
+
+pip3 install -r requirements.txt
+```
+
+### Anaconda (方法三)
+
+1、关于本项目DCU显卡所需的特殊深度学习库可从光合开发者社区下载安装： https://developer.hpccube.com/tool/
+
+```
+DTK软件栈：dtk24.04
+python：python3.10
+torch：2.1.0
+torchvision：0.16.0
+```
+
+Tips：以上dtk软件栈、python、torch等DCU相关工具版本需要严格一一对应
+
+2、其他非特殊库直接按照requirements.txt安装
+
+```
+pip3 install -r requirements.txt
+```
+
+## 数据集
+
+**在本案例中已经构建了案例测试数据，无需手动下载数据集。若需要完整数据集，请按照下载链接进行下载**
+
+`CMU_ARCTIC`：说话人识别的数据集，其将每个说话人的声音特征描述为(1,512)的张量，音频特征文件以npy格式存储。
+
+
+```
+├── CMU_ARCTIC 
+│   ├── cmu_us_awb_arctic-wav-arctic_a0001.npy
+│   ├── cmu_us_awb_arctic-wav-arctic_a0002.npy
+│   ├── cmu_us_awb_arctic-wav-arctic_a0003.npy
+│   ├── ...
+
+```
+ - `cmu_us_awb_arctic-wav-arctic_a0001.npy`：说话人的声音特征文件。
+
+    [CMU_ARCTIC数据集下载](https://hf-mirror.com/datasets/Matthijs/cmu-arctic-xvectors)
+
+
+`librispeech_asr`：语音识别数据集，数据集中包括音频文件以及文本转录文件。其中音频文件以flac格式存储，文本转录文件以txt格式存储。
+
+
+```
+LibriSpeech
+├── train-clean-100
+│   ├── 1272
+│   │   ├── 1272-128104
+│   │   │   ├── 1272-128104-0000.flac
+│   │   │   ├── 1272-128104-0001.flac
+│   │   │   ├── 1272-128104-0002.flac
+│   │   │   ├── 1272-128104-0003.flac
+│   │   │   ├── ...
+│   │   │   ├── 1272-128104.trans.txt
+│   │   └── ...
+│   └── ...
+├── train-clean-360
+├── train-other-500
+├── dev-clean
+├── dev-other
+├── test-clean
+└── test-othe
+```
+
+  - `train-clean-100`：包含大约 100 小时的清晰语音。
+  - `1272`：说话人ID(1272)。
+  - `1272-128104`:说话人ID(1272)-文本章节ID(128204)。
+  - `1272-128104-0000.flac`:说话人ID(1272)-文本章节ID(128204)-文本片段ID(0)的音频文件。
+  - `1272-128104.trans.txt`:说话人ID(1272)-文本章节ID(128204)的转录文本文件。
+    [librispeech_asr数据集下载](http://www.openslr.org/12)
+
+
+## 推理
+
+**推理前先下载所需权重文件**
+- 使用HF镜像中microsoft官方提供的模型权重文件
+- 官方下载地址：
+  - [tts模型权重下载地址](https://hf-mirror.com/microsoft/speecht5_tts)
+  - [vc模型权重下载地址](https://hf-mirror.com/microsoft/speecht5_vc)
+  - [asr模型权重下载地址](https://hf-mirror.com/microsoft/speecht5_asr)
+  - [hifigan模型权重下载地址](https://hf-mirror.com/microsoft/speecht5_hifigan)
+
+### TTS推理
+
+
+```
+python speech_tts.py -hip 7 -m model/tts -v model/hifigan -t "hi, nice to meet you." -s data/CMU_ARCTIC/cmu_us_awb_arctic-wav-arctic_a0001.npy
+```
+  - -hip： 显卡序号，默认为0。
+    - 当默认使用‘0’卡时：需要先export HIP_VISIBLE_DEVICES设置可见卡；
+    - 当不默认使用‘0’卡时
+  - -m： tts模型路径
+  - -v： 声码器hifigan的模型路径
+  - -t： 文本输入，因为输入文本中包含空格，需要用" "将输入文本包含在内。
+  - -s： CMU_ARCTIC数据集下的发声人特征文件
+  - -res： 结果输出文件tts.wav的存储路径
+
+### VC推理
+```
+python speech_vc.py -hip 7 -m model/speecht5_vc -v model/speecht5_hifigan -is data/librispeech/dev-clean/1272/128104/1272-128104-0000.flac -s data/CMU_ARCTIC/cmu_us_awb_arctic-wav-arctic_a0001.npy
+```
+  - -hip： 显卡序号，默认为0。
+    - 当默认使用‘0’卡时：需要先export HIP_VISIBLE_DEVICES设置可见卡；
+    - 当不默认使用‘0’卡时
+  - -m： vc模型路径
+  - -v： 声码器hifigan的模型路径
+  - -is：语音输入
+  - -s： CMU_ARCTIC数据集下的发声人特征文件
+  - -res： 结果输出文件tts.wav的存储路径
+
+### ASR推理
+```
+python speech_asr.py -hip 7 -m model/speecht5_asr -is data/librispeech/dev-clean/1272/128104/1272-128104-0000.flac
+```
+  - -hip： 显卡序号，默认为0。
+    - 当默认使用‘0’卡时：需要先export HIP_VISIBLE_DEVICES设置可见卡；
+    - 当不默认使用‘0’卡时
+  - -m： asr模型路径
+  - -is：语音输入
+  - -res： 结果输出文件tts.wav的存储路径
+
+## result
+
+### TTS
+  - 输入：“hi,nice to meet you”
+  - 输出：./res/tts.wav
+
+
+### VC
+  - 输入：./data/librispeech/dev-clean/1272/128104/1272-128104-0000.flac
+  - 输出：./res/vc.wav
+
+### ASR
+  - 输入：./data/librispeech/dev-clean/1272/128104/1272-128104-0000.flac
+  - 输出：./res/asr.txt
+
+## 应用场景
+
+### 算法分类
+语音类大模型
+
+### 热点应用行业
+金融，通信，广媒
+
+
+## 源码仓库及问题反馈
+
+https://developer.hpccube.com/codes/modelzoo/SpeechT5_pytorch
+
+## 参考
+
+[GitHub - microsoft/SpeechT5](https://github.com/microsoft/SpeechT5/tree/main/SpeechT5)
\ No newline at end of file

data/librispeech/dev-clean/1272/128104/1272-128104.trans.txt

+1272-128104-0000 MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL
+1272-128104-0001 NOR IS MISTER QUILTER'S MANNER LESS INTERESTING THAN HIS MATTER
+1272-128104-0002 HE TELLS US THAT AT THIS FESTIVE SEASON OF THE YEAR WITH CHRISTMAS AND ROAST BEEF LOOMING BEFORE US SIMILES DRAWN FROM EATING AND ITS RESULTS OCCUR MOST READILY TO THE MIND
+1272-128104-0003 HE HAS GRAVE DOUBTS WHETHER SIR FREDERICK LEIGHTON'S WORK IS REALLY GREEK AFTER ALL AND CAN DISCOVER IN IT BUT LITTLE OF ROCKY ITHACA
+1272-128104-0004 LINNELL'S PICTURES ARE A SORT OF UP GUARDS AND AT EM PAINTINGS AND MASON'S EXQUISITE IDYLLS ARE AS NATIONAL AS A JINGO POEM MISTER BIRKET FOSTER'S LANDSCAPES SMILE AT ONE MUCH IN THE SAME WAY THAT MISTER CARKER USED TO FLASH HIS TEETH AND MISTER JOHN COLLIER GIVES HIS SITTER A CHEERFUL SLAP ON THE BACK BEFORE HE SAYS LIKE A SHAMPOOER IN A TURKISH BATH NEXT MAN
+1272-128104-0005 IT IS OBVIOUSLY UNNECESSARY FOR US TO POINT OUT HOW LUMINOUS THESE CRITICISMS ARE HOW DELICATE IN EXPRESSION
+1272-128104-0006 ON THE GENERAL PRINCIPLES OF ART MISTER QUILTER WRITES WITH EQUAL LUCIDITY
+1272-128104-0007 PAINTING HE TELLS US IS OF A DIFFERENT QUALITY TO MATHEMATICS AND FINISH IN ART IS ADDING MORE FACT
+1272-128104-0008 AS FOR ETCHINGS THEY ARE OF TWO KINDS BRITISH AND FOREIGN
+1272-128104-0009 HE LAMENTS MOST BITTERLY THE DIVORCE THAT HAS BEEN MADE BETWEEN DECORATIVE ART AND WHAT WE USUALLY CALL PICTURES MAKES THE CUSTOMARY APPEAL TO THE LAST JUDGMENT AND REMINDS US THAT IN THE GREAT DAYS OF ART MICHAEL ANGELO WAS THE FURNISHING UPHOLSTERER
+1272-128104-0010 NEAR THE FIRE AND THE ORNAMENTS FRED BROUGHT HOME FROM INDIA ON THE MANTEL BOARD
+1272-128104-0011 IN FACT HE IS QUITE SEVERE ON MISTER RUSKIN FOR NOT RECOGNISING THAT A PICTURE SHOULD DENOTE THE FRAILTY OF MAN AND REMARKS WITH PLEASING COURTESY AND FELICITOUS GRACE THAT MANY PHASES OF FEELING
+1272-128104-0012 ONLY UNFORTUNATELY HIS OWN WORK NEVER DOES GET GOOD
+1272-128104-0013 MISTER QUILTER HAS MISSED HIS CHANCE FOR HE HAS FAILED EVEN TO MAKE HIMSELF THE TUPPER OF PAINTING
+1272-128104-0014 BY HARRY QUILTER M A