Skip to content

GitLab

Commit b75857fb authored by chenzk's avatar chenzk
Browse files

v1.0

parents
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 2177 additions and 0 deletions
docs/ko/index.md 0 → 100644
View file @ b75857fb
# 소개
<div>
<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
</a>
<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
</a>
<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
</a>
</div>
!!! warning
이 코드베이스의 불법적인 사용에 대해서는 책임을 지지 않습니다. DMCA(Digital Millennium Copyright Act) 및 해당 지역의 관련 법률을 참조하십시오. <br/>
이 코드베이스와 모든 모델은 CC-BY-NC-SA-4.0 라이선스에 따라 배포됩니다.
<p align="center">
<img src="../assets/figs/diagram.png" width="75%">
</p>
## 요구 사항
- GPU 메모리: 4GB (추론용), 8GB (파인튜닝용)
- 시스템: Linux, Windows
## Windows 설정
고급 Windows 사용자는 WSL2 또는 Docker를 사용하여 코드베이스를 실행하는 것을 고려할 수 있습니다.
```bash
# 파이썬 3.10 가상 환경 생성, virtualenv도 사용할 수 있습니다.
conda create -n fish-speech python=3.10
conda activate fish-speech
# pytorch 설치
pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
# fish-speech 설치
pip3 install -e .
# (가속 활성화) triton-windows 설치
pip install https://github.com/AnyaCoder/fish-speech/releases/download/v0.1.0/triton_windows-0.1.0-py3-none-any.whl
```
비전문 Windows 사용자는 Linux 환경 없이 프로젝트를 실행할 수 있는 다음 기본 방법을 고려할 수 있습니다 (모델 컴파일 기능 포함, 즉 `torch.compile`):
1. 프로젝트 패키지 추출.
2. `install_env.bat`을 클릭하여 환경 설치.
3. 컴파일 가속을 활성화하려면 아래 단계를 따르세요:
1. LLVM 컴파일러 다운로드:
- [LLVM-17.0.6 (공식 사이트)](https://huggingface.co/fishaudio/fish-speech-1/resolve/main/LLVM-17.0.6-win64.exe?download=true)
- [LLVM-17.0.6 (미러 사이트)](https://hf-mirror.com/fishaudio/fish-speech-1/resolve/main/LLVM-17.0.6-win64.exe?download=true)
- `LLVM-17.0.6-win64.exe`를 다운로드 후 더블클릭하여 설치하고, 설치 경로 선택 시 `Add Path to Current User` 옵션을 체크하여 환경 변수를 추가합니다.
- 설치가 완료되었는지 확인합니다.
2. Microsoft Visual C++ 재배포 가능 패키지를 다운로드하여 .dll 누락 문제 해결:
- [MSVC++ 14.40.33810.0 다운로드](https://aka.ms/vs/17/release/vc_redist.x64.exe)
3. Visual Studio Community Edition을 다운로드하여 LLVM의 헤더 파일 의존성을 해결:
- [Visual Studio 다운로드](https://visualstudio.microsoft.com/zh-hans/downloads/)
- Visual Studio Installer를 설치한 후 Visual Studio Community 2022를 다운로드.
- `Desktop development with C++` 옵션을 선택하여 설치.
4. [CUDA Toolkit 12.x](https://developer.nvidia.com/cuda-12-1-0-download-archive?target_os=Windows&target_arch=x86_64) 다운로드 및 설치.
4. `start.bat`을 더블 클릭하여 훈련 추론 WebUI 관리 인터페이스를 엽니다. 필요한 경우 아래 지침에 따라 `API_FLAGS`를 수정할 수 있습니다.
!!! info "Optional"
추론을 위해 WebUI를 사용하고자 하시나요?
프로젝트 루트 디렉토리의 `API_FLAGS.txt` 파일을 편집하고 첫 세 줄을 아래와 같이 수정하세요:
```
--infer
# --api
# --listen ...
...
```
!!! info "Optional"
API 서버를 시작하고 싶으신가요?
프로젝트 루트 디렉토리의 `API_FLAGS.txt` 파일을 편집하고 첫 세 줄을 아래와 같이 수정하세요:
```
# --infer
--api
--listen ...
...
```
!!! info "Optional"
`run_cmd.bat`을 더블 클릭하여 이 프로젝트의 conda/python 명령줄 환경에 진입할 수 있습니다.
## Linux 설정
[pyproject.toml](../../pyproject.toml)에서 자세한 내용을 확인하세요.
```bash
# 파이썬 3.10 가상 환경 생성, virtualenv도 사용할 수 있습니다.
conda create -n fish-speech python=3.10
conda activate fish-speech
# (Ubuntu / Debian 사용자) sox + ffmpeg 설치
apt install libsox-dev ffmpeg
# (Ubuntu / Debian 사용자) pyaudio 설치
apt install build-essential \
cmake \
libasound-dev \
portaudio19-dev \
libportaudio2 \
libportaudiocpp0
# pytorch 설치
pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
# fish-speech 설치
pip3 install -e .[stable]
```
## macos 설정
MPS에서 추론을 수행하려면 `--device mps` 플래그를 추가하세요.
추론 속도 비교는 [이 PR](https://github.com/fishaudio/fish-speech/pull/461#issuecomment-2284277772)을 참조하십시오.
!!! warning
Apple Silicon 장치에서는 `compile` 옵션이 공식적으로 지원되지 않으므로 추론 속도가 향상된다는 보장은 없습니다.
```bash
# 파이썬 3.10 가상 환경 생성, virtualenv도 사용할 수 있습니다.
conda create -n fish-speech python=3.10
conda activate fish-speech
# pytorch 설치
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
# fish-speech 설치
pip install -e .[stable]
```
## Docker 설정
1. NVIDIA Container Toolkit 설치:
Docker에서 모델 훈련 및 추론에 GPU를 사용하려면 NVIDIA Container Toolkit을 설치해야 합니다:
Ubuntu 사용자:
```bash
# 저장소 추가
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
# nvidia-container-toolkit 설치
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
# Docker 서비스 재시작
sudo systemctl restart docker
```
다른 Linux 배포판 사용자는: [NVIDIA Container Toolkit 설치 가이드](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)를 참조하십시오.
2. fish-speech 이미지 가져오기 및 실행
```bash
# 이미지 가져오기
docker pull fishaudio/fish-speech:latest-dev
# 이미지 실행
docker run -it \
--name fish-speech \
--gpus all \
-p 7860:7860 \
fishaudio/fish-speech:latest-dev \
zsh
# 다른 포트를 사용하려면 -p 매개변수를 YourPort:7860으로 수정하세요
```
3. 모델 종속성 다운로드
Docker 컨테이너 내부의 터미널에서 아래 명령어를 사용하여 필요한 `vqgan` 및 `llama` 모델을 Huggingface 리포지토리에서 다운로드합니다.
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
4. 환경 변수 설정 및 WebUI 접근
Docker 컨테이너 내부의 터미널에서 `export GRADIO_SERVER_NAME="0.0.0.0"`를 입력하여 Docker 내부에서 Gradio 서비스에 외부 접근을 허용합니다.
이후, 터미널에서 `python tools/run_webui.py` 명령어를 입력하여 WebUI 서비스를 시작합니다.
WSL 또는 macOS를 사용하는 경우 [http://localhost:7860](http://localhost:7860)에서 WebUI 인터페이스를 열 수 있습니다.
서버에 배포된 경우, localhost를 서버의 IP로 교체하세요.
## 변경 사항
- 2024/09/10: Fish-Speech 1.4 버전으로 업데이트, 데이터셋 크기 증가 및 양자화기의 n_groups를 4에서 8로 변경.
- 2024/07/02: Fish-Speech 1.2 버전으로 업데이트, VITS 디코더 제거 및 제로샷 능력 크게 향상.
- 2024/05/10: Fish-Speech 1.1 버전으로 업데이트, WER 감소 및 음색 유사성을 개선하기 위해 VITS 디코더 구현.
- 2024/04/22: Fish-Speech 1.0 버전 완료, VQGAN 및 LLAMA 모델 대폭 수정.
- 2023/12/28: `lora` 파인튜닝 지원 추가.
- 2023/12/27: `gradient checkpointing`, `causual sampling`, 및 `flash-attn` 지원 추가.
- 2023/12/19: WebUI 및 HTTP API 업데이트.
- 2023/12/18: 파인튜닝 문서 및 관련 예시 업데이트.
- 2023/12/17: `text2semantic` 모델 업데이트, 음소 없는 모드 지원.
- 2023/12/13: 베타 버전 출시, VQGAN 모델 및 LLAMA 기반 언어 모델(음소 지원만 포함).
## 감사의 말
- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
- [GPT VITS](https://github.com/innnky/gpt-vits)
- [MQTTS](https://github.com/b04901014/MQTTS)
- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
- [Transformers](https://github.com/huggingface/transformers)
- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)
docs/ko/inference.md 0 → 100644
View file @ b75857fb
# 추론
추론은 명령줄, HTTP API, 그리고 웹 UI에서 지원됩니다.
!!! note
전체 추론 과정은 다음의 여러 단계로 구성됩니다:
1. VQGAN을 사용하여 약 10초 분량의 음성을 인코딩합니다.
2. 인코딩된 시맨틱 토큰과 해당 텍스트를 예시로 언어 모델에 입력합니다.
3. 새로운 텍스트를 입력하면, 모델이 해당하는 시맨틱 토큰을 생성합니다.
4. 생성된 시맨틱 토큰을 VITS / VQGAN에 입력하여 음성을 디코딩하고 생성합니다.
## 모델 다운로드
필요한 `vqgan` 및 `llama` 모델을 Hugging Face 리포지토리에서 다운로드하세요.
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
## 명령줄 추론
### 1. 음성에서 프롬프트 생성:
!!! note
모델이 음색을 무작위로 선택하도록 하려면 이 단계를 건너뛸 수 있습니다.
!!! warning "향후 버전 경고"
원래 경로(tools/vqgan/infernce.py)에서 접근할 수 있는 인터페이스는 유지했지만, 이 인터페이스는 향후 몇몇 버전에서 삭제될 수 있습니다. 가능한 한 빨리 코드를 변경하십시오.
```bash
python fish_speech/models/vqgan/inference.py \
-i "paimon.wav" \
--checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
```
이 명령을 실행하면 `fake.npy` 파일을 얻게 됩니다.
### 2. 텍스트에서 시맨틱 토큰 생성:
!!! warning "향후 버전 경고"
원래 경로(tools/llama/generate.py)에서 접근할 수 있는 인터페이스는 유지했지만, 이 인터페이스는 향후 몇몇 버전에서 삭제될 수 있습니다. 가능한 한 빨리 코드를 변경하십시오.
```bash
python fish_speech/models/text2semantic/inference.py \
--text "변환할 텍스트" \
--prompt-text "참고할 텍스트" \
--prompt-tokens "fake.npy" \
--checkpoint-path "checkpoints/fish-speech-1.5" \
--num-samples 2 \
--compile
```
이 명령을 실행하면 작업 디렉토리에 `codes_N` 파일이 생성되며, N은 0부터 시작하는 정수입니다.
!!! note
빠른 추론을 위해 `--compile` 옵션을 사용하여 CUDA 커널을 결합할 수 있습니다 (~초당 30 토큰 -> ~초당 500 토큰).
`--compile` 매개변수를 주석 처리하여 가속화 옵션을 사용하지 않을 수도 있습니다.
!!! info
bf16을 지원하지 않는 GPU의 경우 `--half` 매개변수를 사용해야 할 수 있습니다.
### 3. 시맨틱 토큰에서 음성 생성:
#### VQGAN 디코더
!!! warning "향후 버전 경고"
원래 경로(tools/vqgan/infernce.py)에서 접근할 수 있는 인터페이스는 유지했지만, 이 인터페이스는 향후 몇몇 버전에서 삭제될 수 있습니다. 가능한 한 빨리 코드를 변경하십시오.
```bash
python fish_speech/models/vqgan/inference.py \
-i "codes_0.npy" \
--checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
```
## HTTP API 추론
추론을 위한 HTTP API를 제공하고 있습니다. 아래의 명령어로 서버를 시작할 수 있습니다:
```bash
python -m tools.api_server \
--listen 0.0.0.0:8080 \
--llama-checkpoint-path "checkpoints/fish-speech-1.5" \
--decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
--decoder-config-name firefly_gan_vq
```
추론 속도를 높이고 싶다면 `--compile` 매개변수를 추가할 수 있습니다.
이후, http://127.0.0.1:8080/ 에서 API를 확인하고 테스트할 수 있습니다.
아래는 `tools/api_client.py`를 사용하여 요청을 보내는 예시입니다.
```bash
python -m tools.api_client \
--text "입력할 텍스트" \
--reference_audio "참고 음성 경로" \
--reference_text "참고 음성의 텍스트 내용" \
--streaming True
```
위 명령은 참고 음성 정보를 바탕으로 원하는 음성을 합성하고, 스트리밍 방식으로 반환합니다.
다음 예시는 여러 개의 참고 음성 경로와 텍스트를 한꺼번에 사용할 수 있음을 보여줍니다. 명령에서 공백으로 구분하여 입력합니다.
```bash
python -m tools.api_client \
--text "입력할 텍스트" \
--reference_audio "참고 음성 경로1" "참고 음성 경로2" \
--reference_text "참고 음성 텍스트1" "참고 음성 텍스트2"\
--streaming False \
--output "generated" \
--format "mp3"
```
위 명령어는 여러 참고 음성 정보를 바탕으로 `MP3` 형식의 음성을 합성하여, 현재 디렉토리에 `generated.mp3`로 저장합니다.
`--reference_audio`와 `--reference_text` 대신에 `--reference_id`(하나만 사용 가능)를 사용할 수 있습니다. 프로젝트 루트 디렉토리에 `references/<your reference_id>` 폴더를 만들어 해당 음성과 주석 텍스트를 넣어야 합니다. 참고 음성은 최대 90초까지 지원됩니다.
!!! info
제공되는 파라미터는 `python -m tools.api_client -h`를 사용하여 확인할 수 있습니다.
## GUI 추론
[클라이언트 다운로드](https://github.com/AnyaCoder/fish-speech-gui/releases)
## WebUI 추론
다음 명령으로 WebUI를 시작할 수 있습니다:
```bash
python -m tools.run_webui \
--llama-checkpoint-path "checkpoints/fish-speech-1.5" \
--decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
--decoder-config-name firefly_gan_vq
```
> 추론 속도를 높이고 싶다면 `--compile` 매개변수를 추가할 수 있습니다.
!!! note
라벨 파일과 참고 음성 파일을 미리 메인 디렉토리의 `references` 폴더에 저장해 두면, WebUI에서 바로 호출할 수 있습니다. (해당 폴더는 직접 생성해야 합니다.)
!!! note
WebUI를 구성하기 위해 `GRADIO_SHARE`, `GRADIO_SERVER_PORT`, `GRADIO_SERVER_NAME`과 같은 Gradio 환경 변수를 사용할 수 있습니다.
즐기세요!
docs/ko/samples.md 0 → 100644
View file @ b75857fb
# 샘플
ver 1.4
## Credits
[Seed-TTS (2024)](https://bytedancespeech.github.io/seedtts_tech_report/)에 감사드리며, 평가 데이터를 제공해 주셔서 이 데모를 완성할 수 있었습니다.
모든 프롬프트 음성은 Seed-TTS 효과 데모 페이지에서 가져왔으며, 모든 생성된 음성은 fish-speech 버전 1.4에서 첫 번째로 생성된 것입니다.
## 제로샷 인컨텍스트 학습
- TODO: 한국어 제로샷 인컨텍스트 학습 샘플 추가. (현재는 영어와 중국어 데모만 제공됩니다.)
<table>
<thead>
<tr>
<th style="vertical-align : middle;text-align: center">언어</th>
<th style="vertical-align : middle;text-align: center">프롬프트</th>
<th style="vertical-align : middle;text-align: center">동일 언어 생성</th>
<th style="vertical-align : middle;text-align: center">교차 언어 생성</th>
</tr>
</thead>
<tbody>
<tr>
<td style="vertical-align : middle;text-align:center;" rowspan="3">EN</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/4245145269330795065.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/4245145269330795065/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>I don't really care what you call me. I've been a silent spectator, watching species evolve, empires rise and fall. But always remember, I am mighty and enduring. Respect me and I'll nurture you; ignore me and you shall face the consequences.</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/4245145269330795065/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。”</td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2486365921931244890.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2486365921931244890/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Dealing with family secrets is never easy. Yet, sometimes, omission is a form of protection, intending to safeguard some from the harsh truths. One day, I hope you understand the reasons behind my actions. Until then, Anna, please, bear with me.</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2486365921931244890/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>处理家庭秘密从来都不是一件容易的事。然而，有时候，隐瞒是一种保护形式，旨在保护一些人免受残酷的真相伤害。有一天，我希望你能理解我行为背后的原因。在那之前，安娜，请容忍我。</td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/-9102975986427238220.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/-9102975986427238220/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>The combinations of different textures and flavors create a perfect harmony. The succulence of the steak, the tartness of the cranberries, the crunch of pine nuts, and creaminess of blue cheese make it a truly delectable delight. Enjoy your culinary adventure!</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/-9102975986427238220/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>听着你的话，我心里五味杂陈。虽然我愿意一直在你身边，承担一切不幸，但我知道只有让你自己面对，才能真正让你变得更强大。所以，你要记得，无论面对何种困难，都请你坚强，我会在心里一直支持你的。</td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;" rowspan="3">ZH</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2648200402409733590.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2648200402409733590/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>突然，身边一阵笑声。我看着他们，意气风发地挺直了胸膛，甩了甩那稍显肉感的双臂，轻笑道："我身上的肉，是为了掩饰我爆棚的魅力，否则，岂不吓坏了你们呢？"</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2648200402409733590/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Suddenly, there was a burst of laughter beside me. I looked at them, stood up straight with high spirit, shook the slightly fleshy arms, and smiled lightly, saying, "The flesh on my body is to hide my bursting charm. Otherwise, wouldn't it scare you?"</td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/8913957783621352198.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/8913957783621352198/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>他闭上眼睛，期望这一切都能过去。然而，当他再次睁开眼睛，眼前的景象让他不禁倒吸一口气。雾气中出现的禁闭岛，陌生又熟悉，充满未知的危险。他握紧拳头，心知他的生活即将发生翻天覆地的改变。</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/8913957783621352198/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>He closed his eyes, expecting that all of this could pass. However, when he opened his eyes again, the sight in front of him made him couldn't help but take a deep breath. The closed island that appeared in the fog, strange and familiar, was full of unknown dangers. He tightened his fist, knowing that his life was about to undergo earth-shaking changes.</td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/prompts/2631296891109983590.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2631296891109983590/same-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>顿时，气氛变得沉郁起来。乍看之下，一切的困扰仿佛都围绕在我身边。我皱着眉头，感受着那份压力，但我知道我不能放弃，不能认输。于是，我深吸一口气，心底的声音告诉我：“无论如何，都要冷静下来，重新开始。”</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/zero-shot/2631296891109983590/cross-lang-fish.wav" autoplay="">Your browser does not support the audio element.</audio><br>Suddenly, the atmosphere became gloomy. At first glance, all the troubles seemed to surround me. I frowned, feeling that pressure, but I know I can't give up, can't admit defeat. So, I took a deep breath, and the voice in my heart told me, "Anyway, must calm down and start again."</td>
</tr>
</tbody>
</table>
## 화자 파인튜닝
<table>
<thead>
<tr>
<th style="text-align: center"> </th>
<th style="text-align: center">텍스트</th>
<th style="text-align: center">생성된 음성</th>
</tr>
</thead>
<tbody>
<tr>
<td style="vertical-align : middle;text-align:center;" rowspan="3">화자1</td>
<td style="vertical-align : middle;text-align:center;">好呀，哈哈哈哈哈，喜欢笑的人运气都不会差哦，希望你每天笑口常开~</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/prompts/4781135337205789117.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;">哇！恭喜你中了大乐透，八百万可真不少呢！有什么特别的计划或想法吗？</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/4781135337205789117/fish_1_to_2.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;">哼，你这么问是想请本小姐吃饭吗？如果对象是你的话，那也不是不可以。</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/4781135337205789117/fish_1_to_3.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;" rowspan="3">화자2</td>
<td style="vertical-align : middle;text-align:center;">是呀，他还想换个地球仪哈哈哈，看来给你积累了一些快乐值了，你还想不想再听一个其他的笑话呀？</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/prompts/-1325430967143158944.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;">嘿嘿，你是不是也想拥有甜甜的恋爱呢？《微微一笑很倾城》是你的不二选择，男女主是校花校草类型，他们通过游戏结识，再到两人见面，全程没有一点误会，真的齁甜，想想都忍不住“姨妈笑”~</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/-1325430967143158944/fish_1_to_2.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;">小傻瓜，嗯……算是个很可爱很亲切的名字，有点“独特”哦，不过我有些好奇，你为什么会给我选这个昵称呢？</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/fine-tune/-1325430967143158944/fish_1_to_3.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
</tbody>
</table>
<br>
## 콘텐츠 편집
<table>
<thead>
<tr><th style="text-align: center">언어</th>
<th style="text-align: center">원본 텍스트</th>
<th style="text-align: center">원본 음성</th>
<th style="text-align: center">목표 텍스트</th>
<th style="text-align: center">편집된 음성</th>
</tr></thead>
<tbody>
<tr>
<td style="vertical-align : middle;text-align:center;" rowspan="2">EN</td>
<td style="vertical-align : middle;text-align:center;">They can't order me to stop dreaming. If you dream a thing more than once, it's sure to come true. Have faith in your dreams, and someday your rainbow will come shining through.</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/2372076002032794455.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;">They can't <b>require</b> me to stop <b>imagining.</b> If you envision a thing more than once, it's <b>bound</b> to come <b>about</b>. Have <b>trust</b> in your <b>visions</b>, and someday your <b>radiance</b> will come <b>beaming</b> through.</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/2372076002032794455/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;">Are you familiar with it? Slice the steak and place the strips on top, then garnish with the dried cranberries, pine nuts, and blue cheese. I wonder how people rationalise the decision?</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/3347127306902202498.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;">Are you <b>acquainted</b> with it? <b>Cut the pork</b> and place the strips on top, then garnish with the dried <b>cherries, almonds,</b> and <b>feta</b> cheese. I <b>query</b> how people <b>justify</b> the <b>choice?</b></td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/3347127306902202498/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;" rowspan="2">ZH</td>
<td style="vertical-align : middle;text-align:center;">自古以来，庸君最怕党政了，可圣君他就不怕，不但不怕，反能利用。要我说，你就让明珠索额图互相争宠，只要你心里明白，左右逢源，你就能立于不败之地。</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/1297014176484007082.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;"><b>从古至今</b>，庸君最怕<b>朝纲了</b>，可<b>明</b>君他就不怕，不但不怕，反能<b>借助</b>。要我说，你就让<b>李四张三</b>互相争宠，只要你心里<b>清楚</b>，左右<b>周旋</b>，你就能<b>处</b>于不败之<b>境</b>。</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/1297014176484007082/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
<tr>
<td style="vertical-align : middle;text-align:center;">对，这就是我，万人敬仰的太乙真人，虽然有点婴儿肥，但也掩不住我逼人的帅气。</td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/prompts/-40165564411515767.wav" autoplay="">Your browser does not support the audio element.</audio></td>
<td style="vertical-align : middle;text-align:center;">对，这就是我，<b>众人尊崇</b>的太<b>白金星</b>，虽然有点<b>娃娃脸</b>，但也<b>遮</b>不住我<b>迷人</b>的<b>魅力。</b></td>
<td style="vertical-align : middle;text-align:center;"><audio controls="controls" style="width: 190px;"><source src="https://anyacoder.github.io/fishaudio.github.io/samples/content-edit/-40165564411515767/edit-fish.wav" autoplay="">Your browser does not support the audio element.</audio></td>
</tr>
</tbody>
</table>
docs/ko/start_agent.md 0 → 100644
View file @ b75857fb
# 에이전트 시작하기
!!! note
전체 문서는 claude3.5 Sonnet에 의해 번역되었으며, 원어민인 경우 번역에 문제가 있다고 생각되면 이슈나 풀 리퀘스트를 보내주셔서 대단히 감사합니다!
## 요구사항
- GPU 메모리: 최소 8GB(양자화 사용 시), 16GB 이상 권장
- 디스크 사용량: 10GB
## 모델 다운로드
다음 명령어로 모델을 받을 수 있습니다:
```bash
huggingface-cli download fishaudio/fish-agent-v0.1-3b --local-dir checkpoints/fish-agent-v0.1-3b
```
'checkpoints' 폴더에 파일들을 넣으세요.
또한 [inference](inference.md)에 설명된 대로 fish-speech 모델도 다운로드해야 합니다.
checkpoints에는 2개의 폴더가 있어야 합니다.
`checkpoints/fish-speech-1.4`와 `checkpoints/fish-agent-v0.1-3b`입니다.
## 환경 준비
이미 Fish-speech가 있다면 다음 명령어를 추가하여 바로 사용할 수 있습니다:
```bash
pip install cachetools
```
!!! 참고
컴파일을 위해 Python 3.12 미만 버전을 사용해 주세요.
없다면 아래 명령어를 사용하여 환경을 구축하세요:
```bash
sudo apt-get install portaudio19-dev
pip install -e .[stable]
```
## 에이전트 데모 실행
fish-agent를 구축하려면 메인 폴더에서 아래 명령어를 사용하세요:
```bash
python -m tools.api_server --llama-checkpoint-path checkpoints/fish-agent-v0.1-3b/ --mode agent --compile
```
`--compile` 인자는 Python < 3.12에서만 지원되며, 토큰 생성 속도를 크게 향상시킵니다.
한 번에 컴파일되지 않습니다(기억해 두세요).
그런 다음 다른 터미널을 열고 다음 명령어를 사용하세요:
```bash
python -m tools.e2e_webui
```
이렇게 하면 기기에 Gradio WebUI가 생성됩니다.
모델을 처음 사용할 때는 (`--compile`이 True인 경우) 잠시 컴파일이 진행되므로 기다려 주세요.
## Gradio Webui
<p align="center">
<img src="../../assets/figs/agent_gradio.png" width="75%">
</p>
즐거운 시간 되세요!
## 성능
테스트 결과, 4060 노트북은 겨우 실행되며 매우 부하가 큰 상태로, 초당 약 8토큰 정도만 처리합니다. 4090은 컴파일 상태에서 초당 약 95토큰을 처리하며, 이것이 저희가 권장하는 사양입니다.
# 에이전트 소개
이 데모는 초기 알파 테스트 버전으로, 추론 속도 최적화가 필요하며 수정해야 할 버그가 많이 있습니다. 버그를 발견하거나 수정하고 싶으시다면 이슈나 풀 리퀘스트를 보내주시면 매우 감사하겠습니다.
docs/pt/finetune.md 0 → 100644
View file @ b75857fb
# Ajuste Fino
É óbvio que ao abrir esta página, você não deve estar muito satisfeito com o desempenho do modelo pré-treinado com poucos exemplos. Você pode querer ajustar o modelo para melhorar seu desempenho em seu conjunto de dados.
Na atual versão, a única coisa que você precisa ajustar é a parte do 'LLAMA'.
## Ajuste Fino do LLAMA
### 1. Preparando o conjunto de dados
```
.
├── SPK1
│ ├── 21.15-26.44.lab
│ ├── 21.15-26.44.mp3
│ ├── 27.51-29.98.lab
│ ├── 27.51-29.98.mp3
│ ├── 30.1-32.71.lab
│ └── 30.1-32.71.mp3
└── SPK2
├── 38.79-40.85.lab
└── 38.79-40.85.mp3
```
Você precisa converter seu conjunto de dados para o formato acima e colocá-lo em `data`. O arquivo de áudio pode ter as extensões `.mp3`, `.wav` ou `.flac`, e o arquivo de anotação deve ter a extensão `.lab`.
!!! info
O arquivo de anotação `.lab` deve conter apenas a transcrição do áudio, sem a necessidade de formatação especial. Por exemplo, se o arquivo `hi.mp3` disser "Olá, tchau", o arquivo `hi.lab` conterá uma única linha de texto: "Olá, tchau".
!!! warning
É recomendado aplicar normalização de volume ao conjunto de dados. Você pode usar o [fish-audio-preprocess](https://github.com/fishaudio/audio-preprocess) para fazer isso.
```bash
fap loudness-norm data-raw data --clean
```
### 2. Extração em lote de tokens semânticos
Certifique-se de ter baixado os pesos do VQGAN. Se não, execute o seguinte comando:
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
Em seguida, você pode executar o seguinte comando para extrair os tokens semânticos:
```bash
python tools/vqgan/extract_vq.py data \
--num-workers 1 --batch-size 16 \
--config-name "firefly_gan_vq" \
--checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
```
!!! note
Você pode ajustar `--num-workers` e `--batch-size` para aumentar a velocidade de extração, mas certifique-se de não exceder o limite de memória da sua GPU.  
Para o formato VITS, você pode especificar uma lista de arquivos usando `--filelist xxx.list`.
Este comando criará arquivos `.npy` no diretório `data`, como mostrado abaixo:
```
.
├── SPK1
│ ├── 21.15-26.44.lab
│ ├── 21.15-26.44.mp3
│ ├── 21.15-26.44.npy
│ ├── 27.51-29.98.lab
│ ├── 27.51-29.98.mp3
│ ├── 27.51-29.98.npy
│ ├── 30.1-32.71.lab
│ ├── 30.1-32.71.mp3
│ └── 30.1-32.71.npy
└── SPK2
├── 38.79-40.85.lab
├── 38.79-40.85.mp3
└── 38.79-40.85.npy
```
### 3. Empacotar o conjunto de dados em protobuf
```bash
python tools/llama/build_dataset.py \
--input "data" \
--output "data/protos" \
--text-extension .lab \
--num-workers 16
```
Após executar o comando, você deverá ver o arquivo `quantized-dataset-ft.protos` no diretório `data`.
### 4. E finalmente, chegamos ao ajuste fino com LoRA
Da mesma forma, certifique-se de ter baixado os pesos do `LLAMA`. Se não, execute o seguinte comando:
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
E então, execute o seguinte comando para iniciar o ajuste fino:
```bash
python fish_speech/train.py --config-name text2semantic_finetune \
project=$project \
+lora@model.model.lora_config=r_8_alpha_16
```
!!! note
Se quiser, você pode modificar os parâmetros de treinamento, como `batch_size`, `gradient_accumulation_steps`, etc., para se ajustar à memória da sua GPU, modificando `fish_speech/configs/text2semantic_finetune.yaml`.
!!! note
Para usuários do Windows, é recomendado usar `trainer.strategy.process_group_backend=gloo` para evitar problemas com `nccl`.
Após concluir o treinamento, consulte a seção [inferência](inference.md).
!!! info
Por padrão, o modelo aprenderá apenas os padrões de fala do orador e não o timbre. Ainda pode ser preciso usar prompts para garantir a estabilidade do timbre.
Se quiser que ele aprenda o timbre, aumente o número de etapas de treinamento, mas isso pode levar ao overfitting (sobreajuste).
Após o treinamento, é preciso converter os pesos do LoRA em pesos regulares antes de realizar a inferência.
```bash
python tools/llama/merge_lora.py \
--lora-config r_8_alpha_16 \
--base-weight checkpoints/fish-speech-1.5 \
--lora-weight results/$project/checkpoints/step_000000010.ckpt \
--output checkpoints/fish-speech-1.5-yth-lora/
```
!!! note
É possível também tentar outros checkpoints. Sugerimos usar o checkpoint que melhor atenda aos seus requisitos, pois eles geralmente têm um desempenho melhor em dados fora da distribuição (OOD).
docs/pt/index.md 0 → 100644
View file @ b75857fb
# Introdução
<div>
<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
</a>
<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
</a>
<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
</a>
</div>
!!! warning
Não nos responsabilizamos por qualquer uso ilegal do código-fonte. Consulte as leis locais sobre DMCA (Digital Millennium Copyright Act) e outras leis relevantes em sua região. <br/>
Este repositório de código e os modelos são distribuídos sob a licença CC-BY-NC-SA-4.0.
<p align="center">
<img src="../assets/figs/diagram.png" width="75%">
</p>
## Requisitos
- Memória da GPU: 4GB (para inferência), 8GB (para ajuste fino)
- Sistema: Linux, Windows
## Configuração do Windows
Usuários profissionais do Windows podem considerar o uso do WSL2 ou Docker para executar a base de código.
```bash
# Crie um ambiente virtual Python 3.10, também é possível usar o virtualenv
conda create -n fish-speech python=3.10
conda activate fish-speech
# Instale o pytorch
pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
# Instale o fish-speech
pip3 install -e .
# (Ativar aceleração) Instalar triton-windows
pip install https://github.com/AnyaCoder/fish-speech/releases/download/v0.1.0/triton_windows-0.1.0-py3-none-any.whl
```
Usuários não profissionais do Windows podem considerar os seguintes métodos básicos para executar o projeto sem um ambiente Linux (com capacidades de compilação de modelo, ou seja, `torch.compile`):
1. Extraia o pacote do projeto.
2. Clique em `install_env.bat` para instalar o ambiente.
3. Se você quiser ativar a aceleração de compilação, siga estas etapas:
1. Baixe o compilador LLVM nos seguintes links:
- [LLVM-17.0.6 (Download do site oficial)](https://huggingface.co/fishaudio/fish-speech-1/resolve/main/LLVM-17.0.6-win64.exe?download=true)
- [LLVM-17.0.6 (Download do site espelho)](https://hf-mirror.com/fishaudio/fish-speech-1/resolve/main/LLVM-17.0.6-win64.exe?download=true)
- Após baixar o `LLVM-17.0.6-win64.exe`, clique duas vezes para instalar, selecione um local de instalação apropriado e, o mais importante, marque a opção `Add Path to Current User` para adicionar a variável de ambiente.
- Confirme que a instalação foi concluída.
2. Baixe e instale o Microsoft Visual C++ Redistributable para resolver possíveis problemas de arquivos .dll ausentes:
- [Download do MSVC++ 14.40.33810.0](https://aka.ms/vs/17/release/vc_redist.x64.exe)
3. Baixe e instale o Visual Studio Community Edition para obter as ferramentas de compilação do MSVC++ e resolver as dependências dos arquivos de cabeçalho do LLVM:
- [Download do Visual Studio](https://visualstudio.microsoft.com/pt-br/downloads/)
- Após instalar o Visual Studio Installer, baixe o Visual Studio Community 2022.
- Conforme mostrado abaixo, clique no botão `Modificar`, encontre a opção `Desenvolvimento de área de trabalho com C++` e selecione para fazer o download.
4. Baixe e instale o [CUDA Toolkit 12.x](https://developer.nvidia.com/cuda-12-1-0-download-archive?target_os=Windows&target_arch=x86_64)
4. Clique duas vezes em `start.bat` para abrir a interface de gerenciamento WebUI de inferência de treinamento. Se necessário, você pode modificar as `API_FLAGS` conforme mostrado abaixo.
!!! info "Opcional"
Você quer iniciar o WebUI de inferência?
Edite o arquivo `API_FLAGS.txt` no diretório raiz do projeto e modifique as três primeiras linhas como segue:
```
--infer
# --api
# --listen ...
...
```
!!! info "Opcional"
Você quer iniciar o servidor de API?
Edite o arquivo `API_FLAGS.txt` no diretório raiz do projeto e modifique as três primeiras linhas como segue:
```
# --infer
--api
--listen ...
...
```
!!! info "Opcional"
Clique duas vezes em `run_cmd.bat` para entrar no ambiente de linha de comando conda/python deste projeto.
## Configuração para Linux
Para mais detalhes, consulte [pyproject.toml](../../pyproject.toml).
```bash
# Crie um ambiente virtual python 3.10, você também pode usar virtualenv
conda create -n fish-speech python=3.10
conda activate fish-speech
# Instale o pytorch
pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
# Para os Usuário do Ubuntu / Debian: Instale o sox + ffmpeg
apt install libsox-dev ffmpeg
# Para os Usuário do Ubuntu / Debian: Instale o pyaudio
apt install build-essential \
cmake \
libasound-dev \
portaudio19-dev \
libportaudio2 \
libportaudiocpp0
# Instale o fish-speech
pip3 install -e .[stable]
```
## Configuração para macos
Se você quiser realizar inferências no MPS, adicione a flag `--device mps`.
Para uma comparação das velocidades de inferência, consulte [este PR](https://github.com/fishaudio/fish-speech/pull/461#issuecomment-2284277772).
!!! aviso
A opção `compile` não é oficialmente suportada em dispositivos Apple Silicon, então não há garantia de que a velocidade de inferência irá melhorar.
```bash
# create a python 3.10 virtual environment, you can also use virtualenv
conda create -n fish-speech python=3.10
conda activate fish-speech
# install pytorch
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
# install fish-speech
pip install -e .[stable]
```
## Configuração do Docker
1. Instale o NVIDIA Container Toolkit:
Para usar a GPU com Docker para treinamento e inferência de modelos, você precisa instalar o NVIDIA Container Toolkit:
Para usuários Ubuntu:
```bash
# Adicione o repositório remoto
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
# Instale o nvidia-container-toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
# Reinicie o serviço Docker
sudo systemctl restart docker
```
Para usuários de outras distribuições Linux, consulte o guia de instalação: [NVIDIA Container Toolkit Install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
2. Baixe e execute a imagem fish-speech
```shell
# Baixe a imagem
docker pull fishaudio/fish-speech:latest-dev
# Execute a imagem
docker run -it \
--name fish-speech \
--gpus all \
-p 7860:7860 \
fishaudio/fish-speech:latest-dev \
zsh
# Se precisar usar outra porta, modifique o parâmetro -p para YourPort:7860
```
3. Baixe as dependências do modelo
Certifique-se de estar no terminal do contêiner Docker e, em seguida, baixe os modelos necessários `vqgan` e `llama` do nosso repositório HuggingFace.
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
4. Configure as variáveis de ambiente e acesse a WebUI
No terminal do contêiner Docker, digite `export GRADIO_SERVER_NAME="0.0.0.0"` para permitir o acesso externo ao serviço gradio dentro do Docker.
Em seguida, no terminal do contêiner Docker, digite `python tools/run_webui.py` para iniciar o serviço WebUI.
Se estiver usando WSL ou MacOS, acesse [http://localhost:7860](http://localhost:7860) para abrir a interface WebUI.
Se estiver implantando em um servidor, substitua localhost pelo IP do seu servidor.
## Histórico de Alterações
- 10/09/2024: Fish-Speech atualizado para a versão 1.4, aumentado o tamanho do conjunto de dados, quantizer n_groups 4 -> 8.
- 02/07/2024: Fish-Speech atualizado para a versão 1.2, removido o Decodificador VITS e aprimorado consideravelmente a capacidade de zero-shot.
- 10/05/2024: Fish-Speech atualizado para a versão 1.1, implementado o decodificador VITS para reduzir a WER e melhorar a similaridade de timbre.
- 22/04/2024: Finalizada a versão 1.0 do Fish-Speech, modificados significativamente os modelos VQGAN e LLAMA.
- 28/12/2023: Adicionado suporte para ajuste fino `lora`.
- 27/12/2023: Adicionado suporte para `gradient checkpointing`, `causual sampling` e `flash-attn`.
- 19/12/2023: Atualizada a interface web e a API HTTP.
- 18/12/2023: Atualizada a documentação de ajuste fino e exemplos relacionados.
- 17/12/2023: Atualizado o modelo `text2semantic`, suportando o modo sem fonemas.
- 13/12/2023: Versão beta lançada, incluindo o modelo VQGAN e um modelo de linguagem baseado em LLAMA (suporte apenas a fonemas).
## Agradecimentos
- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
- [GPT VITS](https://github.com/innnky/gpt-vits)
- [MQTTS](https://github.com/b04901014/MQTTS)
- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
- [Transformers](https://github.com/huggingface/transformers)
- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)
docs/pt/inference.md 0 → 100644
View file @ b75857fb
# Inferência
Suporte para inferência por linha de comando, API HTTP e interface web (WebUI).
!!! note
O processo de raciocínio, em geral, consiste em várias partes:
1. Codificar cerca de 10 segundos de voz usando VQGAN.
2. Inserir os tokens semânticos codificados e o texto correspondente no modelo de linguagem como um exemplo.
3. Dado um novo trecho de texto, fazer com que o modelo gere os tokens semânticos correspondentes.
4. Inserir os tokens semânticos gerados no VITS / VQGAN para decodificar e gerar a voz correspondente.
## Baixar modelos
Baixe os modelos `vqgan` e `llama` necessários do nosso repositório Hugging Face.
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
## Inferência por Linha de Comando
### 1. Gerar prompt a partir da voz:
!!! note
Se quiser permitir que o modelo escolha aleatoriamente um timbre de voz, pule esta etapa.
!!! warning "Aviso de Versão Futura"
Mantivemos a interface acessível a partir do caminho original (tools/vqgan/infernce.py), mas esta interface poderá ser removida em algumas versões futuras. Por favor, altere o seu código o mais breve possível.
```bash
python fish_speech/models/vqgan/inference.py \
-i "paimon.wav" \
--checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
```
Você deverá obter um arquivo `fake.npy`.
### 2. Gerar tokens semânticos a partir do texto:
!!! warning "Aviso de Versão Futura"
Mantivemos a interface acessível a partir do caminho original (tools/llama/generate.py), mas esta interface poderá ser removida em algumas versões futuras. Por favor, altere o seu código o mais breve possível.
```bash
python fish_speech/models/text2semantic/inference.py \
--text "O texto que você deseja converter" \
--prompt-text "Seu texto de referência" \
--prompt-tokens "fake.npy" \
--checkpoint-path "checkpoints/fish-speech-1.5" \
--num-samples 2 \
--compile
```
Este comando criará um arquivo `codes_N` no diretório de trabalho, onde N é um número inteiro começando de 0.
!!! note
Use `--compile` para fundir kernels CUDA para ter uma inferência mais rápida (~30 tokens/segundo -> ~500 tokens/segundo).
Mas, se não planeja usar a aceleração CUDA, comente o parâmetro `--compile`.
!!! info
Para GPUs que não suportam bf16, pode ser necessário usar o parâmetro `--half`.
### 3. Gerar vocais a partir de tokens semânticos:
#### Decodificador VQGAN
!!! warning "Aviso de Versão Futura"
Mantivemos a interface acessível a partir do caminho original (tools/vqgan/infernce.py), mas esta interface poderá ser removida em algumas versões futuras. Por favor, altere o seu código o mais breve possível.
```bash
python fish_speech/models/vqgan/inference.py \
-i "codes_0.npy" \
--checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
```
## Inferência por API HTTP
Fornecemos uma API HTTP para inferência. O seguinte comando pode ser usado para iniciar o servidor:
```bash
python -m tools.api_server \
--listen 0.0.0.0:8080 \
--llama-checkpoint-path "checkpoints/fish-speech-1.5" \
--decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
--decoder-config-name firefly_gan_vq
```
> Para acelerar a inferência, adicione o parâmetro `--compile`.
Depois disso, é possível visualizar e testar a API em http://127.0.0.1:8080/.
Abaixo está um exemplo de envio de uma solicitação usando `tools/api_client.py`.
```bash
python -m tools.api_client \
--text "Texto a ser inserido" \
--reference_audio "Caminho para o áudio de referência" \
--reference_text "Conteúdo de texto do áudio de referência" \
--streaming True
```
O comando acima indica a síntese do áudio desejada de acordo com as informações do áudio de referência e a retorna em modo de streaming.
!!! info
Para aprender mais sobre parâmetros disponíveis, você pode usar o comando `python -m tools.api_client -h`
## Inferência por WebUI
Para iniciar a WebUI de Inferência execute o seguinte comando:
```bash
python -m tools.run_webui \
--llama-checkpoint-path "checkpoints/fish-speech-1.5" \
--decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
--decoder-config-name firefly_gan_vq
```
> Para acelerar a inferência, adicione o parâmetro `--compile`.
!!! note
Você pode salvar antecipadamente o arquivo de rótulos e o arquivo de áudio de referência na pasta `references` do diretório principal (que você precisa criar), para que possa chamá-los diretamente na WebUI.
!!! note
É possível usar variáveis de ambiente do Gradio, como `GRADIO_SHARE`, `GRADIO_SERVER_PORT`, `GRADIO_SERVER_NAME`, para configurar a WebUI.
Divirta-se!
docs/pt/samples.md 0 → 100644
View file @ b75857fb
# Amostras
A demonstração da versão 1.4 foi atualizada [aqui](https://speech.fish.audio/samples/)
As amostras da v1.2 estão disponíveis em [Bilibili](https://www.bilibili.com/video/BV1wz421B71D/).
As seguintes amostras são do modelo v1.1.
## Frase em Chinês 1
```
人间灯火倒映湖中，她的渴望让静水泛起涟漪。若代价只是孤独，那就让这份愿望肆意流淌。
流入她所注视的世间，也流入她如湖水般澄澈的目光。
```
<table>
<thead>
<tr>
<th>Orador</th>
<th>Áudio de Entrada</th>
<th>Áudio Sintetizado</th>
</tr>
</thead>
<tbody>
<tr>
<td>Nahida (Genshin Impact)</td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_output.wav" /></td>
</tr>
<tr>
<td>Zhongli (Genshin Impact)</td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_input.wav" /></td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_output.wav" /></td>
</tr>
<tr>
<td>Furina (Genshin Impact)</td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_input.wav" /></td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_output.wav" /></td>
</tr>
<tr>
<td>Orador Aleatório 1</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/4_output.wav" /></td>
</tr>
<tr>
<td>Orador Aleatório 2</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/5_output.wav" /></td>
</tr>
</tbody>
</table>
## Frase em Chinês 2
```
你们这个是什么群啊，你们这是害人不浅啊你们这个群！谁是群主，出来！真的太过分了。你们搞这个群干什么？
我儿子每一科的成绩都不过那个平均分呐，他现在初二，你叫我儿子怎么办啊？他现在还不到高中啊？
你们害死我儿子了！快点出来你这个群主！再这样我去报警了啊！我跟你们说你们这一帮人啊，一天到晚啊，
搞这些什么游戏啊，动漫啊，会害死你们的，你们没有前途我跟你说。你们这九百多个人，好好学习不好吗？
一天到晚在上网。有什么意思啊？麻烦你重视一下你们的生活的目标啊？有一点学习目标行不行？一天到晚上网是不是人啊？
```
<table>
<thead>
<tr>
<th>Orador</th>
<th>Áudio de Entrada</th>
<th>Áudio Sintetizado</th>
</tr>
</thead>
<tbody>
<tr>
<td>Nahida (Genshin Impact)</td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/6_output.wav" /></td>
</tr>
<tr>
<td>Orador Aleatório</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/7_output.wav" /></td>
</tr>
</tbody>
</table>
## Frase em Chinês 3
```
大家好，我是 Fish Audio 开发的开源文本转语音模型。经过十五万小时的数据训练，
我已经能够熟练掌握中文、日语和英语，我的语言处理能力接近人类水平，声音表现形式丰富多变。
作为一个仅有亿级参数的模型，我相信社区成员能够在个人设备上轻松运行和微调，让我成为您的私人语音助手。
```
<table>
<thead>
<tr>
<th>Orador</th>
<th>Áudio de Entrada</th>
<th>Áudio Sintetizado</th>
</tr>
</thead>
<tbody>
<tr>
<td>Orador Aleatório</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/8_output.wav" /></td>
</tr>
</tbody>
</table>
## Frase em Inglês 1
```
In the realm of advanced technology, the evolution of artificial intelligence stands as a
monumental achievement. This dynamic field, constantly pushing the boundaries of what
machines can do, has seen rapid growth and innovation. From deciphering complex data
patterns to driving cars autonomously, AI's applications are vast and diverse.
```
<table>
<thead>
<tr>
<th>Orador</th>
<th>Áudio de Entrada</th>
<th>Áudio Sintetizado</th>
</tr>
</thead>
<tbody>
<tr>
<td>Orador Aleatório 1</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/0_output.wav" /></td>
</tr>
<tr>
<td>Orador Aleatório 2</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/1_output.wav" /></td>
</tr>
</tbody>
</table>
## Frase em Inglês 2
```
Hello everyone, I am an open-source text-to-speech model developed by
Fish Audio. After training with 150,000 hours of data, I have become proficient
in Chinese, Japanese, and English, and my language processing abilities
are close to human level. My voice is capable of a wide range of expressions.
As a model with only hundreds of millions of parameters, I believe community
members can easily run and fine-tune me on their personal devices, allowing
me to serve as your personal voice assistant.
```
<table>
<thead>
<tr>
<th>Orador</th>
<th>Áudio de Entrada</th>
<th>Áudio Sintetizado</th>
</tr>
</thead>
<tbody>
<tr>
<td>Orador Aleatório</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/2_output.wav" /></td>
</tr>
</tbody>
</table>
## Frase em Japonês 1
```
先進技術の領域において、人工知能の進化は画期的な成果として立っています。常に機械ができることの限界を
押し広げているこのダイナミックな分野は、急速な成長と革新を見せています。複雑なデータパターンの解読か
ら自動運転車の操縦まで、AIの応用は広範囲に及びます。
```
<table>
<thead>
<tr>
<th>Orador</th>
<th>Áudio de Entrada</th>
<th>Áudio Sintetizado</th>
</tr>
</thead>
<tbody>
<tr>
<td>Orador Aleatório 1</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/0_output.wav" /></td>
</tr>
<tr>
<td>Orador Aleatório 2</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/1_output.wav" /></td>
</tr>
</tbody>
</table>
## Frase em Japonês 2
```
皆さん、こんにちは。私はフィッシュオーディオによって開発されたオープンソースのテ
キストから音声への変換モデルです。15万時間のデータトレーニングを経て、
中国語、日本語、英語を熟知しており、言語処理能力は人間に近いレベルです。
声の表現も多彩で豊かです。数億のパラメータを持つこのモデルは、コミュニティ
のメンバーが個人のデバイスで簡単に実行し、微調整することができると
信じています。これにより、私を個人の音声アシスタントとして活用できます。
```
<table>
<thead>
<tr>
<th>Orador</th>
<th>Áudio de Entrada</th>
<th>Áudio Sintetizado</th>
</tr>
</thead>
<tbody>
<tr>
<td>Orador Aleatório</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/2_output.wav" /></td>
</tr>
</tbody>
</table>
docs/pt/start_agent.md 0 → 100644
View file @ b75857fb
# Iniciar Agente
!!! note
Todo o documento foi traduzido por claude3.5 Sonnet, se você for um falante nativo e achar a tradução problemática, muito obrigado por nos enviar um problema ou uma solicitação pull!
## Requisitos
- Memória GPU: No mínimo 8GB (com quantização), 16GB ou mais é recomendado.
- Uso de disco: 10GB
## Download do Modelo
Você pode obter o modelo através de:
```bash
huggingface-cli download fishaudio/fish-agent-v0.1-3b --local-dir checkpoints/fish-agent-v0.1-3b
```
Coloque-os na pasta 'checkpoints'.
Você também precisará do modelo fish-speech que pode ser baixado seguindo as instruções em [inference](inference.md).
Então haverá 2 pastas em checkpoints.
O `checkpoints/fish-speech-1.4` e `checkpoints/fish-agent-v0.1-3b`
## Preparação do Ambiente
Se você já tem o Fish-speech, pode usar diretamente adicionando a seguinte instrução:
```bash
pip install cachetools
```
!!! nota
Por favor, use a versão Python abaixo de 3.12 para compilação.
Se você não tem, use os comandos abaixo para construir seu ambiente:
```bash
sudo apt-get install portaudio19-dev
pip install -e .[stable]
```
## Iniciar a Demo do Agente
Para construir o fish-agent, use o comando abaixo na pasta principal:
```bash
python -m tools.api_server --llama-checkpoint-path checkpoints/fish-agent-v0.1-3b/ --mode agent --compile
```
O argumento `--compile` só suporta Python < 3.12, o que aumentará muito a velocidade de geração de tokens.
Não será compilado de uma vez (lembre-se).
Então abra outro terminal e use o comando:
```bash
python -m tools.e2e_webui
```
Isso criará uma WebUI Gradio no dispositivo.
Quando você usar o modelo pela primeira vez, ele irá compilar (se `--compile` estiver True) por um curto período, então aguarde com paciência.
## Gradio Webui
<p align="center">
<img src="../../assets/figs/agent_gradio.png" width="75%">
</p>
Divirta-se!
## Desempenho
Em nossos testes, um laptop com 4060 mal consegue rodar, ficando muito sobrecarregado, gerando apenas cerca de 8 tokens/s. A 4090 gera cerca de 95 tokens/s com compilação, que é o que recomendamos.
# Sobre o Agente
A demo é uma versão alpha inicial de teste, a velocidade de inferência precisa ser otimizada, e há muitos bugs aguardando correção. Se você encontrou um bug ou quer corrigi-lo, ficaremos muito felizes em receber uma issue ou um pull request.
docs/requirements.txt 0 → 100644
View file @ b75857fb
mkdocs-material
mkdocs-static-i18n[material]
mkdocs[i18n]
docs/stylesheets/extra.css 0 → 100644
View file @ b75857fb
.md-grid {
max-width: 1440px;
}
docs/zh/finetune.md 0 → 100644
View file @ b75857fb
# 微调
显然, 当你打开这个页面的时候, 你已经对预训练模型 zero-shot 的效果不算满意. 你想要微调一个模型, 使得它在你的数据集上表现更好.
在目前版本，你只需要微调'LLAMA'部分即可.
## LLAMA 微调
### 1. 准备数据集
```
.
├── SPK1
│ ├── 21.15-26.44.lab
│ ├── 21.15-26.44.mp3
│ ├── 27.51-29.98.lab
│ ├── 27.51-29.98.mp3
│ ├── 30.1-32.71.lab
│ └── 30.1-32.71.mp3
└── SPK2
├── 38.79-40.85.lab
└── 38.79-40.85.mp3
```
你需要将数据集转为以上格式, 并放到 `data` 下, 音频后缀可以为 `.mp3`, `.wav` 或 `.flac`, 标注文件后缀建议为 `.lab`.
!!! info
标注文件 `.lab` 仅需包含音频的转写文本，无需遵循特殊格式要求。例如，如果 `hi.mp3` 中的内容是“你好，再见。”，那么 `hi.lab` 文件中只需包含一行文本：“你好，再见”。
!!! warning
建议先对数据集进行响度匹配, 你可以使用 [fish-audio-preprocess](https://github.com/fishaudio/audio-preprocess) 来完成这一步骤.
```bash
fap loudness-norm data-raw data --clean
```
### 2. 批量提取语义 token
确保你已经下载了 vqgan 权重, 如果没有, 请运行以下命令:
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
对于中国大陆用户, 可使用 mirror 下载.
```bash
HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
随后可运行以下命令来提取语义 token:
```bash
python tools/vqgan/extract_vq.py data \
--num-workers 1 --batch-size 16 \
--config-name "firefly_gan_vq" \
--checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
```
!!! note
你可以调整 `--num-workers` 和 `--batch-size` 来提高提取速度, 但是请注意不要超过你的显存限制.
该命令会在 `data` 目录下创建 `.npy` 文件, 如下所示:
```
.
├── SPK1
│ ├── 21.15-26.44.lab
│ ├── 21.15-26.44.mp3
│ ├── 21.15-26.44.npy
│ ├── 27.51-29.98.lab
│ ├── 27.51-29.98.mp3
│ ├── 27.51-29.98.npy
│ ├── 30.1-32.71.lab
│ ├── 30.1-32.71.mp3
│ └── 30.1-32.71.npy
└── SPK2
├── 38.79-40.85.lab
├── 38.79-40.85.mp3
└── 38.79-40.85.npy
```
### 3. 打包数据集为 protobuf
```bash
python tools/llama/build_dataset.py \
--input "data" \
--output "data/protos" \
--text-extension .lab \
--num-workers 16
```
命令执行完毕后, 你应该能在 `data` 目录下看到 `protos` 文件.
### 4. 最后, 使用 LoRA 进行微调
同样的, 请确保你已经下载了 `LLAMA` 权重, 如果没有, 请运行以下命令:
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
对于中国大陆用户, 可使用 mirror 下载.
```bash
HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
最后, 你可以运行以下命令来启动微调:
```bash
python fish_speech/train.py --config-name text2semantic_finetune \
project=$project \
+lora@model.model.lora_config=r_8_alpha_16
```
!!! note
你可以通过修改 `fish_speech/configs/text2semantic_finetune.yaml` 来修改训练参数如 `batch_size`, `gradient_accumulation_steps` 等, 来适应你的显存.
!!! note
对于 Windows 用户, 你可以使用 `trainer.strategy.process_group_backend=gloo` 来避免 `nccl` 的问题.
训练结束后, 你可以参考 [推理](inference.md) 部分来测试你的模型.
!!! info
默认配置下, 基本只会学到说话人的发音方式, 而不包含音色, 你依然需要使用 prompt 来保证音色的稳定性.
如果你想要学到音色, 请将训练步数调大, 但这有可能会导致过拟合.
训练完成后, 你需要先将 loRA 的权重转为普通权重, 然后再进行推理.
```bash
python tools/llama/merge_lora.py \
--lora-config r_8_alpha_16 \
--base-weight checkpoints/fish-speech-1.5 \
--lora-weight results/$project/checkpoints/step_000000010.ckpt \
--output checkpoints/fish-speech-1.5-yth-lora/
```
!!! note
你也可以尝试其他的 checkpoint, 我们建议你使用最早的满足你要求的 checkpoint, 他们通常在 OOD 上表现更好.
docs/zh/index.md 0 → 100644
View file @ b75857fb
# 介绍
<div>
<a target="_blank" href="https://discord.gg/Es5qTB9BcN">
<img alt="Discord" src="https://img.shields.io/discord/1214047546020728892?color=%23738ADB&label=Discord&logo=discord&logoColor=white&style=flat-square"/>
</a>
<a target="_blank" href="http://qm.qq.com/cgi-bin/qm/qr?_wv=1027&k=jCKlUP7QgSm9kh95UlBoYv6s1I-Apl1M&authKey=xI5ttVAp3do68IpEYEalwXSYZFdfxZSkah%2BctF5FIMyN2NqAa003vFtLqJyAVRfF&noverify=0&group_code=593946093">
<img alt="QQ" src="https://img.shields.io/badge/QQ Group-%2312B7F5?logo=tencent-qq&logoColor=white&style=flat-square"/>
</a>
<a target="_blank" href="https://hub.docker.com/r/fishaudio/fish-speech">
<img alt="Docker" src="https://img.shields.io/docker/pulls/fishaudio/fish-speech?style=flat-square&logo=docker"/>
</a>
</div>
!!! warning "警告"
我们不对代码库的任何非法使用承担任何责任. 请参阅您当地关于 DMCA (数字千年法案) 和其他相关法律法规. <br/>
此代码库与所有模型根据 CC-BY-NC-SA-4.0 许可证发布.
<p align="center">
<img src="../assets/figs/diagram.png" width="75%">
</p>
## 要求
- GPU 内存: 4GB (用于推理), 8GB (用于微调)
- 系统: Linux, Windows
## Windows 配置
Windows 专业用户可以考虑 WSL2 或 docker 来运行代码库。
```bash
# 创建一个 python 3.10 虚拟环境, 你也可以用 virtualenv
conda create -n fish-speech python=3.10
conda activate fish-speech
# 安装 pytorch
pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 --index-url https://download.pytorch.org/whl/cu121
# 安装 fish-speech
pip3 install -e .
# (开启编译加速) 安装 triton-windows
pip install https://github.com/AnyaCoder/fish-speech/releases/download/v0.1.0/triton_windows-0.1.0-py3-none-any.whl
```
Windows 非专业用户可考虑以下为免 Linux 环境的基础运行方法（附带模型编译功能，即 `torch.compile`）：
1. 解压项目压缩包。
2. 点击 `install_env.bat` 安装环境。
3. 若需要开启编译加速则执行这一步:
1. 使用如下链接下载 LLVM 编译器。
- [LLVM-17.0.6（原站站点下载）](https://huggingface.co/fishaudio/fish-speech-1/resolve/main/LLVM-17.0.6-win64.exe?download=true)
- [LLVM-17.0.6（镜像站点下载）](https://hf-mirror.com/fishaudio/fish-speech-1/resolve/main/LLVM-17.0.6-win64.exe?download=true)
- 下载完 `LLVM-17.0.6-win64.exe` 后，双击进行安装，选择合适的安装位置，最重要的是勾选 `Add Path to Current User` 添加环境变量。
- 确认安装完成。
2. 下载安装 Microsoft Visual C++ 可再发行程序包，解决潜在 .dll 丢失问题。
- [MSVC++ 14.40.33810.0 下载](https://aka.ms/vs/17/release/vc_redist.x64.exe)
3. 下载安装 Visual Studio 社区版以获取 MSVC++ 编译工具, 解决 LLVM 的头文件依赖问题。
- [Visual Studio 下载](https://visualstudio.microsoft.com/zh-hans/downloads/)
- 安装好 Visual Studio Installer 之后，下载 Visual Studio Community 2022
- 如下图点击`修改`按钮，找到`使用C++的桌面开发`项，勾选下载
4. 下载安装 [CUDA Toolkit 12.x](https://developer.nvidia.com/cuda-12-1-0-download-archive?target_os=Windows&target_arch=x86_64)
4. 双击 `start.bat` 打开训练推理 WebUI 管理界面. 如有需要，可照下列提示修改`API_FLAGS`.
!!! info "可选"
想启动 推理 WebUI 界面？编辑项目根目录下的 `API_FLAGS.txt`, 前三行修改成如下格式:
```
--infer
# --api
# --listen ...
...
```
!!! info "可选"
想启动 API 服务器？编辑项目根目录下的 `API_FLAGS.txt`, 前三行修改成如下格式:
```
# --infer
--api
--listen ...
...
```
!!! info "可选"
双击 `run_cmd.bat` 进入本项目的 conda/python 命令行环境
## Linux 配置
有关详细信息，请参见 [pyproject.toml](../../pyproject.toml)。
```bash
# 创建一个 python 3.10 虚拟环境, 你也可以用 virtualenv
conda create -n fish-speech python=3.10
conda activate fish-speech
# 安装 pytorch
pip3 install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
# (Ubuntu / Debian 用户) 安装 sox + ffmpeg
apt install libsox-dev ffmpeg
# (Ubuntu / Debian 用户) 安装 pyaudio
apt install build-essential \
cmake \
libasound-dev \
portaudio19-dev \
libportaudio2 \
libportaudiocpp0
# 安装 fish-speech
pip3 install -e .[stable]
```
## macos 配置
如果您想在 MPS 上进行推理，请添加 `--device mps` 标志。
有关推理速度的比较，请参考 [此 PR](https://github.com/fishaudio/fish-speech/pull/461#issuecomment-2284277772)。
!!! 警告
`compile` 选项在 Apple Silicon 设备上尚未正式支持，因此推理速度没有提升的保证。
```bash
# create a python 3.10 virtual environment, you can also use virtualenv
conda create -n fish-speech python=3.10
conda activate fish-speech
# install pytorch
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1
# install fish-speech
pip install -e .[stable]
```
## Docker 配置
1. 安装 NVIDIA Container Toolkit：
Docker 如果想使用 GPU 进行模型训练和推理，需要安装 NVIDIA Container Toolkit ：
对于 Ubuntu 用户：
```bash
# 添加远程仓库
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
# 安装 nvidia-container-toolkit
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
# 重启 Docker 服务
sudo systemctl restart docker
```
对于使用其他 Linux 发行版的用户，安装指南请参考：[NVIDIA Container Toolkit Install-guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)。
注：对于中国大陆的用户，您可能需要使用代理来完成相关工具的安装。
2. 拉取并运行 fish-speech 镜像
```shell
# 拉取镜像
docker pull fishaudio/fish-speech:latest-dev
# 运行镜像
docker run -it \
--name fish-speech \
--gpus all \
-p 7860:7860 \
fishaudio/fish-speech:latest-dev \
zsh
# 如果需要使用其他端口，请修改 -p 参数为 YourPort:7860
```
3. 下载模型依赖
确保您在 docker 容器内的终端，然后再从我们的 huggingface 仓库下载所需的 `vqgan` 和 `llama` 模型。
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
对于中国大陆用户，可以通过镜像站下载。
```bash
HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
4. 配置环境变量，访问 WebUI
在 docker 容器内的终端，输入 `export GRADIO_SERVER_NAME="0.0.0.0"` ，从而让外部可以访问 docker 内的 gradio 服务。
接着在 docker 容器内的终端，输入 `python tools/run_webui.py` 即可开启 WebUI 服务。
如果是 WSL 或者是 MacOS ，访问 [http://localhost:7860](http://localhost:7860) 即可打开 WebUI 界面。
如果是部署在服务器上，更换 localhost 为您的服务器 ip 即可。
## 更新日志
- 2024/09/10: 更新了 Fish-Speech 到 1.4, 增加了数据集大小， quantizer n_groups 4 -> 8.
- 2024/07/02: 更新了 Fish-Speech 到 1.2 版本，移除 VITS Decoder，同时极大幅度提升 zero-shot 能力.
- 2024/05/10: 更新了 Fish-Speech 到 1.1 版本，引入了 VITS Decoder 来降低口胡和提高音色相似度.
- 2024/04/22: 完成了 Fish-Speech 1.0 版本, 大幅修改了 VQGAN 和 LLAMA 模型.
- 2023/12/28: 添加了 `lora` 微调支持.
- 2023/12/27: 添加了 `gradient checkpointing`, `causual sampling` 和 `flash-attn` 支持.
- 2023/12/19: 更新了 Webui 和 HTTP API.
- 2023/12/18: 更新了微调文档和相关例子.
- 2023/12/17: 更新了 `text2semantic` 模型, 支持无音素模式.
- 2023/12/13: 测试版发布, 包含 VQGAN 模型和一个基于 LLAMA 的语言模型 (只支持音素).
## 致谢
- [VITS2 (daniilrobnikov)](https://github.com/daniilrobnikov/vits2)
- [Bert-VITS2](https://github.com/fishaudio/Bert-VITS2)
- [GPT VITS](https://github.com/innnky/gpt-vits)
- [MQTTS](https://github.com/b04901014/MQTTS)
- [GPT Fast](https://github.com/pytorch-labs/gpt-fast)
- [Transformers](https://github.com/huggingface/transformers)
- [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS)
docs/zh/inference.md 0 → 100644
View file @ b75857fb
# 推理
推理支持命令行, http api, 以及 webui 三种方式.
!!! note
总的来说, 推理分为几个部分:
1. 给定一段 ~10 秒的语音, 将它用 VQGAN 编码.
2. 将编码后的语义 token 和对应文本输入语言模型作为例子.
3. 给定一段新文本, 让模型生成对应的语义 token.
4. 将生成的语义 token 输入 VQGAN 解码, 生成对应的语音.
## 下载模型
从我们的 huggingface 仓库下载所需的 `vqgan` 和 `llama` 模型。
```bash
huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
对于中国大陆用户，可使用 mirror 下载。
```bash
HF_ENDPOINT=https://hf-mirror.com huggingface-cli download fishaudio/fish-speech-1.5 --local-dir checkpoints/fish-speech-1.5
```
## 命令行推理
### 1. 从语音生成 prompt:
!!! note
如果你打算让模型随机选择音色, 你可以跳过这一步.
!!! warning "未来版本警告"
我们保留了从原来路径（tools/vqgan/infernce.py）访问的接口，但是这个接口可能在之后几个版本被删除，请尽快更改你的代码。
```bash
python fish_speech/models/vqgan/inference.py \
-i "paimon.wav" \
--checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
```
你应该能得到一个 `fake.npy` 文件.
### 2. 从文本生成语义 token:
!!! warning "未来版本警告"
我们保留了从原来路径（tools/llama/generate.py）访问的接口，但是这个接口可能在之后几个版本被删除，请尽快更改你的代码。
```bash
python fish_speech/models/text2semantic/inference.py \
--text "要转换的文本" \
--prompt-text "你的参考文本" \
--prompt-tokens "fake.npy" \
--checkpoint-path "checkpoints/fish-speech-1.5" \
--num-samples 2 \
--compile
```
该命令会在工作目录下创建 `codes_N` 文件, 其中 N 是从 0 开始的整数.
!!! note
您可能希望使用 `--compile` 来融合 cuda 内核以实现更快的推理 (~30 个 token/秒 -> ~500 个 token/秒).
对应的, 如果你不打算使用加速, 你可以注释掉 `--compile` 参数.
!!! info
对于不支持 bf16 的 GPU, 你可能需要使用 `--half` 参数.
### 3. 从语义 token 生成人声:
#### VQGAN 解码
!!! warning "未来版本警告"
我们保留了从原来路径（tools/vqgan/infernce.py）访问的接口，但是这个接口可能在之后几个版本被删除，请尽快更改你的代码。
```bash
python fish_speech/models/vqgan/inference.py \
-i "codes_0.npy" \
--checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth"
```
## HTTP API 推理
运行以下命令来启动 HTTP 服务:
```bash
python -m tools.api_server \
--listen 0.0.0.0:8080 \
--llama-checkpoint-path "checkpoints/fish-speech-1.5" \
--decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
--decoder-config-name firefly_gan_vq
```
> 如果你想要加速推理，可以加上`--compile`参数。
推荐中国大陆用户运行以下命令来启动 HTTP 服务:
```bash
HF_ENDPOINT=https://hf-mirror.com python -m ...(同上)
```
随后, 你可以在 `http://127.0.0.1:8080/` 中查看并测试 API.
下面是使用`tools/api_client.py`发送请求的示例。
```bash
python -m tools.api_client \
--text "要输入的文本" \
--reference_audio "参考音频路径" \
--reference_text "参考音频的文本内容" \
--streaming True
```
上面的命令表示按照参考音频的信息，合成所需的音频并流式返回.
下面的示例展示了， 可以一次使用**多个** `参考音频路径` 和 `参考音频的文本内容`。在命令里用空格隔开即可。
```bash
python -m tools.api_client \
--text "要输入的文本" \
--reference_audio "参考音频路径1" "参考音频路径2" \
--reference_text "参考音频的文本内容1" "参考音频的文本内容2"\
--streaming False \
--output "generated" \
--format "mp3"
```
上面的命令表示按照多个参考音频的信息，合成所需的`MP3`格式音频，并保存为当前目录的`generated.mp3`文件。
还可以用`--reference_id`(仅能用一个)来代替`--reference_audio`和`--reference_text`, 前提是在项目根目录下创建`references/<your reference_id>`文件夹，
里面放上任意对音频与标注文本。 目前支持的参考音频最多加起来总时长90s。
!!! info
要了解有关可用参数的更多信息，可以使用命令`python -m tools.api_client -h`
## GUI 推理
[下载客户端](https://github.com/AnyaCoder/fish-speech-gui/releases)
## WebUI 推理
你可以使用以下命令来启动 WebUI:
```bash
python -m tools.run_webui \
--llama-checkpoint-path "checkpoints/fish-speech-1.5" \
--decoder-checkpoint-path "checkpoints/fish-speech-1.5/firefly-gan-vq-fsq-8x1024-21hz-generator.pth" \
--decoder-config-name firefly_gan_vq
```
> 如果你想要加速推理，可以加上`--compile`参数。
!!! note
你可以提前将label文件和参考音频文件保存到主目录下的 `references` 文件夹（需要自行创建），这样你可以直接在WebUI中调用它们。
!!! note
你可以使用 Gradio 环境变量, 如 `GRADIO_SHARE`, `GRADIO_SERVER_PORT`, `GRADIO_SERVER_NAME` 来配置 WebUI.
祝大家玩得开心!
docs/zh/samples.md 0 → 100644
View file @ b75857fb
# 例子
v1.4 演示已更新至[此处](https://speech.fish.audio/samples/)。
v1.2 的样本可以在 [Bilibili](https://www.bilibili.com/video/BV1wz421B71D/) 观看。
以下样本来自 v1.1 版本的模型。
## 中文句子 1
```
人间灯火倒映湖中，她的渴望让静水泛起涟漪。若代价只是孤独，那就让这份愿望肆意流淌。
流入她所注视的世间，也流入她如湖水般澄澈的目光。
```
<table>
<thead>
<tr>
<th>说话人</th>
<th>输入音频</th>
<th>合成音频</th>
</tr>
</thead>
<tbody>
<tr>
<td>纳西妲 (原神)</td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_output.wav" /></td>
</tr>
<tr>
<td>钟离 (原神)</td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_input.wav" /></td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/1_output.wav" /></td>
</tr>
<tr>
<td>芙宁娜 (原神)</td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_input.wav" /></td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/2_output.wav" /></td>
</tr>
<tr>
<td>随机说话人 1</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/4_output.wav" /></td>
</tr>
<tr>
<td>随机说话人 2</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/5_output.wav" /></td>
</tr>
</tbody>
</table>
## 中文句子 2
```
你们这个是什么群啊，你们这是害人不浅啊你们这个群！谁是群主，出来！真的太过分了。你们搞这个群干什么？
我儿子每一科的成绩都不过那个平均分呐，他现在初二，你叫我儿子怎么办啊？他现在还不到高中啊？
你们害死我儿子了！快点出来你这个群主！再这样我去报警了啊！我跟你们说你们这一帮人啊，一天到晚啊，
搞这些什么游戏啊，动漫啊，会害死你们的，你们没有前途我跟你说。你们这九百多个人，好好学习不好吗？
一天到晚在上网。有什么意思啊？麻烦你重视一下你们的生活的目标啊？有一点学习目标行不行？一天到晚上网是不是人啊？
```
<table>
<thead>
<tr>
<th>说话人</th>
<th>输入音频</th>
<th>合成音频</th>
</tr>
</thead>
<tbody>
<tr>
<td>纳西妲 (原神)</td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/0_input.wav" /></td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/6_output.wav" /></td>
</tr>
<tr>
<td>随机说话人</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/7_output.wav" /></td>
</tr>
</tbody>
</table>
## 中文句子 3
```
大家好，我是 Fish Audio 开发的开源文本转语音模型。经过十五万小时的数据训练，
我已经能够熟练掌握中文、日语和英语，我的语言处理能力接近人类水平，声音表现形式丰富多变。
作为一个仅有亿级参数的模型，我相信社区成员能够在个人设备上轻松运行和微调，让我成为您的私人语音助手。
```
<table>
<thead>
<tr>
<th>说话人</th>
<th>输入音频</th>
<th>合成音频</th>
</tr>
</thead>
<tbody>
<tr>
<td>随机说话人</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/zh/8_output.wav" /></td>
</tr>
</tbody>
</table>
## 英文句子 1
```
In the realm of advanced technology, the evolution of artificial intelligence stands as a
monumental achievement. This dynamic field, constantly pushing the boundaries of what
machines can do, has seen rapid growth and innovation. From deciphering complex data
patterns to driving cars autonomously, AI's applications are vast and diverse.
```
<table>
<thead>
<tr>
<th>说话人</th>
<th>输入音频</th>
<th>合成音频</th>
</tr>
</thead>
<tbody>
<tr>
<td>随机说话人 1</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/0_output.wav" /></td>
</tr>
<tr>
<td>随机说话人 2</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/1_output.wav" /></td>
</tr>
</tbody>
</table>
## 英文句子 2
```
Hello everyone, I am an open-source text-to-speech model developed by
Fish Audio. After training with 150,000 hours of data, I have become proficient
in Chinese, Japanese, and English, and my language processing abilities
are close to human level. My voice is capable of a wide range of expressions.
As a model with only hundreds of millions of parameters, I believe community
members can easily run and fine-tune me on their personal devices, allowing
me to serve as your personal voice assistant.
```
<table>
<thead>
<tr>
<th>说话人</th>
<th>输入音频</th>
<th>合成音频</th>
</tr>
</thead>
<tbody>
<tr>
<td>随机说话人</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/en/2_output.wav" /></td>
</tr>
</tbody>
</table>
## 日文句子 1
```
先進技術の領域において、人工知能の進化は画期的な成果として立っています。常に機械ができることの限界を
押し広げているこのダイナミックな分野は、急速な成長と革新を見せています。複雑なデータパターンの解読か
ら自動運転車の操縦まで、AIの応用は広範囲に及びます。
```
<table>
<thead>
<tr>
<th>说话人</th>
<th>输入音频</th>
<th>合成音频</th>
</tr>
</thead>
<tbody>
<tr>
<td>随机说话人 1</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/0_output.wav" /></td>
</tr>
<tr>
<td>随机说话人 2</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/1_output.wav" /></td>
</tr>
</tbody>
</table>
## 日文句子 2
```
皆さん、こんにちは。私はフィッシュオーディオによって開発されたオープンソースのテ
キストから音声への変換モデルです。15万時間のデータトレーニングを経て、
中国語、日本語、英語を熟知しており、言語処理能力は人間に近いレベルです。
声の表現も多彩で豊かです。数億のパラメータを持つこのモデルは、コミュニティ
のメンバーが個人のデバイスで簡単に実行し、微調整することができると
信じています。これにより、私を個人の音声アシスタントとして活用できます。
```
<table>
<thead>
<tr>
<th>说话人</th>
<th>输入音频</th>
<th>合成音频</th>
</tr>
</thead>
<tbody>
<tr>
<td>随机说话人</td>
<td> - </td>
<td><audio controls preload="auto" src="https://demo-r2.speech.fish.audio/v1.1-sft-large/ja/2_output.wav" /></td>
</tr>
</tbody>
</table>
docs/zh/start_agent.md 0 → 100644
View file @ b75857fb
# 启动 Agent
## 要求
- GPU 显存: 至少 8GB（在量化的条件下），推荐 16GB 及以上
- 硬盘使用量: 10GB
## 下载模型
你可以执行下面的语句来获取模型:
```bash
huggingface-cli download fishaudio/fish-agent-v0.1-3b --local-dir checkpoints/fish-agent-v0.1-3b
```
如果你处于国内网络，首先执行:
```bash
export HF_ENDPOINT=https://hf-mirror.com
```
把他们放进名为 'checkpoints' 的文件夹内。
你同样需要 fish-speech 的模型，关于如何获取 fish-speech 模型请查看[inference](inference.md)。
完成后你的 checkpoints 文件夹中会有两个子文件夹：`checkpoints/fish-speech-1.4` 和 `checkpoints/fish-agent-v0.1-3b`。
## Environment Prepare
如果你已经有了 Fish-Speech 环境，你可以在安装下面的包的前提下直接使用：
```bash
pip install cachetools
```
!!! note
请使用小于 3.12 的 python 版本使 compile 可用
如果你没有 Fish-Speech 环境，请执行下面的语句来构造你的环境：
```bash
sudo apt-get install portaudio19-dev
pip install -e .[stable]
```
## 链接 Agent.
你需要使用以下指令来构建 fish-agent
```bash
python -m tools.api_server --llama-checkpoint-path checkpoints/fish-agent-v0.1-3b/ --mode agent --compile
```
`--compile`只能在小于 3.12 版本的 Python 使用，这个功能可以极大程度上提高生成速度。
你需要哦注意 compile 需要进行一段时间.
然后启动另一个终端并执行:
```bash
python -m tools.e2e_webui
```
这会在设备上创建一个 Gradio WebUI。
每当进行第一轮对话的时候，模型需要 compile 一段时间，请耐心等待
## Gradio Webui
<p align="center">
<img src="../../assets/figs/agent_gradio.png" width="75%">
</p>
玩得开心！
## Performance
在我们的测试环境下， 4060 laptop GPU 只能刚刚运行该模型，只有大概 8 tokens/s。 4090 GPU 可以在编译后达到 95 tokens/s，我们推荐使用至少 4080 以上级别的 GPU 来达到较好体验。
# About Agent
该模型仍处于测试阶段。如果你发现了问题，请给我们提 issue 或者 pull request，我们非常感谢。
entrypoint.sh 0 → 100644
View file @ b75857fb
#!/bin/bash
CUDA_ENABLED=${CUDA_ENABLED:-true}
DEVICE=""
if [ "${CUDA_ENABLED}" != "true" ]; then
DEVICE="--device cpu"
fi
exec python tools/run_webui.py ${DEVICE}
fish_speech/callbacks/__init__.py 0 → 100644
View file @ b75857fb
from .grad_norm import GradNormMonitor
__all__ = ["GradNormMonitor"]
fish_speech/callbacks/grad_norm.py 0 → 100644
View file @ b75857fb
This diff is collapsed.
fish_speech/configs/base.yaml 0 → 100644
View file @ b75857fb
This diff is collapsed.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment