Initial commit

data/librispeech/dev-clean/1272/135031/1272-135031.trans.txt

+1272-135031-0000 BECAUSE YOU WERE SLEEPING INSTEAD OF CONQUERING THE LOVELY ROSE PRINCESS HAS BECOME A FIDDLE WITHOUT A BOW WHILE POOR SHAGGY SITS THERE A COOING DOVE
+1272-135031-0001 HE HAS GONE AND GONE FOR GOOD ANSWERED POLYCHROME WHO HAD MANAGED TO SQUEEZE INTO THE ROOM BESIDE THE DRAGON AND HAD WITNESSED THE OCCURRENCES WITH MUCH INTEREST
+1272-135031-0002 I HAVE REMAINED A PRISONER ONLY BECAUSE I WISHED TO BE ONE AND WITH THIS HE STEPPED FORWARD AND BURST THE STOUT CHAINS AS EASILY AS IF THEY HAD BEEN THREADS
+1272-135031-0003 THE LITTLE GIRL HAD BEEN ASLEEP BUT SHE HEARD THE RAPS AND OPENED THE DOOR
+1272-135031-0004 THE KING HAS FLED IN DISGRACE AND YOUR FRIENDS ARE ASKING FOR YOU
+1272-135031-0005 I BEGGED RUGGEDO LONG AGO TO SEND HIM AWAY BUT HE WOULD NOT DO SO
+1272-135031-0006 I ALSO OFFERED TO HELP YOUR BROTHER TO ESCAPE BUT HE WOULD NOT GO
+1272-135031-0007 HE EATS AND SLEEPS VERY STEADILY REPLIED THE NEW KING
+1272-135031-0008 I HOPE HE DOESN'T WORK TOO HARD SAID SHAGGY
+1272-135031-0009 HE DOESN'T WORK AT ALL
+1272-135031-0010 IN FACT THERE IS NOTHING HE CAN DO IN THESE DOMINIONS AS WELL AS OUR NOMES WHOSE NUMBERS ARE SO GREAT THAT IT WORRIES US TO KEEP THEM ALL BUSY
+1272-135031-0011 NOT EXACTLY RETURNED KALIKO
+1272-135031-0012 WHERE IS MY BROTHER NOW
+1272-135031-0013 INQUIRED SHAGGY IN THE METAL FOREST
+1272-135031-0014 WHERE IS THAT
+1272-135031-0015 THE METAL FOREST IS IN THE GREAT DOMED CAVERN THE LARGEST IN ALL OUR DOMINIONS REPLIED KALIKO
+1272-135031-0016 KALIKO HESITATED
+1272-135031-0017 HOWEVER IF WE LOOK SHARP WE MAY BE ABLE TO DISCOVER ONE OF THESE SECRET WAYS
+1272-135031-0018 OH NO I'M QUITE SURE HE DIDN'T
+1272-135031-0019 THAT'S FUNNY REMARKED BETSY THOUGHTFULLY
+1272-135031-0020 I DON'T BELIEVE ANN KNEW ANY MAGIC OR SHE'D HAVE WORKED IT BEFORE
+1272-135031-0021 I DO NOT KNOW CONFESSED SHAGGY
+1272-135031-0022 TRUE AGREED KALIKO
+1272-135031-0023 KALIKO WENT TO THE BIG GONG AND POUNDED ON IT JUST AS RUGGEDO USED TO DO BUT NO ONE ANSWERED THE SUMMONS
+1272-135031-0024 HAVING RETURNED TO THE ROYAL CAVERN KALIKO FIRST POUNDED THE GONG AND THEN SAT IN THE THRONE WEARING RUGGEDO'S DISCARDED RUBY CROWN AND HOLDING IN HIS HAND THE SCEPTRE WHICH RUGGEDO HAD SO OFTEN THROWN AT HIS HEAD

data/librispeech/dev-clean/1272/141231/1272-141231.trans.txt

+1272-141231-0000 A MAN SAID TO THE UNIVERSE SIR I EXIST
+1272-141231-0001 SWEAT COVERED BRION'S BODY TRICKLING INTO THE TIGHT LOINCLOTH THAT WAS THE ONLY GARMENT HE WORE
+1272-141231-0002 THE CUT ON HIS CHEST STILL DRIPPING BLOOD THE ACHE OF HIS OVERSTRAINED EYES EVEN THE SOARING ARENA AROUND HIM WITH THE THOUSANDS OF SPECTATORS WERE TRIVIALITIES NOT WORTH THINKING ABOUT
+1272-141231-0003 HIS INSTANT OF PANIC WAS FOLLOWED BY A SMALL SHARP BLOW HIGH ON HIS CHEST
+1272-141231-0004 ONE MINUTE A VOICE SAID AND THE TIME BUZZER SOUNDED
+1272-141231-0005 A MINUTE IS NOT A VERY LARGE MEASURE OF TIME AND HIS BODY NEEDED EVERY FRACTION OF IT
+1272-141231-0006 THE BUZZER'S WHIRR TRIGGERED HIS MUSCLES INTO COMPLETE RELAXATION
+1272-141231-0007 ONLY HIS HEART AND LUNGS WORKED ON AT A STRONG MEASURED RATE
+1272-141231-0008 HE WAS IN REVERIE SLIDING ALONG THE BORDERS OF CONSCIOUSNESS
+1272-141231-0009 THE CONTESTANTS IN THE TWENTIES NEEDED UNDISTURBED REST THEREFORE NIGHTS IN THE DORMITORIES WERE AS QUIET AS DEATH
+1272-141231-0010 PARTICULARLY SO ON THIS LAST NIGHT WHEN ONLY TWO OF THE LITTLE CUBICLES WERE OCCUPIED THE THOUSANDS OF OTHERS STANDING WITH DARK EMPTY DOORS
+1272-141231-0011 THE OTHER VOICE SNAPPED WITH A HARSH URGENCY CLEARLY USED TO COMMAND
+1272-141231-0012 I'M HERE BECAUSE THE MATTER IS OF UTMOST IMPORTANCE AND BRANDD IS THE ONE I MUST SEE NOW STAND ASIDE
+1272-141231-0013 THE TWENTIES
+1272-141231-0014 HE MUST HAVE DRAWN HIS GUN BECAUSE THE INTRUDER SAID QUICKLY PUT THAT AWAY YOU'RE BEING A FOOL OUT
+1272-141231-0015 THERE WAS SILENCE THEN AND STILL WONDERING BRION WAS ONCE MORE ASLEEP
+1272-141231-0016 TEN SECONDS
+1272-141231-0017 HE ASKED THE HANDLER WHO WAS KNEADING HIS ACHING MUSCLES
+1272-141231-0018 A RED HAIRED MOUNTAIN OF A MAN WITH AN APPARENTLY INEXHAUSTIBLE STORE OF ENERGY
+1272-141231-0019 THERE COULD BE LITTLE ART IN THIS LAST AND FINAL ROUND OF FENCING
+1272-141231-0020 JUST THRUST AND PARRY AND VICTORY TO THE STRONGER
+1272-141231-0021 EVERY MAN WHO ENTERED THE TWENTIES HAD HIS OWN TRAINING TRICKS
+1272-141231-0022 THERE APPEARED TO BE AN IMMEDIATE ASSOCIATION WITH THE DEATH TRAUMA AS IF THE TWO WERE INEXTRICABLY LINKED INTO ONE
+1272-141231-0023 THE STRENGTH THAT ENABLES SOMEONE IN A TRANCE TO HOLD HIS BODY STIFF AND UNSUPPORTED EXCEPT AT TWO POINTS THE HEAD AND HEELS
+1272-141231-0024 THIS IS PHYSICALLY IMPOSSIBLE WHEN CONSCIOUS
+1272-141231-0025 OTHERS HAD DIED BEFORE DURING THE TWENTIES AND DEATH DURING THE LAST ROUND WAS IN SOME WAYS EASIER THAN DEFEAT
+1272-141231-0026 BREATHING DEEPLY BRION SOFTLY SPOKE THE AUTO HYPNOTIC PHRASES THAT TRIGGERED THE PROCESS
+1272-141231-0027 WHEN THE BUZZER SOUNDED HE PULLED HIS FOIL FROM HIS SECOND'S STARTLED GRASP AND RAN FORWARD
+1272-141231-0028 IROLG LOOKED AMAZED AT THE SUDDEN FURY OF THE ATTACK THEN SMILED
+1272-141231-0029 HE THOUGHT IT WAS A LAST BURST OF ENERGY HE KNEW HOW CLOSE THEY BOTH WERE TO EXHAUSTION
+1272-141231-0030 BRION SAW SOMETHING CLOSE TO PANIC ON HIS OPPONENT'S FACE WHEN THE MAN FINALLY RECOGNIZED HIS ERROR
+1272-141231-0031 A WAVE OF DESPAIR ROLLED OUT FROM IROLG BRION SENSED IT AND KNEW THE FIFTH POINT WAS HIS
+1272-141231-0032 THEN THE POWERFUL TWIST THAT THRUST IT ASIDE IN AND UNDER THE GUARD

docker/Dockerfile

+FROM image.sourcefind.cn:5000/dcu/admin/base/pytorch:2.1.0-centos7.6-dtk24.04-py310
+RUN source /opt/dtk/env.sh
\ No newline at end of file

inference/speech_asr.py

+from transformers import SpeechT5Processor, SpeechT5ForSpeechToText
+from transformers import logging
+from datasets import load_dataset
+import torch
+import argparse
+import librosa
+import numpy as np
+import os
+
+def parse_opt(known=False):
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-hip', '--hip-device', type=int, default=0, help="initial hip devices")
+    parser.add_argument('-m', '--model-path', type=str, default="", help="initial model path")
+    parser.add_argument('-is', '--input_speech', type=str, default="Autumn, the season of change.", help="input speech")
+    parser.add_argument('-res', '--result_path', type=str, default="../res", help="the path to save wav file")
+    opt = parser.parse_known_args()[0] if known else parser.parse_args()
+    return opt
+
+def main(opt):
+    device = torch.device(f"cuda:{int(opt.hip_device)}")
+    print(f"Using device: {device}")
+
+    # 初始化speechT5_asr模型
+    logging.set_verbosity_warning()
+    processor = SpeechT5Processor.from_pretrained(opt.model_path)
+    model = SpeechT5ForSpeechToText.from_pretrained(opt.model_path).to(device)
+
+    # input:encoder的语音输入
+    # input:encoder的输入-单声道/16kHZ
+    example_speech, sampling_rate = librosa.load(opt.input_speech, sr=16000, dtype=np.float64)
+    inputs = processor(audio=example_speech, sampling_rate=sampling_rate, return_tensors="pt").to(device)
+
+    # output:decoder的文本输出
+    predicted_ids = model.generate(**inputs, max_length=100)
+    transcription = processor.batch_decode(predicted_ids.cpu(), skip_special_tokens=True)
+    print("text: {}".format(transcription[0]))
+    with open(os.path.join(opt.result_path, "asr.txt"), "+w") as f:
+        f.write("text: {}".format(transcription[0]))
+
+if __name__ == "__main__":
+    main(parse_opt())
\ No newline at end of file

inference/speech_tts.py

+from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech
+from transformers import SpeechT5HifiGan
+from datasets import load_dataset
+import torch
+import os 
+import numpy as np
+import soundfile as sf
+import argparse
+
+def parse_opt(known=False):
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-hip', '--hip-device', type=int, default=0, help="initial hip devices")
+    parser.add_argument('-m', '--model-path', type=str, default="", help="initial model path")
+    parser.add_argument('-v', '--vocoder-path', type=str, default="", help="the vocoder model path")
+    parser.add_argument('-t', '--text', type=str, default="Autumn, the season of change.", help="input text")
+    parser.add_argument('-s', '--speaker', type=str, default="", help="the feature of speaker:path of xxx.npy")
+    parser.add_argument('-res', '--result_path', type=str, default="../res", help="the path to save wav file")
+    opt = parser.parse_known_args()[0] if known else parser.parse_args()
+    return opt
+
+def main(opt):    
+    device = torch.device(f"cuda:{int(opt.hip_device)}")
+    print(f"Using device: {device}")
+    # 设置HF的下载路径为国内镜像
+    os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"
+
+    # 初始化speechT5_tts模型
+    processor = SpeechT5Processor.from_pretrained(opt.model_path)
+    model = SpeechT5ForTextToSpeech.from_pretrained(opt.model_path).to(device)
+
+    # input:encoder的文本输入
+    inputs = processor(text=opt.text, return_tensors="pt").to(device)
+
+    # input:decoder的发音人的语音特征输入(speaker embedding)
+    speaker_embeddings = np.load(opt.speaker).astype(np.float64)
+    speaker_embeddings = torch.tensor(speaker_embeddings, dtype=torch.float32).unsqueeze(0).to(device)
+
+    # output:使用hifigan声码器将mfcc转为语音
+    vocoder = SpeechT5HifiGan.from_pretrained(opt.vocoder_path).to(device)
+    speech = model.generate_speech(inputs["input_ids"], speaker_embeddings, vocoder=vocoder)
+    sf.write(os.path.join(opt.result_path, "tts.wav"), speech.cpu().numpy(), samplerate=16000)
+
+if __name__ == "__main__":
+    main(opt=parse_opt())
\ No newline at end of file

inference/speech_vc.py

+from transformers import SpeechT5Processor, SpeechT5ForSpeechToSpeech, SpeechT5HifiGan
+from transformers import SpeechT5HifiGan
+from datasets import load_dataset
+import torch
+import os 
+import numpy as np
+import soundfile as sf
+import librosa
+import argparse
+
+def parse_opt(known=False):
+    parser = argparse.ArgumentParser()
+    parser.add_argument('-hip', '--hip-device', type=int, default=0, help="initial hip devices")
+    parser.add_argument('-m', '--model-path', type=str, default="", help="initial model path")
+    parser.add_argument('-v', '--vocoder-path', type=str, default="", help="the vocoder model path")
+    parser.add_argument('-is', '--input_speech', type=str, default="Autumn, the season of change.", help="input speech")
+    parser.add_argument('-s', '--speaker', type=str, default="", help="the feature of speaker:path of xxx.npy")
+    parser.add_argument('-res', '--result_path', type=str, default="../res", help="the path to save wav file")
+    opt = parser.parse_known_args()[0] if known else parser.parse_args()
+    return opt
+
+def main(opt):
+    device = torch.device(f"cuda:{int(opt.hip_device)}")
+    print(f"Using device: {device}")
+    # 设置HF的下载路径为国内镜像
+    os.environ["HF_ENDPOINT"] = "https://hf-mirror.com"
+
+    # 模型加载
+    processor = SpeechT5Processor.from_pretrained(opt.model_path)
+    model = SpeechT5ForSpeechToSpeech.from_pretrained(opt.model_path).to(device)
+
+    # input:encoder的输入-单声道/16kHZ
+    example_speech, sampling_rate = librosa.load(opt.input_speech, sr=16000, dtype=np.float64)
+    inputs = processor(audio=example_speech, sampling_rate=sampling_rate, return_tensors="pt").to(device)
+
+    # input:decoder的输入-speaker_embedding
+    speaker_embeddings = np.load(opt.speaker).astype(np.float64)
+    speaker_embeddings = torch.tensor(speaker_embeddings, dtype=torch.float32).unsqueeze(0).to(device)
+
+    # output:decoder的输出-将声谱图转为音频波形
+    vocoder = SpeechT5HifiGan.from_pretrained(opt.vocoder_path).to(device)
+    speech = model.generate_speech(inputs["input_values"], speaker_embeddings, vocoder=vocoder)
+    sf.write(os.path.join(opt.result_path, "vc.wav"), speech.cpu().numpy(), samplerate=sampling_rate)
+
+if __name__ == "__main__":
+    main(parse_opt())
\ No newline at end of file

requirements.txt

+transformers==4.43.3
+datasets==2.20.0
+soundfile==0.12.1
+librosa==0.10.2.post1
+sentencepiece==0.2.0
\ No newline at end of file

res/asr.txt

+text: mister quilter is the apostle of the middle classes and we are glad to welcome his gospel
\ No newline at end of file