BladeDISC aasist code

LICENSE

+Copyright (c) 2021-present NAVER Corp.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

NOTICE

+AASIST
+Copyright (c) 2021-present NAVER Corp.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
+---
+
+This project contains subcomponents with separate copyright notices and license terms. 
+Your use of the source code for these subcomponents is subject to the terms and conditions of the following licenses.
+
+=====================================================================================================================
+
+asvspoof-challenge/2021/LA/Baseline-RawNet2
+https://github.com/asvspoof-challenge/2021/tree/main/LA/Baseline-RawNet2
+
+
+MIT License
+
+Copyright (c) 2021 eurecom-asp
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+=====
+
+Jungjee/RawNet
+https://github.com/Jungjee/RawNet
+
+
+MIT License
+
+Copyright (c) 2021 Jee-weon Jung
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+=====
+
+min t-DCF implementation for ASVspoof2019
+https://www.asvspoof.org/resources/tDCF_python_v2.zip
+
+
+A Python code package for computing t-DCF and EER metrics for ASVspoof2019.
+(Version 2.0)
+
+This work is licensed under the Creative Commons
+Attribution-NonCommercial-ShareAlike 4.0 International
+License. To view a copy of this license, visit
+http://creativecommons.org/licenses/by-nc-sa/4.0/
+or send a letter to
+Creative Commons, 444 Castro Street, Suite 900,
+Mountain View, California, 94041, USA.
+
+=====

README.md

+# AASIST 
+## 模型介绍
+是开源的音频反欺骗的模型
+
+## 模型结构
+aasist,参考链接https://github.com/clovaai/aasist， 参考论文ASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks
+
+## AASIST代码参考版本
+
+原始代码位置：https://github.com/clovaai/aasist.git
+
+## 数据集
+下载方式：
+python ./download_dataset.py
+或者可以手动从ASVspoof2019 dataset: https://datashare.ed.ac.uk/handle/10283/3336链接进行下载LA.zip文件
+下载后unzip解压
+
+## 推理
+
+### 环境配置
+提供[光源](https://www.sourcefind.cn/#/service-details)拉取的训练的docker镜像：
+* 训练镜像：
+
+python依赖安装：
+
+pip3 install -r requirement.txt
+
+### 推理命令,使用BladeDISC进行加速
+
+To evaluate AASIST [1]:
+export TORCH_MHLO_OP_WHITE_LIST="aten::max;aten::batch_norm;aten::abs,aten::selu;prim::NumToTensor;aten::zeros_like;aten::size;aten::narrow;aten::cat;aten::selu_"
+python3 main.py --eval --config ./config/AASIST.conf
+python3 main_opt.py --eval --config ./config/AASIST.conf
+
+
+To evaluate AASIST-L [1]:
+export TORCH_MHLO_OP_WHITE_LIST="aten::max;aten::batch_norm;aten::abs,aten::selu;prim::NumToTensor;aten::zeros_like;aten::size;aten::narrow;aten::cat;aten::selu_"
+python3 main.py --eval --config ./config/AASIST-L.conf
+python3 main_opt.py --eval --config ./config/AASIST-L.conf
+
+### 单卡测试
+测试命令：
+
+bash run.sh
+### 性能和准确率数据
+性能测试以AASIST-L为例(记录每次推理的执行时间)：
+
+case | 优化前(关闭disc) | 优化后(打开disc) | 性能提升 |
+| :---- | ----: | :----: | ---- |
+AASIST-L | 0.018s | 0.008s | 2.25倍
+
+准确率测试通过对比优化前与优化后的输出结果,精度差异较小，判定准确率为正常的
+
+## 参考
+* https://github.com/clovaai/aasist.git

README_aasist.md

+# AASIST
+
+This repository provides the overall framework for training and evaluating audio anti-spoofing systems proposed in ['AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks'](https://arxiv.org/abs/2110.01200)
+
+### Getting started
+`requirements.txt` must be installed for execution. We state our experiment environment for those who prefer to simulate as similar as possible. 
+- Installing dependencies
+```
+pip install -r requirements.txt
+```
+- Our environment (for GPU training)
+  - Based on a docker image: `pytorch:1.6.0-cuda10.1-cudnn7-runtime`
+  - GPU: 1 NVIDIA Tesla V100
+    - About 16GB is required to train AASIST using a batch size of 24
+  - gpu-driver: 418.67
+
+### Data preparation
+We train/validate/evaluate AASIST using the ASVspoof 2019 logical access dataset [4].
+```
+python ./download_dataset.py
+```
+(Alternative) Manual preparation is available via 
+- ASVspoof2019 dataset: https://datashare.ed.ac.uk/handle/10283/3336
+  1. Download `LA.zip` and unzip it
+  2. Set your dataset directory in the configuration file
+
+### Training 
+The `main.py` includes train/validation/evaluation.
+
+To train AASIST [1]:
+```
+python main.py --config ./config/AASIST.conf
+```
+To train AASIST-L [1]:
+```
+python main.py --config ./config/AASIST-L.conf
+```
+
+#### Training baselines
+
+We additionally enabled the training of RawNet2[2] and RawGAT-ST[3]. 
+
+To Train RawNet2 [2]:
+```
+python main.py --config ./config/RawNet2_baseline.conf
+```
+
+To train RawGAT-ST [3]:
+```
+python main.py --config ./config/RawGATST_baseline.conf
+```
+
+### Pre-trained models
+We provide pre-trained AASIST and AASIST-L.
+
+To evaluate AASIST [1]:
+- It shows `EER: 0.83%`, `min t-DCF: 0.0275`
+```
+python main.py --eval --config ./config/AASIST.conf
+```
+To evaluate AASIST-L [1]:
+- It shows `EER: 0.99%`, `min t-DCF: 0.0309`
+- Model has `85,306` parameters
+```
+python main.py --eval --config ./config/AASIST-L.conf
+```
+
+
+### Developing custom models
+Simply by adding a configuration file and a model architecture, one can train and evaluate their models.
+
+To train a custom model:
+```
+1. Define your model
+  - The model should be a class named "Model"
+2. Make a configuration by modifying "model_config"
+  - architecture: filename of your model.
+  - hyper-parameters to be tuned can be also passed using variables in "model_config"
+3. run python main.py --config {CUSTOM_CONFIG_NAME}
+```
+
+### License
+```
+Copyright (c) 2021-present NAVER Corp.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+```
+
+### Acknowledgements
+This repository is built on top of several open source projects. 
+- [ASVspoof 2021 baseline repo](https://github.com/asvspoof-challenge/2021/tree/main/LA/Baseline-RawNet2)
+- [min t-DCF implementation](https://www.asvspoof.org/resources/tDCF_python_v2.zip)
+
+The repository for baseline RawGAT-ST model will be open
+-  https://github.com/eurecom-asp/RawGAT-ST-antispoofing
+
+The dataset we use is ASVspoof 2019 [4]
+- https://www.asvspoof.org/index2019.html
+
+### References
+[1] AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks
+```bibtex
+@INPROCEEDINGS{Jung2021AASIST,
+  author={Jung, Jee-weon and Heo, Hee-Soo and Tak, Hemlata and Shim, Hye-jin and Chung, Joon Son and Lee, Bong-Jin and Yu, Ha-Jin and Evans, Nicholas},
+  booktitle={arXiv preprint arXiv:2110.01200}, 
+  title={AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks}, 
+  year={2021}
+```
+
+[2] End-to-End anti-spoofing with RawNet2
+```bibtex
+@INPROCEEDINGS{Tak2021End,
+  author={Tak, Hemlata and Patino, Jose and Todisco, Massimiliano and Nautsch, Andreas and Evans, Nicholas and Larcher, Anthony},
+  booktitle={Proc. ICASSP}, 
+  title={End-to-End anti-spoofing with RawNet2}, 
+  year={2021},
+  pages={6369-6373}
+}
+```
+
+[3] End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection
+```bibtex
+@inproceedings{tak21_asvspoof,
+  author={Tak, Hemlata and Jung, Jee-weon and Patino, Jose and Kamble, Madhu and Todisco, Massimiliano and Evans, Nicholas},
+  booktitle={Proc. ASVSpoof Challenge},
+  title={End-to-end spectro-temporal graph attention networks for speaker verification anti-spoofing and speech deepfake detection},
+  year={2021},
+  pages={1--8}
+```
+
+[4] ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech
+```bibtex
+@article{wang2020asvspoof,
+  title={ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech},
+  author={Wang, Xin and Yamagishi, Junichi and Todisco, Massimiliano and Delgado, H{\'e}ctor and Nautsch, Andreas and Evans, Nicholas and Sahidullah, Md and Vestman, Ville and Kinnunen, Tomi and Lee, Kong Aik and others},
+  journal={Computer Speech \& Language},
+  volume={64},
+  pages={101114},
+  year={2020},
+  publisher={Elsevier}
+}
+```

config/AASIST-L.conf

+{
+    "database_path": "./LA/",
+    "asv_score_path": "ASVspoof2019_LA_asv_scores/ASVspoof2019.LA.asv.eval.gi.trl.scores.txt",
+    "model_path": "./models/weights/AASIST-L.pth",
+    "batch_size": 24,
+    "num_epochs": 100,
+    "loss": "CCE",
+    "track": "LA",
+    "eval_all_best": "True",
+    "eval_output": "eval_scores_using_best_dev_model.txt",
+    "cudnn_deterministic_toggle": "True",
+    "cudnn_benchmark_toggle": "False",
+    "model_config": {
+        "architecture": "AASIST",
+        "nb_samp": 64600,
+        "first_conv": 128,
+        "filts": [70, [1, 32], [32, 32], [32, 24], [24, 24]],
+        "gat_dims": [24, 32],
+        "pool_ratios": [0.4, 0.5, 0.7, 0.5],
+        "temperatures": [2.0, 2.0, 100.0, 100.0]
+    },
+    "optim_config": {
+        "optimizer": "adam", 
+        "amsgrad": "False",
+        "base_lr": 0.0001,
+        "lr_min": 0.000005,
+        "betas": [0.9, 0.999],
+        "weight_decay": 0.0001,
+        "scheduler": "cosine"
+    }
+}

config/AASIST.conf

+{
+    "database_path": "./LA/",
+    "asv_score_path": "ASVspoof2019_LA_asv_scores/ASVspoof2019.LA.asv.eval.gi.trl.scores.txt",
+    "model_path": "./models/weights/AASIST.pth",
+    "batch_size": 24,
+    "num_epochs": 100,
+    "loss": "CCE",
+    "track": "LA",
+    "eval_all_best": "True",
+    "eval_output": "eval_scores_using_best_dev_model.txt",
+    "cudnn_deterministic_toggle": "True",
+    "cudnn_benchmark_toggle": "False",
+    "model_config": {
+        "architecture": "AASIST",
+        "nb_samp": 64600,
+        "first_conv": 128,
+        "filts": [70, [1, 32], [32, 32], [32, 64], [64, 64]],
+        "gat_dims": [64, 32],
+        "pool_ratios": [0.5, 0.7, 0.5, 0.5],
+        "temperatures": [2.0, 2.0, 100.0, 100.0]
+    },
+    "optim_config": {
+        "optimizer": "adam", 
+        "amsgrad": "False",
+        "base_lr": 0.0001,
+        "lr_min": 0.000005,
+        "betas": [0.9, 0.999],
+        "weight_decay": 0.0001,
+        "scheduler": "cosine"
+    }
+}

config/RawGATST_baseline.conf

+{
+    "database_path": "./LA/",
+    "asv_score_path": "ASVspoof2019_LA_asv_scores/ASVspoof2019.LA.asv.eval.gi.trl.scores.txt",
+    "model_path": "/home1/irteam/jeeweon/git/AsvSpoofDetection/exp_result/LAmodelRawNetGatSpoofST_ep100_bs24_lr0.0001/weights/epoch_12.pth",
+    "batch_size": 24,
+    "num_epochs": 100,
+    "loss": "CCE",
+    "track": "LA",
+    "eval_output": "eval_scores_using_best_dev_model.txt",
+    "cudnn_deterministic_toggle": "True",
+    "cudnn_benchmark_toggle": "False",
+    "model_config": {
+        "architecture": "RawNetGatSpoofST",
+        "nb_samp": 64600,
+        "first_conv": 128,
+        "filts": [70, [1, 32], [32, 32], [32, 64], [64, 64]]
+    },
+    "optim_config": {
+        "optimizer": "adam", 
+        "amsgrad": "False",
+        "base_lr": 0.0001,
+        "lr_min": 0.000005,
+        "betas": [0.9, 0.999],
+        "weight_decay": 0.0001,
+        "scheduler": "cosine"
+    }
+}

config/RawNet2_baseline.conf

+{
+    "database_path": "./LA/",
+    "asv_score_path": "ASVspoof2019_LA_asv_scores/ASVspoof2019.LA.asv.eval.gi.trl.scores.txt",
+    "model_path": "/home1/irteam/jeeweon/git/AsvSpoofDetection/exp_result/LAmodelRawNet2Spoof_ep100_bs32_lr0.0001/weights/best.pth",
+    "batch_size": 32,
+    "lr": 0.0001,
+    "weight_decay": 0.0001,
+    "num_epochs": 100,
+    "loss": "CCE",
+    "track": "LA",
+    "eval_output": "eval_scores_using_best_dev_model.txt",
+    "cudnn_deterministic_toggle": "True",
+    "cudnn_benchmark_toggle": "False",
+    "model_config": {
+        "architecture": "RawNet2Spoof",
+        "nb_samp": 64600,
+        "first_conv": 1024,
+        "in_channels": 1,
+        "filts": [20, [20, 20], [20, 128], [128, 128]],
+        "blocks": [2, 4],
+        "nb_fc_node": 1024,
+        "gru_node": 1024,
+        "nb_gru_layer": 3,
+        "nb_classes": 2
+    },
+    "optim_config": {
+        "optimizer": "adam", 
+        "amsgrad": "False",
+        "base_lr": 0.0001,
+        "lr_min": 0.000005,
+        "betas": [0.9, 0.999],
+        "weight_decay": 0.0001,
+        "scheduler": "cosine"
+    }
+}

data_utils.py

+import numpy as np
+import soundfile as sf
+import torch
+from torch import Tensor
+from torch.utils.data import Dataset
+
+___author__ = "Hemlata Tak, Jee-weon Jung"
+__email__ = "tak@eurecom.fr, jeeweon.jung@navercorp.com"
+
+
+def genSpoof_list(dir_meta, is_train=False, is_eval=False):
+
+    d_meta = {}
+    file_list = []
+    with open(dir_meta, "r") as f:
+        l_meta = f.readlines()
+
+    if is_train:
+        for line in l_meta:
+            _, key, _, _, label = line.strip().split(" ")
+            file_list.append(key)
+            d_meta[key] = 1 if label == "bonafide" else 0
+        return d_meta, file_list
+
+    elif is_eval:
+        for line in l_meta:
+            _, key, _, _, _ = line.strip().split(" ")
+            #key = line.strip()
+            file_list.append(key)
+        return file_list
+    else:
+        for line in l_meta:
+            _, key, _, _, label = line.strip().split(" ")
+            file_list.append(key)
+            d_meta[key] = 1 if label == "bonafide" else 0
+        return d_meta, file_list
+
+
+def pad(x, max_len=64600):
+    x_len = x.shape[0]
+    if x_len >= max_len:
+        return x[:max_len]
+    # need to pad
+    num_repeats = int(max_len / x_len) + 1
+    padded_x = np.tile(x, (1, num_repeats))[:, :max_len][0]
+    return padded_x
+
+
+def pad_random(x: np.ndarray, max_len: int = 64600):
+    x_len = x.shape[0]
+    # if duration is already long enough
+    if x_len >= max_len:
+        stt = np.random.randint(x_len - max_len)
+        return x[stt:stt + max_len]
+
+    # if too short
+    num_repeats = int(max_len / x_len) + 1
+    padded_x = np.tile(x, (num_repeats))[:max_len]
+    return padded_x
+
+
+class Dataset_ASVspoof2019_train(Dataset):
+    def __init__(self, list_IDs, labels, base_dir):
+        """self.list_IDs	: list of strings (each string: utt key),
+           self.labels      : dictionary (key: utt key, value: label integer)"""
+        self.list_IDs = list_IDs
+        self.labels = labels
+        self.base_dir = base_dir
+        self.cut = 64600  # take ~4 sec audio (64600 samples)
+
+    def __len__(self):
+        return len(self.list_IDs)
+
+    def __getitem__(self, index):
+        key = self.list_IDs[index]
+        X, _ = sf.read(str(self.base_dir / f"flac/{key}.flac"))
+        X_pad = pad_random(X, self.cut)
+        x_inp = Tensor(X_pad)
+        y = self.labels[key]
+        return x_inp, y
+
+
+class Dataset_ASVspoof2019_devNeval(Dataset):
+    def __init__(self, list_IDs, base_dir):
+        """self.list_IDs	: list of strings (each string: utt key),
+        """
+        self.list_IDs = list_IDs
+        self.base_dir = base_dir
+        self.cut = 64600  # take ~4 sec audio (64600 samples)
+
+    def __len__(self):
+        return len(self.list_IDs)
+
+    def __getitem__(self, index):
+        key = self.list_IDs[index]
+        X, _ = sf.read(str(self.base_dir / f"flac/{key}.flac"))
+        X_pad = pad(X, self.cut)
+        x_inp = Tensor(X_pad)
+        return x_inp, key

download_dataset.py

+"""
+AASIST
+Copyright (c) 2021-present NAVER Corp.
+MIT license
+"""
+
+import os
+
+if __name__ == "__main__":
+    cmd = "curl -o ./LA.zip -# https://datashare.ed.ac.uk/bitstream/handle/10283/3336/LA.zip\?sequence\=3\&isAllowed\=y"
+    os.system(cmd)
+    cmd = "unzip LA.zip"
+    os.system(cmd)
\ No newline at end of file

evaluation.py

+import sys
+import os
+
+import numpy as np
+
+
+def calculate_tDCF_EER(cm_scores_file,
+                       asv_score_file,
+                       output_file,
+                       printout=True):
+    # Replace CM scores with your own scores or provide score file as the
+    # first argument.
+    # cm_scores_file =  'score_cm.txt'
+    # Replace ASV scores with organizers' scores or provide score file as
+    # the second argument.
+    # asv_score_file = 'ASVspoof2019.LA.asv.eval.gi.trl.scores.txt'
+
+    # Fix tandem detection cost function (t-DCF) parameters
+    Pspoof = 0.05
+    cost_model = {
+        'Pspoof': Pspoof,  # Prior probability of a spoofing attack
+        'Ptar': (1 - Pspoof) * 0.99,  # Prior probability of target speaker
+        'Pnon': (1 - Pspoof) * 0.01,  # Prior probability of nontarget speaker
+        'Cmiss': 1,  # Cost of ASV system falsely rejecting target speaker
+        'Cfa': 10,  # Cost of ASV system falsely accepting nontarget speaker
+        'Cmiss_asv': 1,  # Cost of ASV system falsely rejecting target speaker
+        'Cfa_asv':
+        10,  # Cost of ASV system falsely accepting nontarget speaker
+        'Cmiss_cm': 1,  # Cost of CM system falsely rejecting target speaker
+        'Cfa_cm': 10,  # Cost of CM system falsely accepting spoof
+    }
+
+    # Load organizers' ASV scores
+    asv_data = np.genfromtxt(asv_score_file, dtype=str)
+    # asv_sources = asv_data[:, 0]
+    asv_keys = asv_data[:, 1]
+    asv_scores = asv_data[:, 2].astype(np.float)
+
+    # Load CM scores
+    cm_data = np.genfromtxt(cm_scores_file, dtype=str)
+    # cm_utt_id = cm_data[:, 0]
+    cm_sources = cm_data[:, 1]
+    cm_keys = cm_data[:, 2]
+    cm_scores = cm_data[:, 3].astype(np.float)
+
+    # Extract target, nontarget, and spoof scores from the ASV scores
+    tar_asv = asv_scores[asv_keys == 'target']
+    non_asv = asv_scores[asv_keys == 'nontarget']
+    spoof_asv = asv_scores[asv_keys == 'spoof']
+
+    # Extract bona fide (real human) and spoof scores from the CM scores
+    bona_cm = cm_scores[cm_keys == 'bonafide']
+    spoof_cm = cm_scores[cm_keys == 'spoof']
+
+    # EERs of the standalone systems and fix ASV operating point to
+    # EER threshold
+    eer_asv, asv_threshold = compute_eer(tar_asv, non_asv)
+    eer_cm = compute_eer(bona_cm, spoof_cm)[0]
+
+    attack_types = [f'A{_id:02d}' for _id in range(7, 20)]
+    if printout:
+        spoof_cm_breakdown = {
+            attack_type: cm_scores[cm_sources == attack_type]
+            for attack_type in attack_types
+        }
+
+        eer_cm_breakdown = {
+            attack_type: compute_eer(bona_cm,
+                                     spoof_cm_breakdown[attack_type])[0]
+            for attack_type in attack_types
+        }
+
+    [Pfa_asv, Pmiss_asv,
+     Pmiss_spoof_asv] = obtain_asv_error_rates(tar_asv, non_asv, spoof_asv,
+                                               asv_threshold)
+
+    # Compute t-DCF
+    tDCF_curve, CM_thresholds = compute_tDCF(bona_cm,
+                                             spoof_cm,
+                                             Pfa_asv,
+                                             Pmiss_asv,
+                                             Pmiss_spoof_asv,
+                                             cost_model,
+                                             print_cost=False)
+
+    # Minimum t-DCF
+    min_tDCF_index = np.argmin(tDCF_curve)
+    min_tDCF = tDCF_curve[min_tDCF_index]
+
+    if printout:
+        with open(output_file, "w") as f_res:
+            f_res.write('\nCM SYSTEM\n')
+            f_res.write('\tEER\t\t= {:8.9f} % '
+                        '(Equal error rate for countermeasure)\n'.format(
+                            eer_cm * 100))
+
+            f_res.write('\nTANDEM\n')
+            f_res.write('\tmin-tDCF\t\t= {:8.9f}\n'.format(min_tDCF))
+
+            f_res.write('\nBREAKDOWN CM SYSTEM\n')
+            for attack_type in attack_types:
+                _eer = eer_cm_breakdown[attack_type] * 100
+                f_res.write(
+                    f'\tEER {attack_type}\t\t= {_eer:8.9f} % (Equal error rate for {attack_type}\n'
+                )
+        os.system(f"cat {output_file}")
+
+    return eer_cm * 100, min_tDCF
+
+
+def obtain_asv_error_rates(tar_asv, non_asv, spoof_asv, asv_threshold):
+
+    # False alarm and miss rates for ASV
+    Pfa_asv = sum(non_asv >= asv_threshold) / non_asv.size
+    Pmiss_asv = sum(tar_asv < asv_threshold) / tar_asv.size
+
+    # Rate of rejecting spoofs in ASV
+    if spoof_asv.size == 0:
+        Pmiss_spoof_asv = None
+    else:
+        Pmiss_spoof_asv = np.sum(spoof_asv < asv_threshold) / spoof_asv.size
+
+    return Pfa_asv, Pmiss_asv, Pmiss_spoof_asv
+
+
+def compute_det_curve(target_scores, nontarget_scores):
+
+    n_scores = target_scores.size + nontarget_scores.size
+    all_scores = np.concatenate((target_scores, nontarget_scores))
+    labels = np.concatenate(
+        (np.ones(target_scores.size), np.zeros(nontarget_scores.size)))
+
+    # Sort labels based on scores
+    indices = np.argsort(all_scores, kind='mergesort')
+    labels = labels[indices]
+
+    # Compute false rejection and false acceptance rates
+    tar_trial_sums = np.cumsum(labels)
+    nontarget_trial_sums = nontarget_scores.size - \
+        (np.arange(1, n_scores + 1) - tar_trial_sums)
+
+    # false rejection rates
+    frr = np.concatenate(
+        (np.atleast_1d(0), tar_trial_sums / target_scores.size))
+    far = np.concatenate((np.atleast_1d(1), nontarget_trial_sums /
+                          nontarget_scores.size))  # false acceptance rates
+    # Thresholds are the sorted scores
+    thresholds = np.concatenate(
+        (np.atleast_1d(all_scores[indices[0]] - 0.001), all_scores[indices]))
+
+    return frr, far, thresholds
+
+
+def compute_eer(target_scores, nontarget_scores):
+    """ Returns equal error rate (EER) and the corresponding threshold. """
+    frr, far, thresholds = compute_det_curve(target_scores, nontarget_scores)
+    abs_diffs = np.abs(frr - far)
+    min_index = np.argmin(abs_diffs)
+    eer = np.mean((frr[min_index], far[min_index]))
+    return eer, thresholds[min_index]
+
+
+def compute_tDCF(bonafide_score_cm, spoof_score_cm, Pfa_asv, Pmiss_asv,
+                 Pmiss_spoof_asv, cost_model, print_cost):
+    """
+    Compute Tandem Detection Cost Function (t-DCF) [1] for a fixed ASV system.
+    In brief, t-DCF returns a detection cost of a cascaded system of this form,
+
+      Speech waveform -> [CM] -> [ASV] -> decision
+
+    where CM stands for countermeasure and ASV for automatic speaker
+    verification. The CM is therefore used as a 'gate' to decided whether or
+    not the input speech sample should be passed onwards to the ASV system.
+    Generally, both CM and ASV can do detection errors. Not all those errors
+    are necessarily equally cost, and not all types of users are necessarily
+    equally likely. The tandem t-DCF gives a principled with to compare
+    different spoofing countermeasures under a detection cost function
+    framework that takes that information into account.
+
+    INPUTS:
+
+      bonafide_score_cm   A vector of POSITIVE CLASS (bona fide or human)
+                          detection scores obtained by executing a spoofing
+                          countermeasure (CM) on some positive evaluation trials.
+                          trial represents a bona fide case.
+      spoof_score_cm      A vector of NEGATIVE CLASS (spoofing attack)
+                          detection scores obtained by executing a spoofing
+                          CM on some negative evaluation trials.
+      Pfa_asv             False alarm (false acceptance) rate of the ASV
+                          system that is evaluated in tandem with the CM.
+                          Assumed to be in fractions, not percentages.
+      Pmiss_asv           Miss (false rejection) rate of the ASV system that
+                          is evaluated in tandem with the spoofing CM.
+                          Assumed to be in fractions, not percentages.
+      Pmiss_spoof_asv     Miss rate of spoof samples of the ASV system that
+                          is evaluated in tandem with the spoofing CM. That
+                          is, the fraction of spoof samples that were
+                          rejected by the ASV system.
+      cost_model          A struct that contains the parameters of t-DCF,
+                          with the following fields.
+
+                          Ptar        Prior probability of target speaker.
+                          Pnon        Prior probability of nontarget speaker (zero-effort impostor)
+                          Psoof       Prior probability of spoofing attack.
+                          Cmiss_asv   Cost of ASV falsely rejecting target.
+                          Cfa_asv     Cost of ASV falsely accepting nontarget.
+                          Cmiss_cm    Cost of CM falsely rejecting target.
+                          Cfa_cm      Cost of CM falsely accepting spoof.
+
+      print_cost          Print a summary of the cost parameters and the
+                          implied t-DCF cost function?
+
+    OUTPUTS:
+
+      tDCF_norm           Normalized t-DCF curve across the different CM
+                          system operating points; see [2] for more details.
+                          Normalized t-DCF > 1 indicates a useless
+                          countermeasure (as the tandem system would do
+                          better without it). min(tDCF_norm) will be the
+                          minimum t-DCF used in ASVspoof 2019 [2].
+      CM_thresholds       Vector of same size as tDCF_norm corresponding to
+                          the CM threshold (operating point).
+
+    NOTE:
+    o     In relative terms, higher detection scores values are assumed to
+          indicate stronger support for the bona fide hypothesis.
+    o     You should provide real-valued soft scores, NOT hard decisions. The
+          recommendation is that the scores are log-likelihood ratios (LLRs)
+          from a bonafide-vs-spoof hypothesis based on some statistical model.
+          This, however, is NOT required. The scores can have arbitrary range
+          and scaling.
+    o     Pfa_asv, Pmiss_asv, Pmiss_spoof_asv are in fractions, not percentages.
+
+    References:
+
+      [1] T. Kinnunen, K.-A. Lee, H. Delgado, N. Evans, M. Todisco,
+          M. Sahidullah, J. Yamagishi, D.A. Reynolds: "t-DCF: a Detection
+          Cost Function for the Tandem Assessment of Spoofing Countermeasures
+          and Automatic Speaker Verification", Proc. Odyssey 2018: the
+          Speaker and Language Recognition Workshop, pp. 312--319, Les Sables d'Olonne,
+          France, June 2018 (https://www.isca-speech.org/archive/Odyssey_2018/pdfs/68.pdf)
+
+      [2] ASVspoof 2019 challenge evaluation plan
+          TODO: <add link>
+    """
+
+    # Sanity check of cost parameters
+    if cost_model['Cfa_asv'] < 0 or cost_model['Cmiss_asv'] < 0 or \
+            cost_model['Cfa_cm'] < 0 or cost_model['Cmiss_cm'] < 0:
+        print('WARNING: Usually the cost values should be positive!')
+
+    if cost_model['Ptar'] < 0 or cost_model['Pnon'] < 0 or cost_model['Pspoof'] < 0 or \
+            np.abs(cost_model['Ptar'] + cost_model['Pnon'] + cost_model['Pspoof'] - 1) > 1e-10:
+        sys.exit(
+            'ERROR: Your prior probabilities should be positive and sum up to one.'
+        )
+
+    # Unless we evaluate worst-case model, we need to have some spoof tests against asv
+    if Pmiss_spoof_asv is None:
+        sys.exit(
+            'ERROR: you should provide miss rate of spoof tests against your ASV system.'
+        )
+
+    # Sanity check of scores
+    combined_scores = np.concatenate((bonafide_score_cm, spoof_score_cm))
+    if np.isnan(combined_scores).any() or np.isinf(combined_scores).any():
+        sys.exit('ERROR: Your scores contain nan or inf.')
+
+    # Sanity check that inputs are scores and not decisions
+    n_uniq = np.unique(combined_scores).size
+    if n_uniq < 3:
+        sys.exit(
+            'ERROR: You should provide soft CM scores - not binary decisions')
+
+    # Obtain miss and false alarm rates of CM
+    Pmiss_cm, Pfa_cm, CM_thresholds = compute_det_curve(
+        bonafide_score_cm, spoof_score_cm)
+
+    # Constants - see ASVspoof 2019 evaluation plan
+    C1 = cost_model['Ptar'] * (cost_model['Cmiss_cm'] - cost_model['Cmiss_asv'] * Pmiss_asv) - \
+        cost_model['Pnon'] * cost_model['Cfa_asv'] * Pfa_asv
+    C2 = cost_model['Cfa_cm'] * cost_model['Pspoof'] * (1 - Pmiss_spoof_asv)
+
+    # Sanity check of the weights
+    if C1 < 0 or C2 < 0:
+        sys.exit(
+            'You should never see this error but I cannot evalute tDCF with negative weights - please check whether your ASV error rates are correctly computed?'
+        )
+
+    # Obtain t-DCF curve for all thresholds
+    tDCF = C1 * Pmiss_cm + C2 * Pfa_cm
+
+    # Normalized t-DCF
+    tDCF_norm = tDCF / np.minimum(C1, C2)
+
+    # Everything should be fine if reaching here.
+    if print_cost:
+
+        print('t-DCF evaluation from [Nbona={}, Nspoof={}] trials\n'.format(
+            bonafide_score_cm.size, spoof_score_cm.size))
+        print('t-DCF MODEL')
+        print('   Ptar         = {:8.5f} (Prior probability of target user)'.
+              format(cost_model['Ptar']))
+        print(
+            '   Pnon         = {:8.5f} (Prior probability of nontarget user)'.
+            format(cost_model['Pnon']))
+        print(
+            '   Pspoof       = {:8.5f} (Prior probability of spoofing attack)'.
+            format(cost_model['Pspoof']))
+        print(
+            '   Cfa_asv      = {:8.5f} (Cost of ASV falsely accepting a nontarget)'
+            .format(cost_model['Cfa_asv']))
+        print(
+            '   Cmiss_asv    = {:8.5f} (Cost of ASV falsely rejecting target speaker)'
+            .format(cost_model['Cmiss_asv']))
+        print(
+            '   Cfa_cm       = {:8.5f} (Cost of CM falsely passing a spoof to ASV system)'
+            .format(cost_model['Cfa_cm']))
+        print(
+            '   Cmiss_cm     = {:8.5f} (Cost of CM falsely blocking target utterance which never reaches ASV)'
+            .format(cost_model['Cmiss_cm']))
+        print(
+            '\n   Implied normalized t-DCF function (depends on t-DCF parameters and ASV errors), s=CM threshold)'
+        )
+
+        if C2 == np.minimum(C1, C2):
+            print(
+                '   tDCF_norm(s) = {:8.5f} x Pmiss_cm(s) + Pfa_cm(s)\n'.format(
+                    C1 / C2))
+        else:
+            print(
+                '   tDCF_norm(s) = Pmiss_cm(s) + {:8.5f} x Pfa_cm(s)\n'.format(
+                    C2 / C1))
+
+    return tDCF_norm, CM_thresholds

exp_result/LA_AASIST-L_ep100_bs24/config.conf

+{
+    "database_path": "./LA/",
+    "asv_score_path": "ASVspoof2019_LA_asv_scores/ASVspoof2019.LA.asv.eval.gi.trl.scores.txt",
+    "model_path": "./models/weights/AASIST-L.pth",
+    "batch_size": 24,
+    "num_epochs": 100,
+    "loss": "CCE",
+    "track": "LA",
+    "eval_all_best": "True",
+    "eval_output": "eval_scores_using_best_dev_model.txt",
+    "cudnn_deterministic_toggle": "True",
+    "cudnn_benchmark_toggle": "False",
+    "model_config": {
+        "architecture": "AASIST",
+        "nb_samp": 64600,
+        "first_conv": 128,
+        "filts": [70, [1, 32], [32, 32], [32, 24], [24, 24]],
+        "gat_dims": [24, 32],
+        "pool_ratios": [0.4, 0.5, 0.7, 0.5],
+        "temperatures": [2.0, 2.0, 100.0, 100.0]
+    },
+    "optim_config": {
+        "optimizer": "adam", 
+        "amsgrad": "False",
+        "base_lr": 0.0001,
+        "lr_min": 0.000005,
+        "betas": [0.9, 0.999],
+        "weight_decay": 0.0001,
+        "scheduler": "cosine"
+    }
+}