# Initial result: single channel, no front end, 8ch simulation data + WSJ SI284 training
# According to the challenge results (https://reverb2014.dereverberation.com/result_asr.html)
# The simulation data results below seem to be very good while the real data results are significantly
# worse than those reported including the official HTK baseline.
# The end-to-end system tends to be over-tuned only with the training conditions.
#
# Results after adding WPE & BeamformIt: single channel, 2 channel and 8 channel, 8ch simulation data (no enhanced data in training) + WSJ SI284 training
# Comparing to the challenge results (https://reverb2014.dereverberation.com/result_asr.html),
# 8ch with WPE+BeamformIt during evaluation for both real and simulation is better than some submissions and comparable to other submissions
# Performance of Delcroix et al. is better than our result but they seem to be using their own dataset for acoustic model.
#
# ./local/score_for_reverb.sh --wer true --nlsyms data/lang_1char/non_lang_syms.txt 'exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_*_1ch_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000/data.json' data/lang_1char/tr_simu_8ch_si284_units.txt exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_summary_1ch_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000

RESULTS - 1ch - No Front End
----------------------------

SimData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.6    5.5     0.9     1.1     7.5     51.5    |
SimData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       92.7    6.2     1.1     1.1     8.3     57.0    |
SimData_et_near_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       92.4    6.0     1.5     0.8     8.3     57.6    |
SimData_et_far_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       87.4    10.3    2.3     1.2     13.8    74.9    |
SimData_et_near_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       90.6    7.8     1.6     1.1     10.5    60.8    |
SimData_et_far_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       85.4    12.7    1.9     2.1     16.7    75.4    |
RealData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     3131    |       44.8    44.9    10.3    5.2     60.4    95.2    |
RealData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     2962    |       47.4    41.8    10.8    5.1     57.7    94.6    |
########################################

# ./local/score_for_reverb.sh --wer true --nlsyms data/lang_1char/non_lang_syms.txt 'exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_*_1ch_wpe_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000/data.json' data/lang_1char/tr_simu_8ch_si284_units.txt exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_summary_1ch_wpe_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000

RESULTS - 1ch - WPE
----------------------------

SimData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.7    5.3     1.0     1.0     7.3     51.8    |
SimData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.4    5.6     1.0     1.0     7.6     53.2    |
SimData_et_near_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       92.5    6.0     1.5     0.7     8.2     58.7    |
SimData_et_far_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       88.6    9.3     2.0     1.2     12.6    74.7    |
SimData_et_near_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       91.3    7.3     1.4     1.0     9.7     59.9    |
SimData_et_far_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       86.5    11.7    1.8     2.1     15.6    73.8    |
RealData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     3131    |       50.6    40.2    9.2     5.7     55.2    93.0    |
RealData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     2962    |       51.9    38.3    9.8     4.9     53.0    93.0    |
########################################

# ./local/score_for_reverb.sh --wer true --nlsyms data/lang_1char/non_lang_syms.txt 'exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_*_2ch_wpe_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000/data.json' data/lang_1char/tr_simu_8ch_si284_units.txt exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_summary_2ch_wpe_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000

RESULTS - 2ch - WPE
----------------------------

SimData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.6    5.4     1.0     1.1     7.5     51.5    |
SimData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.6    5.3     1.0     1.1     7.4     52.6    |
SimData_et_near_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       92.4    6.2     1.4     0.7     8.4     58.1    |
SimData_et_far_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       89.7    8.7     1.6     1.2     11.5    68.6    |
SimData_et_near_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       91.6    7.1     1.3     1.1     9.5     60.2    |
SimData_et_far_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       87.1    11.5    1.3     2.4     15.2    71.5    |
RealData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     3131    |       56.8    35.1    8.1     5.4     48.6    93.0    |
RealData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     2962    |       58.2    33.1    8.6     4.6     46.4    90.9    |
########################################

# ./local/score_for_reverb.sh --wer true --nlsyms data/lang_1char/non_lang_syms.txt 'exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_*_2ch_beamformit_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000/data.json' data/lang_1char/tr_simu_8ch_si284_units.txt exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_summary_2ch_beamformit_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000

RESULTS - 2ch - WPE+BeamformIt
----------------------------

SimData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.7    5.4     0.9     1.1     7.4     51.0    |
SimData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.3    5.8     0.9     1.1     7.8     51.8    |
SimData_et_near_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       92.9    5.8     1.4     0.8     7.9     57.3    |
SimData_et_far_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       90.7    7.7     1.6     1.2     10.5    66.4    |
SimData_et_near_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       91.8    7.0     1.2     1.3     9.5     58.0    |
SimData_et_far_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       88.1    10.6    1.3     2.1     13.9    73.2    |
RealData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     3131    |       75.1    22.9    2.0     5.4     30.3    84.9    |
RealData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     2962    |       73.7    23.1    3.2     4.9     31.2    82.3    |
########################################

# ./local/score_for_reverb.sh --wer true --nlsyms data/lang_1char/non_lang_syms.txt 'exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_*_8ch_wpe_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000/data.json' data/lang_1char/tr_simu_8ch_si284_units.txt exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_summary_8ch_wpe_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000

RESULTS - 8ch - WPE
----------------------------

SimData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.7    5.3     1.0     1.1     7.4     51.0    |
SimData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.4    5.4     1.1     1.1     7.7     52.1    |
SimData_et_near_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       92.9    5.7     1.4     0.6     7.7     56.7    |
SimData_et_far_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       91.5    7.1     1.4     1.0     9.6     62.8    |
SimData_et_near_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       92.8    6.0     1.2     0.8     8.0     56.1    |
SimData_et_far_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       91.6    7.0     1.4     1.0     9.4     60.5    |
RealData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     3131    |       65.3    28.0    6.7     4.8     39.5    88.2    |
RealData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     2962    |       68.7    24.7    6.6     4.1     35.4    81.7    |
########################################

# ./local/score_for_reverb.sh --wer true --nlsyms data/lang_1char/non_lang_syms.txt 'exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_*_8ch_beamformit_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000/data.json' data/lang_1char/tr_simu_8ch_si284_units.txt exp/tr_simu_8ch_si284_chainer_vggblstm_e3_subsample1_2_2_1_1_unit1024_proj1024_d1_unit1024_location1024_aconvc10_aconvf100_mtlalpha0.5_adadelta_bs32_mli600_mlo150_ng2/decode_summary_8ch_beamformit_beam30_emodel.acc.best_p0_len0.0-0.0_ctcw0.3_wordrnnlm1.0_1layer_unit1000_sgd_bs300_word65000

RESULTS - 8ch - WPE+BeamformIt
----------------------------

SimData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.8    5.2     1.0     0.9     7.1     49.0    |
SimData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.2    5.7     1.0     1.0     7.8     50.1    |
SimData_et_near_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       93.6    5.2     1.2     0.6     7.1     55.6    |
SimData_et_far_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       93.0    5.7     1.3     0.8     7.7     58.7    |
SimData_et_near_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       93.2    5.6     1.2     0.9     7.6     54.4    |
SimData_et_far_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       93.1    5.7     1.2     0.9     7.8     53.9    |
RealData_et_near_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     3131    |       82.8    15.5    1.7     3.7     20.9    78.5    |
RealData_et_far_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       186     2962    |       81.5    16.2    2.3     3.1     21.6    78.0    |
########################################

RESULTS - Cln
----------------------------

SimData_et_cln_room1:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     5904    |       93.6    5.4     1.0     0.7     7.1     50.4    |
SimData_et_cln_room2:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       363     6223    |       93.0    5.7     1.2     0.6     7.6     52.6    |
SimData_et_cln_room3:
|       SPKR    |       # Snt   # Wrd   |       Corr    Sub     Del     Ins     Err     S.Err   |
|       Sum/Avg |       362     5863    |       93.2    5.7     1.1     0.6     7.5     50.3    |
########################################

########################################
SE Scores - 8ch - WPE+BeamformIt
########################################

Data type   : SimData
----------------------------

==============================================
           Cepstral distance in dB
----------------------------------------------
            	      	  mean	      	median
----------------------------------------------
            	   org	   enh	   org	   enh
----------------------------------------------
  dt_far_room1	  2.65	  1.97	  2.36	  1.74
  dt_far_room2	  5.08	  4.66	  4.94	  4.30
  dt_far_room3	  4.82	  4.03	  4.60	  3.63
 dt_near_room1	  1.96	  1.67	  1.67	  1.37
 dt_near_room2	  4.58	  4.33	  4.30	  3.88
 dt_near_room3	  4.20	  3.71	  3.91	  3.26
----------------------------------------------
       average	  3.88	  3.39	  3.63	  3.03
==============================================


==============================================
            SRMR  (only mean used)
----------------------------------------------
            	      	  mean	      	median
----------------------------------------------
            	   org	   enh	   org	   enh
----------------------------------------------
  dt_far_room1	  4.63	  4.91	     -	     -
  dt_far_room2	  2.94	  5.13	     -	     -
  dt_far_room3	  2.76	  4.87	     -	     -
 dt_near_room1	  4.37	  4.62	     -	     -
 dt_near_room2	  3.67	  4.39	     -	     -
 dt_near_room3	  3.66	  4.54	     -	     -
----------------------------------------------
       average	  3.67	  4.74	     -	     -
==============================================


==============================================
             Log likelihood ratio
----------------------------------------------
            	      	  mean	      	median
----------------------------------------------
            	   org	   enh	   org	   enh
----------------------------------------------
  dt_far_room1	  0.38	  0.33	  0.35	  0.30
  dt_far_room2	  0.77	  0.56	  0.64	  0.43
  dt_far_room3	  0.85	  0.52	  0.77	  0.45
 dt_near_room1	  0.34	  0.34	  0.33	  0.32
 dt_near_room2	  0.51	  0.50	  0.43	  0.33
 dt_near_room3	  0.65	  0.50	  0.59	  0.43
----------------------------------------------
       average	  0.58	  0.46	  0.52	  0.38
==============================================


==============================================
    Frequency-weighted segmental SNR in dB
----------------------------------------------
            	      	  mean	      	median
----------------------------------------------
            	   org	   enh	   org	   enh
----------------------------------------------
  dt_far_room1	  6.75	  8.99	  8.93	 11.06
  dt_far_room2	  0.53	  3.84	  0.37	  5.91
  dt_far_room3	  0.14	  3.76	  0.39	  6.57
 dt_near_room1	  8.10	  9.50	 10.47	 11.32
 dt_near_room2	  3.07	  5.10	  4.58	  8.12
 dt_near_room3	  2.32	  4.54	  4.41	  8.15
----------------------------------------------
       average	  3.48	  5.96	  4.86	  8.52
==============================================

Data type   : RealData
----------------------------

==============================
            SRMR
------------------------------
            	   org	   enh
------------------------------
  dt_far_room1	  3.51	  6.03
 dt_near_room1	  4.05	  6.68
------------------------------
       average	  3.78	  6.36
==============================
