add fairseq0.10.2

7df61696 · Sugon_ldc · 7df61696 · 7df61696 · 7df61696 · 7df61696
Commit 7df61696 authored Jul 28, 2023 by Sugon_ldc
20 changed files
--- a/examples/m2m_100/README.md
+++ b/examples/m2m_100/README.md
--- a/examples/m2m_100/install_dependecies.sh
+++ b/examples/m2m_100/install_dependecies.sh
--- a/examples/m2m_100/process_data/clean_histogram.py
+++ b/examples/m2m_100/process_data/clean_histogram.py
--- a/examples/m2m_100/process_data/dedup_data.py
+++ b/examples/m2m_100/process_data/dedup_data.py
--- a/examples/m2m_100/process_data/remove_too_much_punc.py
+++ b/examples/m2m_100/process_data/remove_too_much_punc.py
--- a/examples/m2m_100/tok.sh
+++ b/examples/m2m_100/tok.sh
--- a/examples/m2m_100/tokenizers/README.md
+++ b/examples/m2m_100/tokenizers/README.md
--- a/examples/m2m_100/tokenizers/seg_ja.sh
+++ b/examples/m2m_100/tokenizers/seg_ja.sh
--- a/examples/m2m_100/tokenizers/seg_ko.sh
+++ b/examples/m2m_100/tokenizers/seg_ko.sh
--- a/examples/m2m_100/tokenizers/thirdparty/.gitignore
+++ b/examples/m2m_100/tokenizers/thirdparty/.gitignore
--- a/examples/m2m_100/tokenizers/tokenize_indic.py
+++ b/examples/m2m_100/tokenizers/tokenize_indic.py
--- a/examples/m2m_100/tokenizers/tokenize_thai.py
+++ b/examples/m2m_100/tokenizers/tokenize_thai.py
--- a/examples/m2m_100/tokenizers/tokenize_zh.py
+++ b/examples/m2m_100/tokenizers/tokenize_zh.py
--- a/examples/m2m_100/tokenizers/tokenizer_ar.sh
+++ b/examples/m2m_100/tokenizers/tokenizer_ar.sh
--- a/examples/mbart/README.md
+++ b/examples/mbart/README.md
--- a/examples/megatron_11b/README.md
+++ b/examples/megatron_11b/README.md
--- a/examples/megatron_11b/detok.py
+++ b/examples/megatron_11b/detok.py
--- a/examples/multilingual/README.md
+++ b/examples/multilingual/README.md
--- a/examples/multilingual/finetune_multilingual_model.sh
+++ b/examples/multilingual/finetune_multilingual_model.sh
--- a/examples/multilingual/multilingual_fairseq_gen.sh
+++ b/examples/multilingual/multilingual_fairseq_gen.sh