update uni-fold

a1c29028 · zhangqha · a1c29028 · a1c29028 · a1c29028 · a1c29028
Commit a1c29028 authored Apr 17, 2023 by zhangqha
20 changed files
--- a/Uni-Core-main/examples/bert/example_data/README.md
+++ b/Uni-Core-main/examples/bert/example_data/README.md
+## A simple BERT example
+
+1. download data `wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip & unzip wikitext-2-v1.zip`
+2. run `python preprocess.py ./wikitext-2/wiki.train.tokens ./train.lmdb`
+3. run `python preprocess.py ./wikitext-2/wiki.valid.tokens ./valid.lmdb`
\ No newline at end of file
--- a/Uni-Core-main/examples/bert/example_data/dict.txt
+++ b/Uni-Core-main/examples/bert/example_data/dict.txt
--- a/Uni-Core-main/examples/bert/example_data/preprocess.py
+++ b/Uni-Core-main/examples/bert/example_data/preprocess.py
--- a/Uni-Core-main/examples/bert/model.py
+++ b/Uni-Core-main/examples/bert/model.py
--- a/Uni-Core-main/examples/bert/task.py
+++ b/Uni-Core-main/examples/bert/task.py
--- a/Uni-Core-main/examples/bert/train_bert_test.sh
+++ b/Uni-Core-main/examples/bert/train_bert_test.sh
--- a/Uni-Core-main/requirements.txt
+++ b/Uni-Core-main/requirements.txt
--- a/Uni-Core-main/setup.py
+++ b/Uni-Core-main/setup.py
--- a/Uni-Core-main/tests/test_softmax.py
+++ b/Uni-Core-main/tests/test_softmax.py
--- a/Uni-Core-main/unicore/__init__.py
+++ b/Uni-Core-main/unicore/__init__.py
--- a/Uni-Core-main/unicore/checkpoint_utils.py
+++ b/Uni-Core-main/unicore/checkpoint_utils.py
--- a/Uni-Core-main/unicore/data/__init__.py
+++ b/Uni-Core-main/unicore/data/__init__.py
--- a/Uni-Core-main/unicore/data/append_token_dataset.py
+++ b/Uni-Core-main/unicore/data/append_token_dataset.py
--- a/Uni-Core-main/unicore/data/base_wrapper_dataset.py
+++ b/Uni-Core-main/unicore/data/base_wrapper_dataset.py
--- a/Uni-Core-main/unicore/data/bert_tokenize_dataset.py
+++ b/Uni-Core-main/unicore/data/bert_tokenize_dataset.py
--- a/Uni-Core-main/unicore/data/data_utils.py
+++ b/Uni-Core-main/unicore/data/data_utils.py
--- a/Uni-Core-main/unicore/data/dictionary.py
+++ b/Uni-Core-main/unicore/data/dictionary.py
--- a/Uni-Core-main/unicore/data/from_numpy_dataset.py
+++ b/Uni-Core-main/unicore/data/from_numpy_dataset.py
--- a/Uni-Core-main/unicore/data/iterators.py
+++ b/Uni-Core-main/unicore/data/iterators.py
--- a/Uni-Core-main/unicore/data/lmdb_dataset.py
+++ b/Uni-Core-main/unicore/data/lmdb_dataset.py