Unverified Commit 86896de0 authored by Suraj Patil's avatar Suraj Patil Committed by GitHub
Browse files

update tatoeba workflow (#9051)

parent 7c8f5f64
...@@ -19,7 +19,7 @@ Setup transformers following instructions in README.md, (I would fork first). ...@@ -19,7 +19,7 @@ Setup transformers following instructions in README.md, (I would fork first).
git clone git@github.com:huggingface/transformers.git git clone git@github.com:huggingface/transformers.git
cd transformers cd transformers
pip install -e . pip install -e .
pip install pandas pip install pandas GitPython wget
``` ```
Get required metadata Get required metadata
...@@ -35,7 +35,7 @@ git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git ...@@ -35,7 +35,7 @@ git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git
To convert a few models, call the conversion script from command line: To convert a few models, call the conversion script from command line:
```bash ```bash
python src/transformers/convert_marian_tatoeba_to_pytorch.py --models heb-eng eng-heb --save_dir converted python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models heb-eng eng-heb --save_dir converted
``` ```
To convert lots of models you can pass your list of Tatoeba model names to `resolver.convert_models` in a python client or script. To convert lots of models you can pass your list of Tatoeba model names to `resolver.convert_models` in a python client or script.
...@@ -48,10 +48,22 @@ resolver.convert_models(['heb-eng', 'eng-heb']) ...@@ -48,10 +48,22 @@ resolver.convert_models(['heb-eng', 'eng-heb'])
### Upload converted models ### Upload converted models
Since version v3.5.0, the model sharing workflow is switched to git-based system . Refer to [model sharing doc](https://huggingface.co/transformers/master/model_sharing.html#model-sharing-and-uploading) for more details.
To upload all converted models,
1. Install [git-lfs](https://git-lfs.github.com/).
2. Login to `transformers-cli`
```bash ```bash
cd converted
transformers-cli login transformers-cli login
for FILE in *; do transformers-cli upload $FILE; done ```
3. Run the `upload_models` script
```bash
./scripts/tatoeba/upload_models.sh
``` ```
......
#!/bin/bash
for FILE in converted/*; do
model_name=`basename $FILE`
transformers-cli repo create $model_name -y
git clone https://huggingface.co/Helsinki-NLP/$model_name
mv $FILE/* $model_name/
cd $model_name
git add . && git commit -m "initial commit"
git push
cd ..
done
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment