Unverified Commit 86896de0 authored by Suraj Patil's avatar Suraj Patil Committed by GitHub
Browse files

update tatoeba workflow (#9051)

parent 7c8f5f64
......@@ -19,7 +19,7 @@ Setup transformers following instructions in README.md, (I would fork first).
git clone git@github.com:huggingface/transformers.git
cd transformers
pip install -e .
pip install pandas
pip install pandas GitPython wget
```
Get required metadata
......@@ -35,7 +35,7 @@ git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git
To convert a few models, call the conversion script from command line:
```bash
python src/transformers/convert_marian_tatoeba_to_pytorch.py --models heb-eng eng-heb --save_dir converted
python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models heb-eng eng-heb --save_dir converted
```
To convert lots of models you can pass your list of Tatoeba model names to `resolver.convert_models` in a python client or script.
......@@ -48,10 +48,22 @@ resolver.convert_models(['heb-eng', 'eng-heb'])
### Upload converted models
Since version v3.5.0, the model sharing workflow is switched to git-based system . Refer to [model sharing doc](https://huggingface.co/transformers/master/model_sharing.html#model-sharing-and-uploading) for more details.
To upload all converted models,
1. Install [git-lfs](https://git-lfs.github.com/).
2. Login to `transformers-cli`
```bash
cd converted
transformers-cli login
for FILE in *; do transformers-cli upload $FILE; done
```
3. Run the `upload_models` script
```bash
./scripts/tatoeba/upload_models.sh
```
......
#!/bin/bash
for FILE in converted/*; do
model_name=`basename $FILE`
transformers-cli repo create $model_name -y
git clone https://huggingface.co/Helsinki-NLP/$model_name
mv $FILE/* $model_name/
cd $model_name
git add . && git commit -m "initial commit"
git push
cd ..
done
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment