Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
chenpangpang
transformers
Commits
86896de0
Unverified
Commit
86896de0
authored
Dec 11, 2020
by
Suraj Patil
Committed by
GitHub
Dec 11, 2020
Browse files
update tatoeba workflow (#9051)
parent
7c8f5f64
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
28 additions
and
4 deletions
+28
-4
scripts/tatoeba/README.md
scripts/tatoeba/README.md
+16
-4
scripts/tatoeba/upload_models.sh
scripts/tatoeba/upload_models.sh
+12
-0
No files found.
scripts/tatoeba/README.md
View file @
86896de0
...
...
@@ -19,7 +19,7 @@ Setup transformers following instructions in README.md, (I would fork first).
git clone git@github.com:huggingface/transformers.git
cd
transformers
pip
install
-e
.
pip
install
pandas
pip
install
pandas
GitPython wget
```
Get required metadata
...
...
@@ -35,7 +35,7 @@ git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git
To convert a few models, call the conversion script from command line:
```
bash
python src/transformers/convert_marian_tatoeba_to_pytorch.py
--models
heb-eng eng-heb
--save_dir
converted
python src/transformers/
models/marian/
convert_marian_tatoeba_to_pytorch.py
--models
heb-eng eng-heb
--save_dir
converted
```
To convert lots of models you can pass your list of Tatoeba model names to
`resolver.convert_models`
in a python client or script.
...
...
@@ -48,10 +48,22 @@ resolver.convert_models(['heb-eng', 'eng-heb'])
### Upload converted models
Since version v3.5.0, the model sharing workflow is switched to git-based system . Refer to
[
model sharing doc
](
https://huggingface.co/transformers/master/model_sharing.html#model-sharing-and-uploading
)
for more details.
To upload all converted models,
1.
Install
[
git-lfs
](
https://git-lfs.github.com/
)
.
2.
Login to
`transformers-cli`
```
bash
cd
converted
transformers-cli login
for
FILE
in
*
;
do
transformers-cli upload
$FILE
;
done
```
3.
Run the
`upload_models`
script
```
bash
./scripts/tatoeba/upload_models.sh
```
...
...
scripts/tatoeba/upload_models.sh
0 → 100755
View file @
86896de0
#!/bin/bash
for
FILE
in
converted/
*
;
do
model_name
=
`
basename
$FILE
`
transformers-cli repo create
$model_name
-y
git clone https://huggingface.co/Helsinki-NLP/
$model_name
mv
$FILE
/
*
$model_name
/
cd
$model_name
git add
.
&&
git commit
-m
"initial commit"
git push
cd
..
done
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment