README.md 2.33 KB
Newer Older
Sylvain Gugger's avatar
Sylvain Gugger committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
<!---
Copyright 2020 The HuggingFace Team. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

17
18
19
20
21
Setup transformers following instructions in README.md, (I would fork first).
```bash
git clone git@github.com:huggingface/transformers.git
cd transformers
pip install -e .
Suraj Patil's avatar
Suraj Patil committed
22
pip install pandas GitPython wget
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
```

Get required metadata
```
curl https://cdn-datasets.huggingface.co/language_codes/language-codes-3b2.csv  > language-codes-3b2.csv
curl https://cdn-datasets.huggingface.co/language_codes/iso-639-3.csv > iso-639-3.csv
```

Install Tatoeba-Challenge repo inside transformers
```bash
git clone git@github.com:Helsinki-NLP/Tatoeba-Challenge.git
```

To convert a few models, call the conversion script from command line:
```bash
Suraj Patil's avatar
Suraj Patil committed
38
python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models heb-eng eng-heb --save_dir converted
39
40
41
42
43
44
45
46
47
48
49
50
```

To convert lots of models you can pass your list of Tatoeba model names to `resolver.convert_models` in a python client or script.

```python
from transformers.convert_marian_tatoeba_to_pytorch import TatoebaConverter
resolver = TatoebaConverter(save_dir='converted')
resolver.convert_models(['heb-eng', 'eng-heb'])
```


### Upload converted models
51
Since version v3.5.0, the model sharing workflow is switched to git-based system . Refer to [model sharing doc](https://huggingface.co/transformers/main/model_sharing.html#model-sharing-and-uploading) for more details.
Suraj Patil's avatar
Suraj Patil committed
52
53
54
55
56
57
58

To upload all converted models, 

1. Install [git-lfs](https://git-lfs.github.com/).

2. Login to `transformers-cli`

59
60
```bash
transformers-cli login
Suraj Patil's avatar
Suraj Patil committed
61
62
63
64
65
66
```

3. Run the `upload_models` script

```bash
./scripts/tatoeba/upload_models.sh
67
68
69
70
71
72
```


### Modifications
- To change naming logic, change the code near `os.rename`. The model card creation code may also need to change.
- To change model card content, you must modify `TatoebaCodeResolver.write_model_card`