Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
README.md 4.98 KB
Newer Older
Zane Lim's avatar
Add model card for singbert lite. Update widget for singbert and singbert-large. (#6827)  
Zane Lim committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168 
---
language: en
tags:
- singapore
- sg
- singlish
- malaysia
- ms
- manglish
- albert-base-v2
license: mit
datasets:
- reddit singapore, malaysia
- hardwarezone
widget:
- text: "dont play [MASK] leh"
- text: "die [MASK] must try"
---

# Model name

SingBert Lite - Bert for Singlish (SG) and Manglish (MY).

## Model description

Similar to [SingBert](https://huggingface.co/zanelim/singbert) but the lite-version, which was initialized from [Albert base v2](https://github.com/google-research/albert#albert), with pre-training finetuned on
[singlish](https://en.wikipedia.org/wiki/Singlish) and [manglish](https://en.wikipedia.org/wiki/Manglish) data.

## Intended uses & limitations

#### How to use

```python
>>> from transformers import pipeline
>>> nlp = pipeline('fill-mask', model='zanelim/singbert-lite-sg')
>>> nlp("die [MASK] must try")

[{'sequence': '[CLS] die die must try[SEP]',
  'score': 0.7731555700302124,
  'token': 1327,
  'token_str': '鈻乨ie'},
 {'sequence': '[CLS] die also must try[SEP]',
  'score': 0.04763784259557724,
  'token': 67,
  'token_str': '鈻乤lso'},
 {'sequence': '[CLS] die still must try[SEP]',
  'score': 0.01859409362077713,
  'token': 174,
  'token_str': '鈻乻till'},
 {'sequence': '[CLS] die u must try[SEP]',
  'score': 0.015824034810066223,
  'token': 287,
  'token_str': '鈻乽'},
 {'sequence': '[CLS] die is must try[SEP]',
  'score': 0.011271446943283081,
  'token': 25,
  'token_str': '鈻乮s'}]

>>> nlp("dont play [MASK] leh")

[{'sequence': '[CLS] dont play play leh[SEP]',
  'score': 0.4365769624710083,
  'token': 418,
  'token_str': '鈻乸lay'},
 {'sequence': '[CLS] dont play punk leh[SEP]',
  'score': 0.06880936771631241,
  'token': 6769,
  'token_str': '鈻乸unk'},
 {'sequence': '[CLS] dont play game leh[SEP]',
  'score': 0.051739856600761414,
  'token': 250,
  'token_str': '鈻乬ame'},
 {'sequence': '[CLS] dont play games leh[SEP]',
  'score': 0.045703962445259094,
  'token': 466,
  'token_str': '鈻乬ames'},
 {'sequence': '[CLS] dont play around leh[SEP]',
  'score': 0.013458190485835075,
  'token': 140,
  'token_str': '鈻乤round'}]

>>> nlp("catch no [MASK]")

[{'sequence': '[CLS] catch no ball[SEP]',
  'score': 0.6197211146354675,
  'token': 1592,
  'token_str': '鈻乥all'},
 {'sequence': '[CLS] catch no balls[SEP]',
  'score': 0.08441998809576035,
  'token': 7152,
  'token_str': '鈻乥alls'},
 {'sequence': '[CLS] catch no joke[SEP]',
  'score': 0.0676785409450531,
  'token': 8186,
  'token_str': '鈻乯oke'},
 {'sequence': '[CLS] catch no?[SEP]',
  'score': 0.040638409554958344,
  'token': 60,
  'token_str': '?'},
 {'sequence': '[CLS] catch no one[SEP]',
  'score': 0.03546864539384842,
  'token': 53,
  'token_str': '鈻乷ne'}]

>>> nlp("confirm plus [MASK]")

[{'sequence': '[CLS] confirm plus chop[SEP]',
  'score': 0.9608421921730042,
  'token': 17144,
  'token_str': '鈻乧hop'},
 {'sequence': '[CLS] confirm plus guarantee[SEP]',
  'score': 0.011784233152866364,
  'token': 9120,
  'token_str': '鈻乬uarantee'},
 {'sequence': '[CLS] confirm plus confirm[SEP]',
  'score': 0.010571340098977089,
  'token': 10265,
  'token_str': '鈻乧onfirm'},
 {'sequence': '[CLS] confirm plus egg[SEP]',
  'score': 0.0033525123726576567,
  'token': 6387,
  'token_str': '鈻乪gg'},
 {'sequence': '[CLS] confirm plus bet[SEP]',
  'score': 0.0008760977652855217,
  'token': 5676,
  'token_str': '鈻乥et'}]

```

Here is how to use this model to get the features of a given text in PyTorch:
```python
from transformers import AlbertTokenizer, AlbertModel
tokenizer = AlbertTokenizer.from_pretrained('zanelim/singbert-lite-sg')
model = AlbertModel.from_pretrained("zanelim/singbert-lite-sg")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
```

and in TensorFlow:
```python
from transformers import AlbertTokenizer, TFAlbertModel
tokenizer = AlbertTokenizer.from_pretrained("zanelim/singbert-lite-sg")
model = TFAlbertModel.from_pretrained("zanelim/singbert-lite-sg")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
```

#### Limitations and bias
This model was finetuned on colloquial Singlish and Manglish corpus, hence it is best applied on downstream tasks involving the main
constituent languages- english, mandarin, malay. Also, as the training data is mainly from forums, beware of existing inherent bias.

## Training data
Colloquial singlish and manglish (both are a mixture of English, Mandarin, Tamil, Malay, and other local dialects like Hokkien, Cantonese or Teochew)
corpus. The corpus is collected from subreddits- `r/singapore` and `r/malaysia`, and forums such as `hardwarezone`.

## Training procedure

Initialized with [albert base v2](https://github.com/google-research/albert#albert) vocab and checkpoints (pre-trained weights).

Pre-training was further finetuned on training data with the following hyperparameters
* train_batch_size: 4096
* max_seq_length: 128
* num_train_steps: 125000
* num_warmup_steps: 5000
* learning_rate: 0.00176
* hardware: TPU v3-8