Commits · 31ca3b97ebc1ca37b1d4db6ff3bf062fcbf16b5d · ModelZoo / ResNet50_tensorflow

21 Jul, 2020 2 commits

[MultiheadAttention] Apply suggestions from RFC: · 36101ab4

Hongkun Yu authored Jul 21, 2020

(1) call() consume kwargs and implement _build_from_signature (layers added inside init_scope)
(2) make build/call_attention as public.

PiperOrigin-RevId: 322454619

36101ab4

Internal change · 391c9dd6
A. Unique TensorFlower authored Jul 21, 2020
```
PiperOrigin-RevId: 322428923
```
391c9dd6

20 Jul, 2020 1 commit
- Internal change · cdda0906
  Allen Wang authored Jul 20, 2020
```
PiperOrigin-RevId: 322216928
```
  cdda0906
08 Jul, 2020 1 commit
- Migrate all DenseEinsum to tf.keras.experimental.EinsumDense · 43f5340f
  Hongkun Yu authored Jul 08, 2020
```
PiperOrigin-RevId: 320240466
```
  43f5340f
25 Jun, 2020 1 commit
- Multichannel attention: Override _build_attention method instead of build() · 4140da21
  Hongkun Yu authored Jun 24, 2020
```
PiperOrigin-RevId: 318208409
```
  4140da21
22 Jun, 2020 1 commit
- Internal change · 8aa44501
  Hongkun Yu authored Jun 21, 2020
```
PiperOrigin-RevId: 317596394
```
  8aa44501
19 Jun, 2020 1 commit
- Move TransformerDecoderLayer to modeling/ · 58e805e0
  Hongkun Yu authored Jun 19, 2020
```
PiperOrigin-RevId: 317330705
```
  58e805e0
16 Jun, 2020 1 commit
- Internal change · 802488f1
  Hongkun Yu authored Jun 15, 2020
```
PiperOrigin-RevId: 316593329
```
  802488f1
12 Jun, 2020 2 commits
- Internal Change · 5a4c4e18
  Chen Chen authored Jun 11, 2020
```
PiperOrigin-RevId: 316054828
```
  5a4c4e18
- Internal change · 9cdb5d72
  Chen Chen authored Jun 11, 2020
```
PiperOrigin-RevId: 316053809
```
  9cdb5d72
10 Jun, 2020 3 commits
- Fix get_config() of MaskedSoftmax. · 8a1dbbad
  Chen Chen authored Jun 10, 2020
```
PiperOrigin-RevId: 315767729
```
  8a1dbbad
- Fix typo. · 4cd908ef
  Hongkun Yu authored Jun 10, 2020
```
PiperOrigin-RevId: 315738983
```
  4cd908ef
- Changes the MaskedSoftmax to call(inputs, mask=None) · 99bdc3dc
  Hongkun Yu authored Jun 09, 2020
```
PiperOrigin-RevId: 315630320
```
  99bdc3dc
09 Jun, 2020 1 commit
- Update nlp.modeling.layers.ReZeroTransformer to be consistent with nlp.modeling.layers.Transformer · d4bb3055
  A. Unique TensorFlower authored Jun 09, 2020
```
PiperOrigin-RevId: 315584374
```
  d4bb3055
08 Jun, 2020 1 commit
- Enable bert pretraining on fp16. · ea61bbf0
  Hongkun Yu authored Jun 07, 2020
```
PiperOrigin-RevId: 315214450
```
  ea61bbf0
03 Jun, 2020 4 commits

Revert "Revert "Add relative positional embedding to KerasBERT (#8617)" (#8619)" · a6c0e677
Hongkun Yu authored Jun 03, 2020
```
This reverts commit 4bb13e61.
```
a6c0e677
Revert "Add relative positional embedding to KerasBERT (#8617)" (#8619) · 4bb13e61
Hongkun Yu authored Jun 03, 2020
```
This reverts commit c3c2386c.
```
4bb13e61

Add relative positional embedding to KerasBERT (#8617) · c3c2386c

xinliupitt authored Jun 03, 2020

* root dir

* zone updated

* print mask

* preview emb

* tf print

* input only

* emb

* tf print

* emb after mask

* masked_softmax print

* print scores

* multi folder

* first pos emb

* check input shape

* add test temp

* import math

* two classes

* prints

* all get_pos replace

* make time scale private

* pos emb comments

* print input

* embedding_inputs

* tf shape

* dimention list

* tf_util

* print tf_util

* concise

* transformer pos change to layer

* keep length var

* length as input

* None as input

* print time signal

* print time signal

* remove print

* test input shape

* double check shape

* double check shape

* double check shape

* more test

* shape check

* shape check

* print 97 info

* print 97 info new

* test if sam

* assert same

* remove assert

* tf print same

* tf print diff

* output example

* output example

* output example

* formal test

* formal test length

* raise valurerror

* test valurerror

* double check

* comments

* remove prints

* rename relative

* delet naive test

* delete docs in xinliu branch

* code reformat

* import order

* indentation fix

* more files

* adjust char number

* disable not callable

* comment to length

* error of length unequal to input_shape

* root dir

* zone updated

* print mask

* preview emb

* tf print

* input only

* emb

* tf print

* emb after mask

* masked_softmax print

* print scores

* multi folder

* remove docs

* remove prints

* root dir

* zone updated

* print mask

* preview emb

* tf print

* input only

* emb

* tf print

* emb after mask

* masked_softmax print

* print scores

* multi folder

* remove docs

* apply revised 3 files

* rm prints

c3c2386c

Internal change · 20897493
Tianqi Liu authored Jun 02, 2020
```
PiperOrigin-RevId: 314451720
```
20897493

02 Jun, 2020 2 commits

Add relative positional embedding to KerasBERT (#8606) · 2db2501b

xinliupitt authored Jun 02, 2020

* root dir

* zone updated

* print mask

* preview emb

* tf print

* input only

* emb

* tf print

* emb after mask

* masked_softmax print

* print scores

* multi folder

* first pos emb

* check input shape

* add test temp

* import math

* two classes

* prints

* all get_pos replace

* make time scale private

* pos emb comments

* print input

* embedding_inputs

* tf shape

* dimention list

* tf_util

* print tf_util

* concise

* transformer pos change to layer

* keep length var

* length as input

* None as input

* print time signal

* print time signal

* remove print

* test input shape

* double check shape

* double check shape

* double check shape

* more test

* shape check

* shape check

* print 97 info

* print 97 info new

* test if sam

* assert same

* remove assert

* tf print same

* tf print diff

* output example

* output example

* output example

* formal test

* formal test length

* raise valurerror

* test valurerror

* double check

* comments

* remove prints

* rename relative

* delet naive test

* delete docs in xinliu branch

* code reformat

* import order

* indentation fix

* more files

* adjust char number

* disable not callable

* comment to length

* error of length unequal to input_shape

2db2501b

Internal change · 8eb91073
Chen Chen authored Jun 02, 2020
```
PiperOrigin-RevId: 314373769
```
8eb91073

30 May, 2020 1 commit
- adds a ClassificationHead layer · 2cf59d12
  Hongkun Yu authored May 29, 2020
```
PiperOrigin-RevId: 313906815
```
  2cf59d12
29 May, 2020 2 commits
- Proposes the full functionality of MultiHeadAttention layer. This change first... · 8a13ca4e
  Hongkun Yu authored May 29, 2020
```
Proposes the full functionality of MultiHeadAttention layer. This change first goes to model garden NLP library.

PiperOrigin-RevId: 313847485
```
  8a13ca4e
- Internal change · f2f26a8b
  Chen Chen authored May 29, 2020
```
PiperOrigin-RevId: 313812017
```
  f2f26a8b
28 May, 2020 1 commit

Use float32 activation in Transformer. · 94b1efc1

Reed Wanderman-Milne authored May 28, 2020

Float32 is used if the model uses mixed precision with bfloat16. Float16 activation are unchanged.

The motivation is that BERT with the LAMB optimizer with a gelu activation has an unstable loss when gelu is in bfloat16. Unfortunately, it is not easy to check if the LAMB optimizer and gelu is used, and perhaps there are other cases that work better with float32 activations instead of bfloat16 activations, so we always do the activation in float32 instead of bfloat16.

PiperOrigin-RevId: 313618322

94b1efc1

19 May, 2020 1 commit
- Allows to customize feedforward layer in transformer_scaffold. · d9c4e454
  Chen Chen authored May 19, 2020
```
PiperOrigin-RevId: 312366167
```
  d9c4e454
18 May, 2020 1 commit
- Internal Change · 6f7a682d
  Chen Chen authored May 18, 2020
```
PiperOrigin-RevId: 312116965
```
  6f7a682d
17 May, 2020 1 commit
- Update nlp.modeling.layers.ReZeroTransformer, to have the same interface with... · ef61bae9
  A. Unique TensorFlower authored May 16, 2020
```
Update nlp.modeling.layers.ReZeroTransformer, to have the same interface with nlp.modeling.layers.Transformer

PiperOrigin-RevId: 311937563
```
  ef61bae9
12 May, 2020 3 commits
- Fixes a typo. · 3ae3aecf
  A. Unique TensorFlower authored May 12, 2020
```
PiperOrigin-RevId: 311196489
```
  3ae3aecf
- Make Transformer Layer back compatible. · ac5fff19
  Hongkun Yu authored May 12, 2020
```
PiperOrigin-RevId: 311165658
```
  ac5fff19
- Internal Change · 2878da2e
  Chen Chen authored May 12, 2020
```
PiperOrigin-RevId: 311072125
```
  2878da2e
10 May, 2020 1 commit
- Internal change · 09c5ae2f
  Hongkun Yu authored May 09, 2020
```
PiperOrigin-RevId: 310767440
```
  09c5ae2f
05 May, 2020 1 commit
- Introduce output_range for transformer layer and encoder last layer. · 563d923a
  Hongkun Yu authored May 05, 2020
```
PiperOrigin-RevId: 310032518
```
  563d923a
21 Apr, 2020 1 commit
- Internal change · 330b34fe
  Hongkun Yu authored Apr 21, 2020
```
PiperOrigin-RevId: 307689094
```
  330b34fe
20 Apr, 2020 1 commit
- In BERT's export to TF Hub, fix shape propagation for seq_length. · b3377b09
  A. Unique TensorFlower authored Apr 20, 2020
```
PiperOrigin-RevId: 307425903
```
  b3377b09
19 Apr, 2020 1 commit
- Fix get_config() in ReZeroTransformer. · 295b3b4f
  Le Hou authored Apr 19, 2020
```
PiperOrigin-RevId: 307297217
```
  295b3b4f
17 Apr, 2020 4 commits
- Add TalkingHeadsAttention to README. · 85b50c88
  Chen Chen authored Apr 17, 2020
```
PiperOrigin-RevId: 307082084
```
  85b50c88
- Add ReZeroTransformer to README. · f499e880
  Le Hou authored Apr 17, 2020
```
PiperOrigin-RevId: 307079383
```
  f499e880
- Internal change · 01c4ee81
  Chen Chen authored Apr 16, 2020
```
PiperOrigin-RevId: 306986791
```
  01c4ee81
- Add ReZero support (https://arxiv.org/abs/2003.04887) · dbd74385
  Le Hou authored Apr 16, 2020
```
PiperOrigin-RevId: 306971008
```
  dbd74385