Internal change

PiperOrigin-RevId: 477574174

Internal change
PiperOrigin-RevId: 477574174
7b129e6b · Hongkun Yu · A. Unique TensorFlower · 89bd20c5 · 7b129e6b
Commit 7b129e6b authored Sep 28, 2022 by Hongkun Yu Committed by A. Unique TensorFlower Sep 28, 2022
Hide whitespace changes
Inline Side-by-side

Showing with 3 additions and 1 deletion

official/nlp/modeling/layers/transformer_scaffold.py official/nlp/modeling/layers/transformer_scaffold.py +3 -1

No files found.
--- a/official/nlp/modeling/layers/transformer_scaffold.py
+++ b/official/nlp/modeling/layers/transformer_scaffold.py
@@ -335,7 +335,9 @@ class TransformerScaffold(tf.keras.layers.Layer):
                                               training=training)
        layer_output += source_attention_output
      else:
-        # if not norm_first, assume that the feedforwad does apply layer norm
+        # Attention: if not norm_first, assume that the feedforwad does apply
+        # layer norm. The feedford also apply residual connection. Please
+        # read  the `GatedFeedforward` as a concrete example.
        layer_output = self._feedforward_block(attention_output,
                                               training=training)