Unverified Commit 78061ef5 authored by Grzegorz K. Karch's avatar Grzegorz K. Karch Committed by GitHub
Browse files

Fix accessing hidden_act from model config (#32686)


Signed-off-by: default avatarGrzegorz Karch <gkarch@nvidia.com>
parent 528b3076
...@@ -177,10 +177,15 @@ class DeciLMDecoderLayer(nn.Module): ...@@ -177,10 +177,15 @@ class DeciLMDecoderLayer(nn.Module):
else: else:
intermediate_size = block_config.ffn.intermediate_size intermediate_size = block_config.ffn.intermediate_size
if hasattr(block_config.ffn, "hidden_act"):
hidden_act = block_config.ffn.hidden_act
else:
hidden_act = config.hidden_act
self.mlp = LlamaMLP( self.mlp = LlamaMLP(
hidden_size=self.hidden_size, hidden_size=self.hidden_size,
intermediate_size=intermediate_size, intermediate_size=intermediate_size,
hidden_act=config.hidden_act, hidden_act=hidden_act,
quant_config=quant_config, quant_config=quant_config,
bias=getattr(config, "mlp_bias", False), bias=getattr(config, "mlp_bias", False),
prefix=f"{prefix}.mlp", prefix=f"{prefix}.mlp",
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment