SiLU FFN

[Ricardokevins]

LLaMA2/model.py

Line 216 in 5716de4

x_V = self.fc3(swish)

in LlaMA2 source code, they obtain 'X_V' with the origin 'x', instead of 'swish'

[Ricardokevins]

https://github.com/facebookresearch/llama/blob/6796a91789335a31c8309003339fe44e2fd345c2/llama/model.py#L348

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

[viai957]

in LlaMA2 source code, they obtain 'X_V' with the origin 'x', instead of 'swish' its the same but in here the 'swish' function with beta =1 is nothing but SiLU (Sigmoid Linear Unit) it just spits into different components here just for easier understanding.
This code is a replica of Umar Jamil explaining coding Llama 2 from scratch here is the link to the video : https://www.youtube.com/watch?v=oM4VmoabDAI&t=7212s&ab_channel=UmarJamil