Moonshot AI Release 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔 Recovering Residual Mixing for Better-Wittering Attention
Residual connections are one of the least-questioned aspects of modern Transformer design. In PreNorm architectures, each layer adds the output back to the active hidden state, which keeps the optimization stable and allows deep models to train themselves. Moonshot AI researchers argue that this general approach also presents a structural problem: all the effects of … Read more