Attention Is Not What You Need (2025)
The article revisits whether explicit self-attention is necessary for strong performance, arguing that standard multi-head attention acts as a mathematically opaque form of tensor lifting.
Key points
- •It re-examines the fundamental question of whether explicit self-attention is truly required for high-performance reasoning.
- •It posits that standard multi-head attention is best understood as a form of tensor lifting that maps hidden vectors into a high-dimensional space of pairwise interactions.
- •While this mechanism is highly expressive, it becomes mathematically opaque after many layers due to the constraints imposed by gradient descent.
#Research & Papers#LLM / Model Trends