Hugging Face Daily PapersJulien Siems, Riccardo Grazzi, Korbinian Pöppel1 min readpaperadvanced
Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention
Summary
Complex KDA (CKDA) extends Kimi Delta Attention by widening gate and beta ranges, allowing a single diagonal‑plus‑rank‑one transition to represent any orthogonal matrix, including 2‑D rotations. The paper proves this expressivity, keeps the model non‑expansive, and shows CKDA achieves state‑of‑the‑art length‑extrapolation and competitive language‑model performance.
- Extending KDA gates to [-1,1] and β to [0,2] creates Complex KDA, enabling any orthogonal diagonal‑plus‑rank‑one matrix in a single transition.
- CKDA keeps the diagonal‑plus‑rank‑one, non‑expansive update, preserving the efficiency and stability of linear RNNs.
- Proof that every orthogonal diagonal‑plus‑rank‑one matrix is exactly a CKDA transition; a single layer can track any finite subgroup of SO(3).
- Empirically, CKDA achieves best length‑extrapolation on S3, S4, periodic audio and matches/exceeds Transformers on language modeling while scaling well.
Developers of long‑context LLMs and efficient sequence models should care because CKDA offers a provably expressive yet stable linear RNN alternative with competitive performance.
8/10