Hugging FaceAntonio Tiene, Ali Hashemi, David Jansen, Roman Rausch8 min readadvanced
Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
Summary
The authors cast transformer block removal as a constrained binary optimization problem equivalent to an Ising glass, using a Hessian‑derived energy as a proxy for downstream quality. Solving the resulting QUBO with classical or quantum‑inspired solvers yields up to 23 MMLU points improvement over prior block‑removal baselines at 50 % depth compression.
- A second‑order Taylor expansion of loss yields a Hessian whose diagonal gives block importance and off‑diagonals capture pairwise couplings.
- Low‑energy states of the resulting Ising Hamiltonian correlate strongly with high‑performing pruned models, allowing cheap evaluation of billions of configurations.
- Exact brute‑force enumeration is feasible for modest block counts; for larger models, tabu search and other QUBO solvers find good low‑energy states in seconds.
- The best pruned model is often an excited (non‑ground) state, demonstrating the value of exploring the low‑energy spectrum.
LLM engineers and infrastructure teams looking to reduce inference latency and memory without heavy retraining should care, as the technique offers a systematic, high‑performing way to depth‑prune large models.
8/10
.png)