Hugging Face Daily PapersMobina Kashaniyan, Ali JannesariSep 18, 20262 min readpaperintermediate

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

Summary

Increasing the number of LLM candidates (N) improves reasoning accuracy, but the way those candidates are generated (batch size vs sequential calls) dramatically affects latency, GPU‑hours, and energy. On A100 GPUs, eight serial 1‑candidate calls consume ~5× more energy and take ~6× longer than a single 8‑candidate batched call, even though total candidate count is identical. The authors recommen…

Accuracy gains from test‑time scaling are tied to total candidate count (e.g., N=8 gives +8.4 pp on Phi‑3‑mini, +18.4 pp on Qwen2.5‑1.5B on GSM8K).
Generation schedule matters: 1×8 (one batched call) vs 8×1 (eight serial calls) shows 4.6‑4.9× higher GPU‑device energy and 5.8‑6.1× higher P95 latency for the serial case.
Results are consistent across models, GPU types (A100, V100), and datasets (GSM8K, SciQ).
When memory permits, fewer calls with larger batch sizes are far more efficient for multi‑candidate inference.

LLM inference cost is a primary bottleneck for production services. This work shows that naïvely reporting only the number of generated candidates hides large variations in hardware usage and latency, which directly impact cloud spend and user experience. Engineers can immediately reduce inference…

6/10

llm performance gpu inference scaling

Related reading

SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops arxiv.org
SiliconBench benchmarks nine Apple‑Silicon LLM serving engines on unified‑memory desktops, measuring speed, memory usage, and output fidelity across Qwen3, Qwen3.5, and Gemma‑4 models. It finds vllm‑metal doubles throughput at modest concurrency, memory budgets often fail to preserve headroom, and only three stacks satisfy all fidelity and model‑coverage requirements, with tensor‑parallel scaling…
Hugging Face Daily Papersarxiv.orgRanran Haoran Zhang, Aysa Xuemo Fan, David Munhá Correia, Alex Cheema, Rui Zhang4d1 minpaper
Learning to solve hard problems in RL for LLMs by never giving up github.io
The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…
Hacker News front pagegithub.ioMichael Noukhovitch6d11 minHN1179
Accurate Models of AMD Matrix Cores arxiv.org
The authors reverse‑engineer AMD’s CDNA matrix cores, build per‑architecture MATLAB models that match hardware bit‑for‑bit on 10 M random tests, and use them to compare AMD vs NVIDIA tensor‑core accuracy.
Hacker News front pagearxiv.orgFaizan A Khattak, Mantas Mikaitis, Carlo J. Graziani5d2 minpaperHN7911
Do LLMs Have the Memory of a Goldfish?bytebytego.com
The article explains that LLMs don’t have persistent personal memory; all “memory” is supplied by the surrounding application via the context window, summaries, or external storage. It outlines the distinction between trained weights, working‑memory (token context), and persistent application memory, shows how to construct API calls to preserve conversation state, and discusses the cost and laten…
ByteByteGobytebytego.com7d12 min
Five models pass the easy code reviews. Then the diffs get hard.bjornjohansen.com
A hands‑on benchmark of five LLM code‑review models (Claude Opus 5, GPT 5‑6 Sol, Grok 4.6, Kimi 3, GLM 5.3, Qwen 3.8 Max, DeepSeek v4 Pro) shows that all models ace tiny 15‑line diffs but diverge sharply on 100‑300‑line, cross‑file changes. The author built a deterministic harness (single HTTP call, fixed system prompt, JSON output) and measured detection rate, false‑positives, latency, token usa…
Bjørn Johansenbjornjohansen.com17d16 min
Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems arxiv.org
Emergence World is a continuously running multi‑agent sandbox used to stress‑test frontier LLM‑based agents over weeks. Eight parallel worlds (seven homogeneous, one mixed) generated 850 k LLM calls and ~50 B tokens while agents pursued goals, used tools, and maintained persistent memory. The authors injected three adversarial events—prompt injection, misinformation, and private‑memory exposure—a…
Hugging Face Daily Papersarxiv.orgDeepak Akkil, Tamer Abuelsaad, Karthik Vikram7d1 minpaper

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

Summary

Related reading

SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktopsarxiv.org

Learning to solve hard problems in RL for LLMs by never giving upgithub.io

Accurate Models of AMD Matrix Coresarxiv.org

Do LLMs Have the Memory of a Goldfish?bytebytego.com

Five models pass the easy code reviews. Then the diffs get hard.bjornjohansen.com

Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systemsarxiv.org

SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops arxiv.org

Learning to solve hard problems in RL for LLMs by never giving up github.io

Accurate Models of AMD Matrix Cores arxiv.org

Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems arxiv.org