Hugging Face Daily PapersMingzhe Du, Anh Tuan Luu, Dong Huang1 min readpaperadvanced
Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles
Summary
The paper introduces mutation analysis as a quantitative adequacy metric for GPU‑kernel benchmark oracles. By injecting 10,303 deterministic faults into 188 verified CUDA kernels (7,384 with a known kill witness), they show the official KernelBench checker misses 16.9% of faults—especially 78.6% of precision‑related faults. Their analysis quantifies the impact of existing patches (e.g., KernelBen…
- Mutation analysis provides a concrete, repeatable metric for oracle adequacy in GPU kernel benchmarking.
- The official KernelBench oracle is systematically weak: it fails to detect 1 in 6 witnessed faults, with precision faults being especially problematic.
- Existing patches can be quantitatively decomposed: hidden inputs contribute +4.0 points, tighter tolerances +4.5 points to the detection score.
- A published fuzzing recipe is over‑aggressive, rejecting correct kernels 107 times.
Benchmark oracles drive leaderboard rankings and reinforcement‑learning reward signals for LLM‑generated GPU kernels. If the oracle is weak, it can mislead research directions, inflate performance claims, and waste engineering effort on spurious optimizations. A measurable adequacy metric lets the…
8/10
