Hugging FaceMarc Sun, Arthur Zucker, Lysandre10 min readintermediate
Transformers now runs llama.cpp quants
Summary
Transformers now supports GGUF quantized checkpoints (used by llama.cpp) via the `kernels` library. Load a GGUF model with `from_pretrained` on Apple Silicon, run generation with standard Transformers APIs, or serve it via `transformers serve`. Benchmarks on an M2 Max show token‑throughput comparable to llama.cpp, with fallback to dequantization if kernels are missing. The integration also enable…
- GGUF packs weights, tokenizer, and optional chat template in a single file; common quantizations are Q4_K_M, Q5_K_M, Q6_K.
- Loading a GGUF model requires only `gguf_file` argument; Transformers automatically selects ggml/Metal kernels or falls back to SDPA.
- Benchmark on a MacBook Pro M2 Max (32 GB) shows Transformers’ `generate` throughput close to llama.cpp’s `llama‑bench` for Q4_K_M checkpoints, despite including prefill time.
- `transformers serve` exposes an OpenAI‑compatible endpoint for local GGUF models, usable by clients like Jan or Pi.
Local inference on consumer hardware is becoming practical; integrating GGUF into Transformers lets developers stay in the familiar PyTorch ecosystem while leveraging llama.cpp’s efficient quantized kernels. This lowers the barrier for experimentation, evaluation, and fine‑tuning of quantized model…
6/10
