proomt

Search

Search posts, papers, and topics

gguf

RSS
  1. 1

    Transformers now runs llama.cpp quants

    Transformers now supports GGUF quantized checkpoints (used by llama.cpp) via the `kernels` library. Load a GGUF model with `from_pretrained` on Apple Silicon, run generation with standard Transformers APIs, or serve it via `transformers serve`. Benchmarks on an M2 Max show token‑throughput comparable to llama.cpp, with fallback to dequantization if kernels are missing. The integration also enable…

    Hugging Facehuggingface.co10 minHN2