Transformers now loads GGUF checkpoints without unpacking them

Hugging Face said on Tuesday that transformers can now run GGUF quantizations directly — loading the single-file format built by the llama.cpp team and generating from it with the weights still packed, rather than expanding them back to full precision first. The work reuses llama.cpp's own ggml kernels, pulled in through Hugging Face's kernels library. The announced scope is narrow on purpose: local inference on Apple Silicon, starting with the Qwen3.5 architecture.

What actually shipped

The loading step is one extra argument. You pass a Hub repository and a `gguf_file` filename to `from_pretrained`, and everything after that is the ordinary transformers API — chat templates, `generate`, logits processors, the rest. Per the GGUF documentation, when the `ggml-org/ggml-quantization` kernel is available the loader defaults to MPS, keeps the weights packed and runs matrix multiplications directly on the packed blocks. It also switches the attention implementation to `ggml-org/ggml-attn`, the same flash-attention kernel llama.cpp uses for decode and prefill. If that kernel cannot be fetched, the model quietly falls back — to `sdpa` for attention, and to full dequantization at load, which costs memory.

Requirements are specific enough to matter before anyone plans around this:

• An Apple Silicon Mac. The packed inference path is MPS-only for now.
• transformers from main — the feature is not in a tagged release yet — plus a compatible `kernels` version.
• A PyTorch version covered by the published ggml-quantization kernel builds, which Hugging Face describes as usually the two most recent releases.
• Qwen3.5 dense or Qwen3.5 MoE weights, including compatible Qwen3.8 checkpoints. Every other architecture goes through the legacy loader, which always dequantizes.

One detail worth noting: packed loads force float32 automatically, because Hugging Face measured it as faster on MPS. Passing a different dtype produces a warning rather than the dtype you asked for.

The size argument

The case for GGUF has always been fitting a model into the memory you actually have. Hugging Face uses Unsloth's Qwen3.5-4B as the illustration: the unquantized BF16 file is 8.42 GB, `Q6_K` is 3.53 GB, `Q5_K_M` is 3.14 GB and `Q4_K_M` is 2.74 GB. The recommendation is to start at `Q4_K_M` and move up if there is memory to spare, with the caveat that the quality cost of heavier quantization depends on the model and the task, and should be measured on the work you actually want done.

The speed work is not GGUF-specific

Faster kernels only pay off if the GPU has work queued, so two changes landed in `generate` itself and they apply to every transformers model, not just GGUF ones. PR #48814, merged on 15 September, drops the all-ones padding mask at the start of generation for unpadded decoder-only inputs, so downstream attention code stops re-inspecting it. PR #47975, merged on 4 September, stops synchronizing the accelerator on every decode step by reading the stopping flag one step late; the PR reports a 1.10x to 1.20x speedup on MPS and no regression on CUDA. Hugging Face also wrote its own Metal `topk` kernel to address expert routing in MoE models.

Benchmarks, read carefully

Hugging Face reports that transformers lands close to llama.cpp across three GGUF checkpoints — a small dense model, a larger dense model and an MoE — measured on a MacBook Pro M2 Max with 32 GB of unified memory, macOS 26.6, PyTorch 2.12.1 and kernels 0.17.0. It also states the caveat itself: the two columns are not identical benchmark conditions, because the transformers figure comes from `generate` and includes prefill, while `llama-bench` reports decode-only throughput. Treat the comparison as directional.

Why do this at all

Hugging Face is unusually direct about not competing with itself here: llama.cpp remains, in its words, the recommended engine when the priority is efficient local inference. What transformers adds is the Python side — inspecting activations with hooks, evaluating quantized checkpoints through existing workflows, validating that a GGUF conversion matches the original weights, trying custom decoding, or fine-tuning from a GGUF file by way of `GgufConfig(dequantize=True)`. The same checkpoint can also be served over an OpenAI-compatible endpoint with `transformers serve`, addressed as `<repo>:<file>.gguf`, which is what lets a client like Jan or Pi point at a local machine.

The stated ambition is larger than the format. Because a kernel operates on tensors and does not require the whole model to come from a GGUF file, Hugging Face wants to use ggml's kernels to accelerate architectures llama.cpp has no implementation for — research models, new architectures, custom variants — and eventually vision, audio and multimodal models. That is a roadmap, not a shipped feature. For now the honest summary is a well-scoped first step: one backend, one model family, and a clear list of what does not work yet, including padded batch inference.