Local AI / Engineering

How a Compression Trick Put Large Language Models on a Laptop CPU

Three years ago, running a large language model meant renting a data-center GPU cluster. Today the same class of model runs on a laptop with no GPU at all — and nothing about the model itself changed.

The project responsible is llama.cpp, a C/C++ inference engine started by a single developer, Georgi Gerganov, in March 2023. It now carries more than 75,000 stars on GitHub and has become the reference implementation for running open-weight models locally — on laptops, phones, Raspberry Pis, and everything in between. The algorithm underneath it is called quantization, and its most refined form — k-quants — is the actual subject of this piece.

Why models are so large in the first place

A language model is, at its core, a very large collection of numbers — weights — organized into layers. Trained models typically store each weight as a 16-bit floating point number. A 7-billion-parameter model at that precision needs roughly 14GB just to hold the weights in memory, before you've generated a single token. A 70-billion-parameter model needs well over 100GB. That puts even mid-sized open models firmly out of reach of any laptop.

Quantization asks a blunt question: does every weight actually need 16 bits of precision to do its job? For most of them, the answer turns out to be no.

The first pass: block-wise quantization

The simplest form of quantization splits a model's weights into small blocks — typically 32 weights at a time — and stores each block as low-precision integers (say, 4 bits each) plus a single "scale" number that says how to stretch those integers back toward their original range. This alone cuts a 7B model from about 14GB down to roughly 4GB, with a real but often tolerable drop in output quality.

The obvious problem: you still need to store a scale value for every single block. Across billions of weights, those scale values add up — for a 16-billion-parameter model, on the order of 2GB just for the book-keeping.

The actual innovation: quantizing the quantization

This is where llama.cpp's k-quants system earns its name. Rather than storing each block's scale at full precision, k-quants groups 8 blocks into a "super-block" of 256 weights, and quantizes the scale values themselves down to 8 bits — a second, coarser layer of compression sitting on top of the first.

Super-block (256 weights)
 ├─ super-block scale + offset      (2 × FP16)
 ├─ block 0  (32 weights, 4-bit)  + quantized scale (INT8)
 ├─ block 1  (32 weights, 4-bit)  + quantized scale (INT8)
 ├─ ...
 └─ block 7  (32 weights, 4-bit)  + quantized scale (INT8)

This double-quantization trick — compressing the compression metadata, not just the weights — cuts that 2GB of book-keeping overhead for a 16B model down to roughly 1GB, on top of the savings already gained from quantizing the weights themselves. It also has a second, less obvious benefit: grouping data into contiguous super-blocks means the CPU can read memory sequentially instead of jumping between scattered small blocks, which matters enormously for cache performance on ordinary consumer processors.

Layered on top of this, an optional importance matrix technique (informally called I-quants) runs a small amount of calibration data through the full-precision model first, to work out which weights actually move the output the most. Those weights are quantized more gently; the rest are compressed harder. It's the same intuition behind lossy image compression — spend your bits where the eye (or in this case, the model's output) will actually notice.

What this buys you, in practice

The naming convention for these formats (Q4_K_M, Q5_K_M, Q6_K, and so on) tells you the bit-width and quality tier. In practice, independent benchmarking has found:

• 13B model: ~26GB at full precision → roughly 8GB at Q4_K_M, with quality loss commonly measured in single-digit percentages on standard evaluation sets.
• 3B model: fits in about 2GB of RAM and can generate 15–30 tokens per second on a modern CPU — fast enough to feel conversational.
• 7B model: fits in roughly 4.5GB and still manages several tokens per second without any GPU acceleration at all.

All of this is packaged into a single portable file format called GGUF, which bundles the quantized weights, tokenizer, and chat template together, and can be memory-mapped straight off disk — so a model doesn't even need to be fully loaded into RAM before inference can begin.

The headline number worth sitting with: 4-bit quantization typically shrinks a model by around 75% with quality loss that, for most practical purposes, is barely noticeable.

The catch — because there always is one

This isn't a free lunch, and treating it as one is where a lot of enthusiastic write-ups overstate things. Push quantization too aggressively — below roughly 4 bits per weight without an importance matrix — and measurable quality loss shows up quickly: a recent unified academic evaluation of k-quant formats on an 8-billion-parameter model found that choosing the right quantization level involves a real accuracy trade-off, not just a file-size one, and that the practical guidance circulating in the community is still largely anecdotal rather than rigorously benchmarked. That's an open research gap, not a solved problem.

Why this matters more here than in Silicon Valley

A compression algorithm that turns a 100GB model requirement into an 8GB one is interesting anywhere. It's transformative somewhere like Malawi, where a laptop with 8–16GB of RAM is a realistic developer machine and a cluster of A100 GPUs is not. The same instinct that pushed a Blantyre fintech team toward USSD before an app, or a Lilongwe logistics startup toward a homegrown routing engine, applies just as directly here: the constraint of modest hardware isn't just a limitation to work around — it's a forcing function that rewards genuinely efficient engineering over brute-force compute.

Quantized, locally-run models mean a developer without reliable broadband can still prototype an AI feature offline. It means student research doesn't depend on a cloud API budget. And it means the gap between "what's possible with AI" and "what's possible on the hardware people actually have" keeps shrinking — one quantized weight at a time.