Local LLMs suffer from a problem no amount of free software can fix: how much graphics memory can you afford? You find a model that looks promising, check its requirements, and realize your GPU isn’t even close. That’s exactly what makes PrismML’s Bonsai 27B so interesting.

It squeezes the weights of a 27-billion-parameter model into roughly 3.9GB, putting something surprisingly capable within reach of everyday hardware, almost. You see, there’s a difference between a model’s weight size and the memory needed to run it. Bonsai’s compression is impressive, but that’s not all there is to the story.

A 27B model squeezed down to 1 bit per weight

Same Qwen underneath, with a trick that takes 54GB down to 3.9GB

Bonsai 27B isn’t a new model. It’s built from Qwen3.6-27B, and the underlying architecture is the same. Now, a 27-billion-parameter model at 16-bit precision needs about 54GB for its weights alone, which rules out every consumer GPU on the market. So how is PrismML doing it?

The trick is simpler than you’d expect. In the Q1_0_g128 format, each weight is a single sign bit: 0 means negative, and 1 means positive. Every group of 128 weights shares one FP16 scale factor, and that works out to around 1.125 bits per weight, or roughly 14.2 times smaller than FP16. PrismML even took a jab at its competition, claiming that a popular 2-bit Qwen3.6-27B build averages 2.8 bits per weight actually and takes up 9.4GB.

What PrismML hasn’t shared is how it decides which way each sign points. An independent study confirmed that the released weights are real and work well. It also found that none of the whitepapers explain the method, and that basic 1-bit quantization using the public format falls apart completely.

Getting it running on your system is also straightforward. PrismML’s repository tells you to use its own fork, but you don’t strictly need it anymore because Q1_0 support is now part of mainline llama.cpp on CPU, Metal, CUDA, and Vulkan. That said, the fork is still faster on CPUs, though, giving 13.19 versus 7.91 tokens per second from prompt processing. LM Studio also offers the model and lists 4GB of RAM as the minimum. Don’t expect a 4GB download, though. In my testing, LM Studio pulled 4.73GB because it also fetches the model’s 0.93GB vision component.

Additionally, PrismML suggests a temperature of 0.7, top-p of 0.95, and top-k of 20. If you’re using LM Studio, all three settings are already applied when you download the model.

The 4GB number is the weights, not the bill

Context, the KV cache, and why a 4GB card is a fantasy

Sure, Bonsai’s weights take up only 3.79GB, but they’re not all your GPU needs to hold to run the model. It also has to hold the KV cache and runtime buffers. PrismML measures peak memory at 5.2GB with a 4,000 context and 11.6GB at 100,000, with the cache left at FP16. Turning on 4-bit KV-cache quantization brings the 100,000 figure down to 6.86GB, and the full 262,000 window fits in 9.4GB.

So while a 4GB card can store the model, you actually really need a 6GB or 8GB card to run it comfortably. Long context is cheap here because only 16 of the model’s 64 layers keep a full-attention cache. I tested this on my home server laptop with a 4GB GTX 1650 Ti. LM Studio flagged the model for partial GPU offload right away and defaulted to just four GPU layers.

When I forced full offload at a 4,096-token context, the load failed with a “failed to allocate buffer for kv cache” error. The model only loaded once I dropped offload to 50 percent. That uses about 3.6GB of VRAM and left the rest of the model to the CPU.

How much brain survives the squeeze

Math and coding hold up, instruction following takes the hit

Across 15 benchmarks run in thinking mode, Bonsai averages 76.11, compared with 85.07 for the full-precision original. That’s 89.5 percent of the original score for a significant reduction in size. Math holds up best at 91.66, and coding comes in at 81.88. The losses, however, show up in instruction following, which falls from 78.47 to 65.74, and agentic tool calling, dropping from 80 to 66.03.

PrismML’s response is that regular 2-bit builds fail in sneakier ways. For example, IQ2_XXS scores 57.5 on AIME26 while still hitting 88.93 on MMLU-Redux. All of these numbers have been reported by PrismML itself.

Fast on a GPU, grim on a CPU

Mac numbers, an RTX 5070 Ti, and a Ryzen that struggles

On Macs, PrismML reports 26 tokens per second on an M4 Pro, 44.2 on an M5 Pro, and 66.4 on an M5 Max. An optional speculative-decoding add-on called DSpark pushes an H100 from 104.8 to 143.8 tokens per second.

Those numbers might sound impressive for responsiveness, but independent tests aren’t as optimistic. A DataCamp tester saw about 92 tokens per second on an RTX 5070 Ti, with VRAM use around 11.9GB at a 100,000 context. They still found a Q4 build of Qwen3.6-27B more reliable for coding-agent work.

CPUs are where Bonsai struggles the most. On a Ryzen 5 5600, Big Iron measured 6.7 tokens per second for generation and only 13 tokens per second for reading prompts. Bonsai scored 32 out of 40 on homelab questions, while the much smaller Qwen 3-4B scored 35. Memory use also went up to 16.3GB because of llama-server’s prompt cache.

My own results were much slower. With half the model on the GTX 1650 TI and the rest on a Core i5-9300H, Bonsai generated about 4 tokens per second. With no model layers on the GPU, that fell to 2.4. A prompt of about 1,600 tokens took nearly a minute to produce its first token either way. Thinking mode makes the wait even more noticeable. Bonsai got a simple train-arrival question right, but spent over three minutes thinking before it answered.

Who should actually download this

8GB and 12GB GPUs and Apple silicon, mostly

Bonsai 27B makes the most sense on 8GB or 12GB GPUs and Apple silicon laptops. On that hardware, a normal 27B model won’t fit at all, but Bonsai will. If you have a little more memory to spare, the ternary version keeps 94.6 percent of the original score.

PrismML itself admits that long, multi-file agentic coding isn’t something this release is good at at the moment. Regardless, a 27B-class reasoning model running on an ordinary laptop seemed impossible not so long ago, and now you can download one right now.