Running AI locally is the kind of project that starts with a graphics card upgrade and ends with an empty wallet. But modern models are far superior to early local LLMs, and you don’t necessarily need a powerful workstation to get something useful. The trick is to choose the model that suits your hardware and know what jobs to give it.

You won’t be squeezing a ChatGPT replacement onto a budget laptop with a 4GB GPU and 16GB of RAM, but you might just find a smaller assistant that can handle everyday requests without sending everything to the cloud. These compact models make that trade worthwhile.

LFM3-1.7B: the model that fits where nothing else will

About 1GB on disk, built for agents, not for pretending it’s ChatGPT

Liquid AI’s October 2026 LFM3-1.7B is one of the best starting points for a small LLM you can run on just about everything. Its 1.73-billion-parameter model has an official Q4_K_M download of approximately 1.1GB. That leaves considerably more room for runtime overhead than a model whose weights alone nearly fill your GPU.

Its intended purpose is local agents, tool calling, structured outputs, and specialized applications, not being an all-knowing chatbot. Think of it as a candidate for turning requests into structured actions or extraction into a predictable format, rather than your first choice for an ambitious software project.

Liquid explicitly advises against using it for agentic coding or complex multistep reasoning, and recommends retrieval or search for knowledge-heavy tasks. That limitation also makes the recommendation clearer. You choose it when workflows matter more than sheer knowledge.

MiniCPM5-2B is the one that actually tries to think

Better coding numbers, with a small print you should read first

Released in September 2026, OpenBMB’s MiniCPM5-2B actually contains approximately 2.52 billion parameters, while Q4_K_M weights occupy around 1.56GB. It targets local assistants, coding agents, tool use, and reasoning, making it a logical next candidate when the smallest option feels restrictive.

OpenBMB reports 69.1 on LiveCodeBench v6 and 46.4 on SWE-bench Verified in its evaluation setup. While these are promising results, they’re not proof that the quantized version will run the same on your machine.

For a more practical evaluation, give it a small Python function to repair, a configuration file to explain, or a well-specified transformation. Keep the task bounded and verify the output. The appeal is getting meaningful assistance from relatively small models, not handing over an entire codebase unsupervised.

Granite 4.2-3B lets you decide how hard it thinks

Three reasoning modes, Apache 2.0, and no need to overthink easy questions

Apart from everything mentioned above, Granite 4.2-3B adds another useful option: selectable full-thinking, non-thinking, and low-effort modes. Its Q4_K_M build is approximately 2.24GB, making it a plausible candidate for this memory budget without using the entire GPU.

That added flexibility helps because not every question deserves an extended reasoning trace. You can start with direct answers for simple requests, then enable more reasoning when the task requires it.

IBM intended the model to be used for reasoning, code generation, tool calling, and multilingual dialogue, with the release covered under an Apache 2.0 license. It’s worth considering for a self-hosted assistant where you want control over both deployment and response behavior.

Qwen3.5-4B is where your VRAM starts filing complaints

Vision and broader knowledge, but thinking mode eats tokens for breakfast

Qwen3.5-4B is one of the best compact models to consider if you need both text and vision capabilities. The model’s published language evaluations include 79.1 on MMLU-Pro and 76.2 on GPQA Diamond, although those results shouldn’t be treated as guaranteed four-bit desktop performance.

This is the model to go for if broader knowledge or image input matters more to you than minimizing the overall footprint. It does come with a catch, though. Qwen defaults to thinking mode, and its documentation recommends generous output budgets for more difficult reasoning problems. Cutting context to squeeze it onto limited hardware changes the conditions under which its benchmark results were obtained. So just because it might load comfortably on your hardware, that doesn’t mean you get all the capabilities it advertises.

Ignore the leaderboards and benchmark your own mess

Same prompts, same checks, smallest model that doesn’t embarrass you

The best way to evaluate these models is to make your workload the benchmark. Start by testing LFM3-1.7B for structured tasks, MiniCPM5-2B for bounded reasoning, Granite for adjustable thinking, or Qwen when broader capabilities justify the additional demands.

Use the same real prompts across candidates. Check factual accuracy, valid JSON, working code, response time, and peak memory — not just whether an answer sounds polished. The smallest model you can get away with is the one that clears the quality threshold consistently, with enough hardware headroom left to keep using your computer.