Running a local AI model usually requires painfully high VRAM, and not everyone has the latest monster GPU with the coveted yet still absurd 20-plus gigabytes of VRAM. Thankfully, there do exist a few surprisingly capable, lightweight alternatives that (depending on your exact use case) might even surprise you with what they can do, whether you’re on AMD or Nvidia.

Test bench and performance expectations

All-AMD system with configurable VRAM limits

For the test platform, I used one of the newer Ryzen AI Max-series laptops. It’s quite powerful for its size, but is nowhere near to being a reasonably priced product — but it does one thing quite right.

Which would be in its integrated GPU, which can go toe-to-toe with an RTX 4050 Mobile for a lot less power. There’s also the fact that these laptops have no dedicated VRAM, instead sharing that memory from the total RAM pool. Thankfully, it’s possible to choose from a set of preconfigured VRAM limits in the BIOS, which will be very handy later on.

By capping the BIOS at specific values, we can effectively simulate a card with varying VRAM amounts while keeping performance (almost) identical. As you’ll quickly find out later on, these VRAM limits can make or break your runs. Arch also runs near flawlessly on this machine, making it ideal for this particular experiment.

That said, my particular AI Max model can allocate up to 24 gigabytes of VRAM. Not that it would help out much in this case; we’re going to be bottlenecked by its compute capabilities first. Either way, I chose an 8 GB VRAM limit to represent the basic entry-point for these surprisingly capable models.

Phi-3.5 Mini (3.8 B)

Fast and efficient

Phi-3.5 Mini is extraordinarily fast and efficient, allowing it to run FP16 high-precision models while consuming relatively little video memory. It’s also notable in having the highest tokens-per-second throughput in this list. Unfortunately, it’s not all perfect — and in real-world use, it struggles with multistep problems and is very restrictive about what it can and cannot do, which brings us to the next entry in the list.

Llama 3.1 (8B)

Good general-purpose model

Llama 3.1 is the most balanced pick of the lot, equally good at both reasoning and general use. It’s also optimized for AMD’s ROCm stack, which, combined with 4-bit quantization, slips in at under 6 gigabytes of VRAM usage, making it well-suited to older cards.

While compressing the model to 4-bit makes it much less taxing on VRAM, it comes with a trade-off: reduced accuracy. Still, it’s much more worthwhile to have compression enabled, since it would otherwise fill the limiting 8GB of VRAM.

Mistral 7B (v0.3)

Speedy but consistent

Mistral 7B is much more efficient than Llama, striking a balance between speed and quality. It’s also better at processing initial prompts when quantized to 4-bit.

Unfortunately, Mistral is hard-capped to a 32k context limit, which makes it not as ideal as Llama for something a bit more technical. It’s still good enough for creative writing and simpler tasks, though.

Qwen 2.5 (7B)

Great but a bit too formal

Qwen 2.5 is also quite efficient, trained to perform excellently across multilingual tasks and to some degree interpret across these languages. However, Qwen 2.5 is slightly biased towards generating rigid, formal responses. On the flip side, this makes it great for more technical tasks.

Gemma 2 (9B)

The most powerful one

Gemma 2 9B remains one of the largest models you can realistically run on a GPU with 8GB of VRAM. While its reasoning is unmatched at this class, it absolutely demands 4-bit quantization, leaving no VRAM headroom. Unfortunately, it’s nowhere near as fast as a 7B model and might be ill-suited for quicker tasks.

DeepSeek-R1-Distill-Qwen (7B)

DeepSeek for 8GB VRAM

This is the local, distilled version of DeepSeek R1 running on your hardware — which brings with it the capability for thought reasoning. It runs exceptionally well with Q4 quantization, but can feel a bit slow to respond at times, thanks to the way it processes things in “chains” of reasoning.

DeepSeek-Coder (6.7B)

Coding specialized DeepSeek

As the name suggests, DeepSeek Coder is custom-trained for coding and debugging. It sips memory at 4-bit and has enough headroom for heavy prompts within that 8GB window.

However, it’s a much more specialized model that doesn’t perform as well at creative content generation, so you’ll probably want to look at something else for that.

Gemma 2 (2B)

Deep world knowledge will be an issue

Gemma 2 2B is ultra-lightweight and optimized for devices with exceptionally low VRAM or limited headroom in general. It’s fast and efficient, but is only limited to 2.6 billion parameters, making it severely lacking in deep world knowledge. In other words, use it for instant tasks such as local translation and summarization.

Benchmarking the models

Drawing some conclusions

Of course, we just can’t end things without a proper benchmark of sorts. To achieve the same, I utilized a script (Claude usage disclosure!) tailored for my Arch Linux system on the same laptop/test bench.

I opted to install the ROCm stack and base my tests around it, which was simple enough on Arch Linux using the ollama-rocm package. After enabling the systemd service, I ran a test command to check if my GPU was detected.

Unfortunately, my Ryzen AI Max 390’s iGPU wasn’t being detected, so I had to work around it by forcing it to be an “unsupported” GPU (even though it technically works). With everything in place, it was finally time to run the Python script and test things out. I started by setting the laptop to its maximum TDP, with fans at 100% to avoid unnecessary thermal shutdowns.

I also ran the tests while plugged in, with the configurable VRAM set to two values – the default Auto, which auto-allocates memory to the iGPU as and when required, and an 8 GB hard VRAM cap. Since these settings required a BIOS reset, a reboot was required between the two runs.

Unsurprisingly, the Auto VRAM option seemed to perform better, though only by a slight margin (5 to 10 percent). Don’t go expecting every device to perform the same after a cold boot, though, as thermal saturation is very real, and performance will flatten out after running the machine for a while.

Keep your expectations in check

With only 8GB of VRAM, expect to be severely bottlenecked in the long run. It’s unfortunate, but despite its popularity, 8 gigabytes is just not enough for local AI, and even gaming these days.

While you could theoretically run a larger model, there’s the fact that some of the memory goes into managing the display server — which eats up precious VRAM that could be used otherwise.

KV caches grow with each conversation, so expect them to fill up quickly over time. That being said, you can definitely get smaller models to work with this VRAM limitation, which might serve as a “stepping stone” to larger and more powerful workloads.