There are too many LLMs to test properly. You can chat with a few, pick a favorite, and call it a day, but a model that’s great at conversation might be terrible at coding, structured output, or following an oddly specific instruction.

When I need an LLM for a new task, like ranking a list and returning clean JSON, I don’t want to run the same prompt through dozens of models just to find the right one. That’s what benchmarks are supposed to help with. But unlike PC benchmarks, which anyone can run with a few clicks, LLM benchmarks usually mean trusting someone else’s results or setting up a complicated test suite yourself.

That’s why a good smoke test matters. It should be quick, easy to repeat, and revealing enough to show how a model actually behaves. Apparently, all you need is a pelican on a bicycle.

The absurd pelican test

SVG makes the pelican test work

Developer and AI researcher Simon Willison introduced the test on October 25, 2024. The exact prompt was is: generate an SVG of a pelican riding a bicycle.

He initially ran it against 16 models from OpenAI, Anthropic, Google, and Meta. Willison chose the subject partly because he likes pelicans, don’t we all, and partly because he suspected there wouldn’t be many existing SVG illustrations of pelicans riding bicycles in the models’ training data.

SVGs are images, but only once they’re rendered. You probably know them as images that can be scaled almost infinitely without becoming blurry or pixelated. That’s possible because SVG is an XML-based markup language. Any language model with basic text and coding capabilities is theoretically capable of creating an SVG file. You don’t need a benchmark harness, code interpreter, API script, IDE, or complicated set of constraints. You don’t even need to install anything. You give the model one line of text, copy the output into an SVG file if the interface doesn’t render it automatically, and open it in a browser.

The judgment is easy, too. There’s no answer sheet or abstract score to interpret. You just look at the image. Is the bird recognizably a pelican? Does the bicycle have two wheels? Are those wheels connected to a frame? Is the pelican actually riding the bicycle, or is it floating somewhere behind it? Did the model follow the instruction and return SVG code, or did it ignore you and launch an image-generation tool instead?

That single prompt ends up testing instruction-following, coding, valid syntax, object recognition, spatial reasoning, composition, and even restraint. Some models build a functional bicycle and place the pelican correctly. Others add clouds, gradients, grass, animation, and a sunny background while forgetting to connect the front wheel.

It also tells you something about the product around the model. ChatGPT, Claude, and Gemini don’t always give you direct access to a bare LLM. They wrap the model in system instructions, tools, renderers, and routing decisions. The pelican test exposes those decisions almost immediately.

Even big frontier models struggle with the pelican

Gemini won, but not cleanly

I tested the prompt on Claude, Gemini, and ChatGPT. I didn’t use their APIs. I used the normal interfaces provided through my existing subscriptions, which means I was testing the complete products rather than isolated models. Those interfaces don’t expose exact token counts or token-per-second speeds, so I’ve had to leave those figures as N/A.

You can see the results in the galleries I’ve attached. Gemini really smoked everyone else. ChatGPT’s result was weird and disappointing, but that’s mostly because GPT is so eager to jump straight to its image-generation tool. I had to re-emphasize that I specifically wanted SVG code before it finally produced one. That’s arguably still a valid result. The underlying model may be capable of writing a decent SVG, but the ChatGPT product didn’t follow the original instruction.

The Claude models did OK. What’s interesting is how little the result changed relative to the massive difference in cost between them. Claude Haiku 4.5 costs $1 per million input tokens and $5 per million output tokens through the API. Claude Opus 5 costs $5 and $25 respectively, making it five times as expensive. Fable 5 goes even further at $10 per million input tokens and $50 per million output tokens.

Those aren’t the prices I paid through my subscription, but they give us a useful idea of how Anthropic values each model. Given that difference, you’d expect Opus 5 to produce a meaningfully better pelican than Haiku 4.5. Yet here we are. There’s a difference, but not nearly as much as the pricing would suggest.

I couldn’t test the notoriously expensive Fable 5 because I was already out of usage credits. Frankly, I’m also becoming increasingly fed up with Claude’s usage limits.

Local models aren’t that far behind

In fact, I like them better

Gemini certainly produced the most impressive result among the frontier models, but it also added a lot that I never asked for. It made a polished illustration, but it didn’t really capture what I’d expect from such a simple and random prompt. You add clouds if I ask you to add them. You don’t burn my credits without my consent!

Among the local models, I got the best result from Qwen3.6-27B. I ran a Q4-quantized version. It thought for almost three minutes, but the result was exactly what I would’ve expected from a prompt like this: a recognizable pelican, a recognizable bicycle, and not any unnecessary decoration. Qwen3.6-27B is the flagship dense model in Alibaba’s Qwen3.6 family, so it isn’t exactly tiny, but running the quantized version locally still puts it within reach of high-end consumer hardware.

Ornith 35B Q4 also did a great job with both the bicycle and the pelican. The main problem is that the pelican appears to be falling behind the bicycle rather than properly sitting on it. The individual objects are good, but their spatial relationship isn’t.

For reference, I also ran the test with Gemma 4 E2B. This is one of the smallest Gemma 4 models and is intended for mobile and edge hardware rather than a big workstation. It’s the sort of model you could realistically experiment with on Raspberry Pi-class devices. It refused the request at first. I then convinced it that it could, in fact, create an SVG by writing XML, and this was the output.

That initial refusal is a result in itself. The model had the technical ability to complete the task, but it didn’t understand its own capabilities until I explained them. Gemma 4 31B also did a decent job, although the bicycle’s front tire isn’t connected to the rest of the frame.

GLM-4.7-Flash’s result wasn’t great, but it was still impressive given that it generated the output at a ridiculous 112.7 tokens per second. It’s a 30B-total, 3B-active mixture-of-experts model, which helps explain how it managed to run at almost the same speed as the tiny Gemma 4 E2B.

The most disappointing local result came from North-Mini-Code-1.0. Its SVG was even worse than the one produced by Gemma 4 E2B. To be fair, North Mini Code is a mixture-of-experts model with 30 billion total parameters but only 3 billion active during inference. It’s also specifically optimized for agentic coding and terminal work rather than drawing birds on bicycles. Still, I expected more from a model built around code generation.

The local models don’t have a per-token bill ticking upward while I use them, although the hardware and electricity certainly aren’t free. More importantly, their results don’t look several generations behind the expensive frontier models. In some cases, I actually prefer them.

This benchmark doesn’t seem to have been gamed yet

But it probably will be

Goodhart’s law states that when a measure becomes a target, it ceases to be a good measure. That’s increasingly becoming a problem for LLM benchmarks.

Once model providers know which benchmarks matter, they can optimize around them. That doesn’t necessarily mean an engineer is manually feeding the model a benchmark’s answer sheet. Benchmark questions, solutions, code repositories, discussions, and explanations can simply end up in the model’s enormous training dataset.

The model might then perform well because it has encountered the same problem, or something very close to it, before — not because it has developed a more general ability to solve unfamiliar problems.

We’ve already seen major benchmarks lose their value this way. In February 2026, OpenAI stopped reporting scores for SWE-bench Verified, one of the most widely used coding benchmarks. Its investigation found both flawed tests and evidence that frontier models had encountered some of the benchmark’s problems and original human-written fixes during training. OpenAI concluded that higher results were increasingly measuring benchmark exposure rather than genuine improvements in real-world software development.

Benchmarks also matter commercially. OpenAI and Anthropic are preparing for public listings, and benchmark charts are no longer just research results. They’re marketing assets that help shape investor narratives, expected valuations, and the perception that one company is moving faster than another.

When a company launches a new model and fills its presentation with charts showing it beating every competitor, the valuation story becomes much easier to sell. There’s an enormous incentive to make those charts look good.

The pelican test has remained useful because, as far as we know, providers haven’t specifically trained their models to draw pelicans on bicycles.

Still, all it takes is for the pelican test to become important enough, and it could eventually defeat its own purpose. Once providers start proudly including pelican results in launch presentations, someone will inevitably make sure the next model draws a flawless pelican.

At that point, we’ll have to change the prompt. A flamingo on a unicycle, perhaps. Or a raccoon driving a forklift.