Should You Run AI Locally? The 2026 Quiz, Stack, and Iceberg

Somewhere around Opus 4.6 and the February–March 2026 pileup of frontier releases, AI agents stopped being a party trick. You could hand one a repo, a ticket queue, or an inbox and walk away. That’s also when “should I run this locally?” stopped being a question for privacy nerds only, because the thing you’d be running locally was suddenly worth running.

Local had a year too. Qwen3.8-27B made a single 24 GB card genuinely useful. Strata got a 125B model running on a 12 GB gaming GPU. People started chaining Mac Studios over Thunderbolt 5 to run DeepSeek and Kimi at home. And thanks to the RAM shortage, a used 3090 somehow became an appreciating asset.

Most “should you run AI locally” guides end at “install Ollama and pull Llama.” This one won’t. My own research started me on LM Studio, before I figured out what my 5090 could actually do. Now it runs NInfer for language models and ComfyUI for image and video models, plus the occasional LoRA I’m training.

So should you? That depends on who you are, what you’d hand a model, and what’s under your desk. The quiz asks seven questions about exactly that, then tells you whether to go local, stay in the cloud, or split the work, and hands you the stack to run: models, engine, harness, and whether you want the refusals taken out. It takes two minutes, nothing leaves your browser, and you get a link to share your result.

The honest math

Let’s get the money out of the way, because it’s the weakest argument for local in 2026.

The AI build-out ate the world’s memory supply, and everything with RAM on it got expensive. A used 3090 that went for about $700 at the start of the year now runs roughly $1,000–1,400. A 5090 sells for about twice its $1,999 MSRP, if you can find one. NVIDIA and Framework have both raised prices on their 128 GB boxes, and Apple quietly stopped selling its biggest-memory Mac Studio configurations for a while.

Meanwhile, open models served by API got cheap: the big ones cost somewhere between cents and a couple of dollars per million tokens. Four Mac Studios running a trillion-parameter model at home cost tens of thousands of dollars and produce about 30 tokens a second, which at API prices is a few dollars of output a day.

So local rarely wins on cost. It wins when you run agents around the clock against frontier API prices, or when you already own the hardware. Everyone else runs local for better reasons: data that can’t leave, no weekly caps or terms-of-service surprises, working offline, models nobody else filters, and the plain fun of it.

On quality, the gap is real but small. Epoch AI puts open models about four months behind the closed frontier on average. It’s wider on hard agentic coding, which is exactly where the cloud still earns its subscription.

The iceberg

Everyone starts at the tip. The quiz tells you how deep your stack goes.

The model tier list

Model names churn monthly, so read this as a snapshot from October 2026 and trust the memory column more than the names.

Your memory	The go-to	Also worth a look
A laptop, ~8 GB	Qwen3.5 4B, Gemma 4 E4B	Phi-4 Mini
12–16 GB	gpt-oss-20b (on 16 GB)	Gemma 4 12B, Qwen3.5 9B
24 GB	Qwen3.8-27B	Qwen3.6-35B-A3B for fast agents, Gemma 4 31B, Mistral Small 24B finetunes
32 GB (a 5090)	Qwen3.8-27B with room for precision and context	Gemma 4 31B
128 GB unified	gpt-oss-120b	Nemotron 3 Super, Qwen3.5-122B-A10B, Qwen3.8-Flash-Next
200–500 GB	GLM-5.3-Flash, DeepSeek V4 Flash	MiniMax M3, full GLM-5.3 near the top
A Thunderbolt mesh of Macs	DeepSeek V4, Kimi K3	Qwen3.8’s 2.4T flagship, if you hate free time

The rule of thumb: at 4-bit, a model needs about 0.6 GB per billion total parameters, plus 10–30% for context. Total parameters decide whether it fits; active parameters decide how fast it runs. That’s why mixture-of-experts models took over and the 70B dense class quietly died: a 120B MoE with 5B active parameters is faster than a 70B dense model and smarter too.

Not everything is a chatbot. The specialists worth knowing:

Documents and receipts: PaddleOCR-VL and olmOCR. The top of the OCR leaderboard is a model under 1B parameters.
Images: Z-Image Turbo and FLUX.2 [klein] if you need permissive licenses. FLUX.2 [dev] is non-commercial.
Video: Wan 2.2, the last open Wan, and HunyuanVideo 1.5. The best video models are API-only.
3D: TRELLIS.2 and Hunyuan3D 2.1.
Voice: Whisper-class models for transcription, Kokoro for TTS, and Chatterbox for cloning.

Engines: there’s life after Ollama

Ollama is how most people start, and it’s why a lot of them decide local AI is slow and dumb. It’s built on llama.cpp but trails it on speed and new models. It has shipped small default contexts that silently truncate agent prompts. It stores models in a format nothing else can read, and it once listed small distilled models under the name “deepseek-r1.” There’s a well-upvoted r/LocalLLaMA thread titled “Stop using Ollama,” and it’s not subtle.

What to use instead depends on how deep you want to go:

Start: LM Studio. It’s the best GUI on every OS, running llama.cpp and Apple’s MLX under the hood, and it’ll take you further than you’d expect.
Graduate: llama.cpp’s own llama-server, with router mode or llama-swap for switching models. On a Mac, mlx-lm. On Strix Halo, llama.cpp on Vulkan or AMD’s Lemonade. For roleplay, KoboldCpp.
Squeeze: Unsloth’s dynamic quants, ExLlamaV3 with TabbyAPI on NVIDIA, and ik_llama.cpp when a big MoE has to spill into system RAM.
Serve: vLLM or SGLang when several users or agents hit the same box. SGLang’s prefix caching is kind to agents that resend the same long context.
Bleeding edge: NInfer, a hand-tuned engine for exactly one RTX 5090 and a handful of Qwen 27B and 35B builds, with multi-token prediction. Strata, which runs a 125B MoE on a 12 GB card by keeping the hot experts in VRAM and the rest in system RAM. And exo, which links Macs over Thunderbolt 5 with RDMA and actually gets faster as you add machines.

The harness matters as much as the model

The model is the brain; the harness is the body. It’s the program that feeds the model your files, runs its tool calls, and decides what goes in the context. With cloud models you barely notice it. With local models, it’s half the result.

One public benchmark from April 2026 ran the same local models through different harnesses, 1,360 runs in all, against llama-server. Pass rates ranged from about 62% to 77% depending on the harness alone: Pi at the top, then Qwen’s CLI, Claude Code, OpenCode, and Aider. Q8 versus Q4 made no measurable difference. The model sets the ceiling; the harness decides how close you get to it.

The ones worth knowing:

Pi, and its fork Oh My Pi, for minimal terminal agents that local models handle well.
OpenCode, which has the widest support for local providers.
Claude Code, pointed at llama.cpp or LM Studio with ANTHROPIC_BASE_URL. It works, but Anthropic doesn’t support it.
Codex, with its --oss mode.
Hermes Agent, Goose, and Cline or Kilo. Roo Code shut down in May; Kilo and Zoo Code carry on its fork.
For chat, Open WebUI or LibreChat. For roleplay, SillyTavern.

And the checklist that saves you a weekend:

Run llama-server with --jinja. Without the chat template, tool calls leak out as plain text.
Give agents 32–64K of context at minimum. Most “my local agent is dumb” posts are really “my context is 4K” posts.
Use instruct models for agents and base models for tab completion. They’re bad at each other’s jobs.
When a new model’s tool calling is broken, check for Unsloth’s fixed chat template before you blame the model.

Uncensored, abliterated, and why it matters

Every hosted model ships with somebody else’s lines drawn in it. Sometimes they’re reasonable. Sometimes you get refused for asking how to kill a child process. Run it yourself and you choose where the lines are.

How abliteration works. Research in 2024 found that refusal mostly lives along a single direction in a model’s activations. Abliteration finds that direction and projects it out of the weights, with no retraining.
Heretic automates it. It searches for the edit that removes the most refusals while drifting least from the original model. Norm-preserving abliteration is the newer technique aimed at the “lobotomized” complaint.
Finetunes are the other route. Hermes 4, Dolphin, and TheDrummer’s models are trained rather than edited, and they usually keep tool calling better than abliterated builds.
Some models start out looser. Stock DeepSeek and Mistral refuse the least out of the box, and Gemma refuses the most. The UGI leaderboard is where the community keeps score.

The trade-offs are real. Expect a small hit to capability. And abliteration doesn’t distinguish between dumb refusals and good ones: every refusal goes, so you’re the safety layer now, and the law still applies to what you do with the output.

Who should run what

Software engineers: local is great at tab completion, overnight agents, and code you can’t paste anywhere. Interactive, multi-file agent work is still where the frontier earns its keep, so most engineers end up hybrid.
IT and homelab: runbooks and logs are full of hostnames and credentials, and Home Assistant voice is better when it doesn’t phone home. If you go air-gapped, check for remote embedding calls first; that’s where air gaps usually leak.
Small businesses: receipts, invoices, and customer data are the easiest local win, because document models are tiny now. Paperless-ngx with a local model does the job, and a decent scanner matters more than a big GPU.
Studios and indie game developers: image, 3D, and voice run well locally, and unreleased assets stay unreleased. Read every license first. FLUX.2 [dev] is non-commercial, and Tencent’s Hunyuan license excludes the EU, the UK, and South Korea.
Law, medicine, and finance: after United States v. Heppner (S.D.N.Y., February 2026), where a defendant’s chats with a consumer AI app were held not privileged, consumer chat apps are a real risk. Enterprise plans with zero retention exist, but a box in your office isn’t a third party at all.
Writers and roleplayers: local is home turf, with custom samplers, character cards, and nobody’s content policy. Use KoboldCpp behind SillyTavern and set the context yourself.

If you made it this far and still want to stay in the cloud, that’s a fine answer. The quiz will tell you that too.

How this was made: I tried building this with Hermes Agent on a local model first. It didn’t make the cut (that’s a future post). The research, quiz, and article were built with Claude, then reviewed and edited by me. Hybrid, in other words: local for the attempt, cloud for the hard part. Which is roughly what the quiz will tell most of you.

Questions people ask

Is Ollama bad?

It isn't bad, it's training wheels. It's built on llama.cpp, trails it on speed and new models, stores models in a format nothing else reads, and has shipped small default context windows that silently truncate agent prompts. Start with LM Studio if you want a GUI, and move to llama.cpp's llama-server once you know what you want.

How much VRAM do I need to run a local LLM?

At 4-bit, figure about 0.6 GB per billion total parameters, plus 10–30% for context. A 24 GB card runs the 27B-class models that are genuinely good in 2026, 128 GB of unified memory runs the 120B-class mixture-of-experts models, and the 600B–2.8T giants want a server or a cluster of Macs.

What is an abliterated model?

A model with its refusal behavior edited out of the weights. Refusals mostly live along a single direction in the model's activations; abliteration finds that direction and projects it out, with no retraining. Tools like Heretic automate it while keeping the model as close to the original as possible. Expect a small capability cost, and remember it removes every refusal, not just the silly ones.

Can I use Claude Code or Codex with a local model?

Yes. llama.cpp's server and LM Studio both expose an Anthropic-style Messages endpoint, so Claude Code runs against them with ANTHROPIC_BASE_URL (Anthropic doesn't support this, so you troubleshoot it yourself). Codex has an --oss mode. In one public benchmark, smaller harnesses like Pi actually did better with local models than Claude Code did.

Is a Mac Studio cluster worth it?

As a project, absolutely. As a cost decision, no. Four Mac Studios linked over Thunderbolt 5 run trillion-parameter models at roughly 25–30 tokens a second, for tens of thousands of dollars. The same open models cost cents to a couple of dollars per million tokens through an API, and they come back faster.

Is running AI locally cheaper than the cloud?

Rarely, in 2026. The RAM shortage roughly doubled hardware prices, while open models served by API got cheap. Local pays for itself when you run agents nonstop against frontier API prices, or when you already own the hardware. Most people who run local do it for privacy, independence, offline use, models nobody else filters, or because it's fun.