# Running LLMs Locally with Ollama

I wanted to run AI models on my own machine — no API keys, no usage limits, no sending my code to someone else's server. Same philosophy as [the home server](/articles/self-hosting-home-server): own your tools. Ollama makes this surprisingly easy if you have a decent GPU.

## What Actually Runs Well on 8GB VRAM

After pulling way too many models and benchmarking them, here's what I actually kept:

**The daily drivers:**
- `qwen2.5-coder:7b` — This thing is scary good at Python. Fits entirely in GPU memory, responds fast, and the code quality rivals cloud models for everyday tasks.
- `qwen2.5:14b` — My go-to for writing and anything that needs good language. Slower than the 7B models but the quality jump is worth it.
- `llama3.2:3b` — When I just need a quick answer and don't want to wait. Two seconds and done.

**For when you need to think harder:**
- `deepseek-r1:14b` — Reasoning + code. When the 7B coder gives you nonsense on a complex problem, this one usually gets it.

Pull with `-q4_K_M` quantization for the best quality-to-size balance: `ollama pull qwen2.5:7b-instruct-q4_K_M`

## Local or Cloud? Matching the Task to the Tool

This took me a while to figure out. The answer isn't "always local" — it's about matching the task to the tool.

| Task                   | Local | Cloud | Why                                          |
| ---------------------- | ----- | ----- | -------------------------------------------- |
| Code completion        | ✅     |       | You do this 100 times a day. Privacy matters. |
| Refactoring            | ✅     |       | Repetitive and standardized                  |
| Bug explanations       | ✅     |       | 7B handles this fine                         |
| Architecture decisions |       | ✅     | You need the big brain for this              |
| Documentation          |       | ✅     | One-time cost, quality matters               |
| Writing tests          | ✅     | ✅     | Draft locally, polish with cloud             |

The pattern: if it's frequent, private, or "good enough" works — run it locally. If it's a one-time thing where quality is everything — pay for the cloud model.

## Speed by Model Size on an 8GB Laptop GPU

With an RTX 5050 Laptop GPU (8GB VRAM) — the 4050 Laptop is 6GB, so scale expectations down a
tier if that's what you've got:

Rough impression from daily use, not a benchmark — no fixed prompt set, no repeated runs, just
what it's felt like:

- **3B–7B models:** feels instant. This is your sweet spot for daily use.
- **12B–14B models:** a noticeable pause but totally usable. Better quality.
- **34B+ models:** spills into RAM, gets slow. Only use these when you really need the reasoning power and don't mind waiting.

## WSL2 GPU Setup

If you're on Windows, Ollama runs great through WSL. The [GPU passthrough](https://learn.microsoft.com/en-us/windows/ai/directml/gpu-cuda-in-wsl) just works now — verify with `nvidia-smi` inside WSL. If it fails, run `wsl --update && wsl --shutdown` and restart.

No extra CUDA setup needed. Ollama detects it automatically.

## See Also

- [Open WebUI Setup](/articles/open-webui-setup) — give your local models a proper interface
- [WSL for Development](/articles/wsl-for-development) — the environment where Ollama runs
- [Self-Hosting on Two Old Laptops](/articles/self-hosting-home-server) — same philosophy, different rabbit hole
- [Claude Prompts](/articles/claude-prompts) — structured prompts that work with local models too
