Part of Local AI & LLMs — running large language models on your own hardware.
Every guide to running language models locally begins with installation. Installation is not where people get stuck.
They get stuck one step earlier — hardware bought for a model that was never going to fit on it — and one step later, when the model answers beautifully in a chat window and there is no obvious way to make it do actual work. The middle part, the building and the installing, is the only part that is well documented.
This is the whole path in the order you actually walk it. Every step links to the detailed article, so you can drop in wherever you are stuck.
1. What will your machine actually run?
This question belongs first and almost never is. Parameter count is the number everyone quotes and the least useful one: a 35B model with 3B active parameters behaves nothing like a dense 27B, in speed or in memory. What decides whether a model is usable on your machine is VRAM after loading, and that number is rarely published.
| Model | VRAM after load | Composite tok/s | Smallest card that holds it |
|---|---|---|---|
| Gemma-4-26B-A4B (MoE) | 18 GB | 87.8 | 20 GB |
| Qwen3.6-35B-A3B with MTP | 24 GB | 82.7 | 24 GB |
| Qwen3.6-35B-A3B | 24 GB | 40.8 | 24 GB |
| Qwen3.6-27B (dense) | 17 GB | 33.1 | 20 GB |
Those are measured, not estimated — one 48 GB card, llama-server on Vulkan, Q4_K_M, context 8192, ten runs per prompt. Change the backend and the ranking moves, which is exactly why the conditions are written down. The benchmark lab has the full leaderboard, time-to-first-token next to throughput, and every tested model mapped to the smallest home GPU that will hold it.
Two of those rows are worth staring at. Gemma-4-26B is faster than the 35B models and fits in less memory, because most of its parameters sit idle on any given token. And the fastest model in the table is not the most responsive one — throughput and latency are different measurements, and a model that wins on tok/s can still feel sluggish in an editor.
What makes any of this fit on a consumer card is quantization. The complete guide to offline LLMs covers what dropping weights from 16-bit to 4-bit actually costs you, which is less than people expect, and where it starts to hurt, which is sooner than the benchmarks admit.
2. The engine: control against convenience
There are four common ways to run the same model file, and choosing between them is choosing how much you want to be able to tune.
llama.cpp is the engine most of the others are wrapped around. Running it directly gives you every flag — threads, memory mapping, GPU split, context — and hands you the responsibility for all of them. If you are going to build it yourself on Windows, the step-by-step build covers the toolchain, the CUDA flags and the errors worth recognising.
Ollama packages models the way Docker packages services: one command pulls and runs, and the configuration you never wanted to write is written for you. You trade away the flags. For most people that is the right trade, right up until the day it is not.
Free ebook
Free AI Video, Generated Locally
Working scripts and measured benchmarks. Free.
No spam. Unsubscribe at any time.
GPT4All and Jan are the desktop applications — install, download a model from a list, type. The comparison comes down to a genuine split rather than a winner: one optimises for working immediately, the other for being taken apart.
LM Studio sits in between, and it is the one that grew the most useful second half: a GUI to try models in, an OpenAI-compatible server behind it, and a CLI so you never have to open the GUI again. That path runs through the API article and the CLI article.
3. –n-gpu-layers: the flag between fast and unusable
More people arrive here asking what this one flag means than any other question about local models, so here is the answer before anything else.
--n-gpu-layers (short form -ngl) sets how many of the model’s transformer layers are kept in VRAM. Whatever is left over stays in system RAM and runs on the CPU, at roughly a tenth of the speed.
llama-server -m model.gguf -ngl 0 # nothing on the GPU, pure CPU llama-server -m model.gguf -ngl 20 # first 20 layers on the GPU, rest on the CPU llama-server -m model.gguf -ngl -1 # every layer on the GPU |
In llama-cpp-python, n_gpu_layers=-1 is documented as “all layers”. llama.cpp’s own tools treat any value at or above the model’s layer count the same way — which is why half the examples online use -ngl 99, a lazier spelling of the same instruction.
Two things the flag does not do, and each one costs somebody an evening:
- It is not auto-detection.
-1does not mean “as many as fit”. It means all of them. If the model is bigger than your free VRAM, you get an out-of-memory failure, or the driver quietly starts spilling and generation slows to a crawl. Lowering the number until it fits is your job, not the loader’s. - Layers are not gigabytes. The count is per model, so a number that offloads a 7B model completely will offload maybe a third of a 35B one. There is no universal right value — only the highest one your card survives.
The reason it matters this much: partial offload is not a gentle slope. A model that fits entirely in VRAM and a model missing four layers are not 10 % apart in speed, they are several times apart, because every token now waits on the slowest layers. The llama.cpp article puts this parameter next to the others that share the same trade-off — thread count, and whether the model is memory-mapped or locked into RAM.
4. It runs, and it is still too slow
Once every layer is on the GPU, the next gain comes from generating more than one token at a time. Multi-Token Prediction is speculative decoding built into the model, and on the same hardware and the same file it took throughput from 49.7 to 91.8 tok/s — 1.85× on average, and a clean 2.00× on code generation, where the next few tokens are the most predictable.
The MTP benchmark has the per-task numbers with standard deviations, the flags that turn it on, and the case that matters more than the wins: where it does nothing, and why the same model failed to do it under a different runtime on the same card.
5. The API is the hinge
This is the step that converts a working model into something useful, and it is one line.
from openai import OpenAI client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed") |
Every serious local runtime speaks the OpenAI protocol now, so anything written against that protocol — editors, agent frameworks, your own scripts — points at your machine by changing a URL. Running local LLMs with OpenAI-compatible APIs sets up LM Studio, Ollama and vLLM behind one client, and is honest about which of the three belongs in production and which does not.
6. Making it do the work
With an endpoint running, the question stops being technical.
In the editor. LM Studio as a coding-agent backend covers the three endpoint families it exposes, how to point Continue.dev or Cursor at them, and the measured latency — including how a multi-turn session grows its own context until it slows down.
Without the GUI at all. The lms command line downloads, loads, configures and unloads models from a script — which is what you need the moment a benchmark or a batch job has to run unattended.
One model, more than text. Qwen3.6-35B-A3B in practice uses a single local model for text, images and sampled video frames, with three scripts and the real runs behind them — including where it read a chart correctly and where it did not.
Against your own documents. A local model knows nothing about your files until you give them to it, and pasting does not scale. That is retrieval, and it starts with how you cut the documents up — the decision that determines the quality of everything downstream.
On a schedule, on your own text. The content remix pipeline turns one article into five formats with no cloud API — the clearest example of the payoff, because the cost per run is electricity.
As something that works while you do not. A private AI assistant is the difference between a chat window and delegation: what changes when the model holds your context, and which permissions you should not hand it.
7. Whether any of this is worth it
For a company the calculation is different from a hobbyist’s, and it is not mainly about the API bill. The digital independence manifesto works through the case for small models under your own roof — where specialised small models beat large general ones, what fine-tuning a model on your own data really takes, and what GDPR looks like when the data never leaves the building.
8. The part underneath all of it
Open models exist because open datasets exist, and the speech ones are still thin outside a handful of languages. Mozilla Common Voice is the one you can contribute to in a few minutes, and the reason local speech works at all today — transcription with Whisper and local text-to-speech both run offline on the same machine as the text model.
Where to start, depending on where you are stuck
- I do not know if my card is enough — the measured VRAM table, then the quantization section of the offline guide
- I just want it running today — Ollama, or a desktop app if you would rather not touch a terminal
- It runs and it is painfully slow — the offload flags first, MTP second
- It works and I want it in my editor — the API, then the backend setup
- I have to justify it to somebody — the manifesto
Numbers on this page come from one workstation and are dated on purpose. If yours disagree — different card, different backend, different quantization — tell me what you measured. That is the most useful message I get, and it usually ends up in the article.
Free ebook
Free AI Video, Generated Locally
Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.
No spam. Unsubscribe at any time.
Leave a Reply