Part of Running LLMs on Your Own Hardware: From “Will It Even Fit” to a Model That Does Your Work — running large language models on your own hardware.
05_agent_api_test.py · 10 runs per scenario · model google/gemma-4-12b-qat loaded in LM Studio · AMD PRO W7900 48 GB · Windows 11.
The LM Studio API turns a model on your own machine into a drop-in backend for coding agents and custom apps. You load a model in LM Studio, start the local server, and point your tool at localhost. From there, every request is free, private, and offline.
LM Studio exposes three families of endpoints, and this article shows how to use each one with real, captured examples. It then reports latency measured on gemma-4-12b-qat running inside LM Studio, so you know what to expect on real hardware.
- LM Studio supports three endpoint families: LM Studio native REST, OpenAI-compatible, and Anthropic-compatible.
- Point any OpenAI or Anthropic SDK at
http://localhost:1234and it works unchanged. - On a W7900, gemma-4-12b-qat holds ~44 tok/s with sub-second first-token latency for typical prompts.
- The native REST API handles model management: list, load, and unload without the GUI.
Why run your coding agent on a local backend?
Cloud assistants bill per token, and the meter never stops. A local backend removes that cost entirely. You pay once for the GPU, and every request afterwards is free.
Privacy is the second reason. Your source code and prompts stay on the machine, so nothing reaches a third party. Latency is the third: a local server answers without a round trip to a data center, so chat feels responsive even offline. The only trade-off is hardware, because you need enough VRAM to hold the model.
The economics are simple. A busy developer can push millions of tokens through an assistant each month, and cloud pricing turns that into a recurring bill. A local backend converts the cost into a one-time GPU purchase. After that, an autocomplete storm or an all-day refactor costs nothing extra, which changes how freely you use the model.
The three supported endpoint families
Start the server first. From the LM Studio GUI, open the Developer tab and click Start, or use the CLI:
lms server start
lms load google/gemma-4-12b-qat --gpu maxOnce the server runs on port 1234, three families become available. The table shows when to reach for each.
| Family | Base path | Use it for |
|---|---|---|
| LM Studio native REST | /api/v0/ | Model management: list, load, unload |
| OpenAI-compatible | /v1/ | Chat, completions, embeddings — the workhorse |
| Anthropic-compatible | /v1/messages | Running Claude-style code locally |
Family 1 — LM Studio native REST API
The native API lives under /api/v0/. Its strength is rich model metadata that the OpenAI layer does not expose. A single call tells you the architecture, quantization, load state, and context length of every model.
Request:
GET http://localhost:1234/api/v0/modelsResponse (200 OK, trimmed):
{
"data": [
{
"id": "text-embedding-nomic-embed-text-v1.5",
"type": "embeddings",
"arch": "nomic-bert",
"quantization": "Q4_K_M",
"state": "loaded",
"max_context_length": 2048
},
{
"id": "google/gemma-4-12b-qat",
"type": "vlm",
"arch": "gemma4",
"quantization": "Q4_0",
"state": "loaded",
"max_context_length": 262144,
"capabilities": ["tool_use"]
}
]
}Two details stand out from the real response. First, gemma-4-12b-qat reports "type": "vlm" and "capabilities": ["tool_use"], so it handles images and tool calls, not just text. Second, its max_context_length is 262 144 tokens, even though it was loaded with a smaller working context. Use this endpoint in scripts to check what is loaded before you send work to it.
Family 2 — OpenAI-compatible API
This family under /v1/ is the one you will use most. Any tool built for OpenAI works after you change the base URL, which is why Continue.dev, Cursor, and Aider all connect with a single line of config.
Free ebook
Free AI Video, Generated Locally
Working scripts and measured benchmarks. Free.
No spam. Unsubscribe at any time.
GET /v1/models — list loaded models
GET http://localhost:1234/v1/modelsThe response lists each loaded model by id, such as google/gemma-4-12b-qat and text-embedding-nomic-embed-text-v1.5. Agents call this first to discover which model to target.
POST /v1/chat/completions — the main endpoint
This is the endpoint that powers chat and code edits. Here is a real exchange captured from the server:
POST http://localhost:1234/v1/chat/completions
Content-Type: application/json
{
"model": "google/gemma-4-12b-qat",
"messages": [
{"role": "user", "content": "In one sentence, what is an OpenAI-compatible API?"}
],
"max_tokens": 800
}Response content:
"An OpenAI-compatible API is a software interface that mimics OpenAI's
request and response formats, allowing developers to switch between
different AI models and providers without changing their application's
underlying code."The Python client makes this trivial. Note that Gemma-4 is a reasoning model, so it emits internal reasoning_tokens before the answer. Give it enough max_tokens to finish thinking and still produce output.
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1234/v1", api_key="not-needed")
resp = client.chat.completions.create(
model="google/gemma-4-12b-qat",
messages=[{"role": "user", "content": "Write a unit test for this function..."}],
max_tokens=800,
stream=True,
)
for chunk in resp:
print(chunk.choices[0].delta.content or "", end="", flush=True)POST /v1/completions — legacy text completion
The legacy completions endpoint takes a raw prompt instead of a message list. It still works, but there is a caveat worth knowing.
POST /v1/completions
{"model": "google/gemma-4-12b-qat", "prompt": "The three primary colors are red, blue and", "max_tokens": 12}An instruct-tuned model like Gemma-4 is optimized for the chat format, so raw completions can drift or repeat. For agent work, prefer /v1/chat/completions. Keep the legacy endpoint only for tools that still require it.
POST /v1/embeddings — semantic vectors
Embeddings power semantic search and retrieval. Load an embedding model such as nomic-embed-text, then call:
POST /v1/embeddings
{"model": "text-embedding-nomic-embed-text-v1.5", "input": "local agent backend"}The real response returns a 768-dimensional vector, beginning [0.0462, 0.0285, -0.1597, ...]. Store these vectors in a local index, and your agent gains retrieval over your own documents without any cloud service.
Family 3 — Anthropic-compatible API
LM Studio also speaks the Anthropic Messages API at /v1/messages. Therefore code written for Claude runs locally after only a base-URL change. The request uses Anthropic headers:
POST http://localhost:1234/v1/messages
x-api-key: lmstudio
anthropic-version: 2023-06-01
{
"model": "google/gemma-4-12b-qat",
"max_tokens": 400,
"messages": [{"role": "user", "content": "Say hello in one short sentence."}]
}Response (real):
{
"type": "message",
"role": "assistant",
"content": [{"type": "text", "text": "Hello!"}],
"stop_reason": "end_turn",
"usage": {"input_tokens": 20, "output_tokens": 3, "cache_read_input_tokens": 0}
}Notice the response uses content blocks rather than a plain string, exactly as the real Anthropic API does. The usage object even reports cache_read_input_tokens, so prompt-caching accounting is present. This lets you test Claude-targeted code against a local model before spending on the real API.
Tool use and structured output
The native metadata showed "capabilities": ["tool_use"] for Gemma-4. That matters for agents, because tool use is how a model triggers real actions: search a file, run a query, call an API. You declare the tools in the request, and the model replies with a structured call instead of prose.
resp = client.chat.completions.create(
model="google/gemma-4-12b-qat",
messages=[{"role": "user", "content": "What files changed in the last commit?"}],
tools=[{
"type": "function",
"function": {
"name": "git_diff",
"description": "Return the list of files changed in a commit",
"parameters": {"type": "object", "properties": {"ref": {"type": "string"}}},
},
}],
)
# resp.choices[0].message.tool_calls -> the model's structured requestFor simpler needs, ask for JSON directly and set a low temperature. The model then returns a parseable object that drops straight into your pipeline, which removes brittle text parsing. Together, tool use and JSON output turn a chat model into the decision layer of a real agent.
Managing models without the GUI
The native REST family pairs with the lms CLI for headless automation. You can list, load, and unload models from a script, which is ideal on a server with no desktop:
lms ls # models on disk
lms load google/gemma-4-12b-qat --gpu max
lms ps # what is loaded, and its context
lms unload google/gemma-4-12b-qatBecause loading a model reserves VRAM, unload it when a batch job finishes. That frees the GPU for the next task, and it keeps a shared workstation tidy.
Quick reference — every endpoint at a glance
| Endpoint | Family | Use case |
|---|---|---|
GET /api/v0/models | Native REST | Rich model metadata + state |
GET /v1/models | OpenAI | List loaded models |
POST /v1/chat/completions | OpenAI | Chat and code edits |
POST /v1/completions | OpenAI | Legacy raw completion |
POST /v1/embeddings | OpenAI | Semantic vectors |
POST /v1/messages | Anthropic | Claude-style messages |
Connect your editor
Continue.dev is a VS Code and JetBrains extension. Point it at LM Studio in ~/.continue/config.json:
{
"models": [{
"title": "Gemma-4 (LM Studio)",
"provider": "openai",
"model": "google/gemma-4-12b-qat",
"apiBase": "http://localhost:1234/v1",
"apiKey": "not-needed"
}]
}Cursor accepts the same base URL in its model settings. Aider connects from the terminal with two environment variables, then runs inside any repository:
set OPENAI_API_BASE=http://localhost:1234/v1
set OPENAI_API_KEY=not-needed
aider --model openai/google/gemma-4-12b-qatAll three share the same contract. So once one works, the others need only a base-URL change. That portability is the whole point of the OpenAI-compatible layer.
Benchmark methodology
Numbers here come from a real script run, not estimates. The benchmark sends three coding prompts of growing size, then runs a multi-turn debugging session. Each scenario runs 10 times, and the reported values are means with standard deviation.
- GPU: AMD PRO W7900, 48 GB VRAM
- Model:
google/gemma-4-12b-qat, loaded in LM Studio - Metric: Time To First Token (TTFT) and throughput in tokens per second
- OS: Windows 11
Latency and throughput results
| Scenario | Mean TTFT | Throughput |
|---|---|---|
| Short review (~60 tokens) | 848 ms | 43.5 tok/s |
| Medium review (~350 tokens) | 746 ms | 45.8 tok/s |
| Large refactor (~1 800 tokens) | 1 754 ms | 41.5 tok/s |
Throughput stays close to 44 tok/s across every prompt size, which is comfortable for interactive chat. TTFT varies more, because Gemma-4 is a reasoning model and its thinking phase fires unevenly on complex prompts. When latency matters more than reasoning depth, cap max_tokens tightly and keep prompts focused.
Context growth in a multi-turn session
| Turn | Context used | Mean turn time |
|---|---|---|
| Turn 1 (bug report) | 23 tokens | 6.25 s |
| Turn 2 (follow-up) | 69 tokens | 6.55 s |
| Turn 3 (verification) | 119 tokens | 6.82 s |
The turn times barely move as context grows from 23 to 119 tokens. Therefore, at this scale, growing history adds almost nothing to latency. A full three-turn exchange finishes in under 20 seconds, most of it generation rather than prefill.
Frequently asked questions
Does the LM Studio API need an internet connection?
No. Once the model sits on disk, the server runs fully offline. Every request stays on your machine, so you can code on a plane or behind an air-gapped firewall.
Which endpoint should a coding agent use?
Use POST /v1/chat/completions. It is the most widely supported endpoint, and every major editor extension expects it. Reserve the legacy completions endpoint for tools that still require it.
Can I really run Claude code against LM Studio?
Yes. Point the Anthropic SDK at http://localhost:1234 and use any key. The /v1/messages endpoint returns proper content blocks, so most Claude code runs unchanged for standard message flows.
How much VRAM does gemma-4-12b-qat need?
The QAT build loads in about 7 GB, so it fits comfortably on a 12 GB card and leaves plenty of headroom on a 48 GB W7900 for a long context or a second model.
Why does the first request feel slower?
The first call after loading a model pays a one-time warm-up. Later calls reuse the loaded state and respond faster, so exclude the first request when you measure steady-state latency.
Summary
A model in LM Studio makes a capable, private agent backend. It exposes three endpoint families: native REST for model management, the OpenAI-compatible layer for chat, completions and embeddings, and the Anthropic-compatible endpoint for Claude-style code. On a W7900, gemma-4-12b-qat held around 44 tok/s with sub-second first-token latency on typical prompts, and context growth barely affected turn times. Reproduce every number with the script: 05_agent_api_test.py.
Companion code — every script and benchmark from this series lives in one repository: github.com/bestin-it/qwen-local-ai-scripts.
Free ebook
Free AI Video, Generated Locally
Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.
No spam. Unsubscribe at any time.
Leave a Reply