Part of Running LLMs on Your Own Hardware: From “Will It Even Fit” to a Model That Does Your Work — running large language models on your own hardware.
scripts/serving_bench.py (200 records, 20 repetitions) and kev.benchmark · AMD Radeon PRO W7900 48 GB · PyTorch 2.9.1+rocm7.2.1 · Windows 11 Pro for Workstations.
Running Kev on AMD ROCm costs you time, not accuracy. On a Radeon PRO W7900 the 4B decision model answers six typed questions in 171 ms, against 29.5 ms on an H100. However, the quality numbers land exactly on the published values: 0.817 out-of-domain accuracy, 0.243 Brier, 0.042 ECE. Across 280 parity questions, not a single answer changed against the fp32 reference path. Therefore the AMD card is a real option for batch and offline decision work, and a poor one for online traffic.
- Kev-4B on a Radeon PRO W7900 needs 171 ms per 6-question request against 29.5 ms on an H100 — about 5.8× slower, roughly NVIDIA L4 class.
- ROCm reproduced the published accuracy exactly: 0.817 out of domain and 0.872 in domain, with zero argmax changes versus fp32 across 280 questions.
- Kev-4B beats Kev-9B in practice: it automates 62% of decisions at a 5% error budget, while Kev-9B manages only 45%.
- The ROCm build of PyTorch 2.9.1 on Windows ships no Triton, so
flash-linear-attentionand the fused Qwen3.5 kernels never load. - The state prefix cache cuts a 2,392-token request from 767 ms to 178 ms, a 77% saving.
This article covers inference only: latency, throughput, memory and answer quality for the released Kev-4B and Kev-9B checkpoints. It does not cover fine-tuning, and it does not test Linux, where the ROCm story differs.
What Kev is, and why the GPU choice matters
Kev is a family of open decision models built on Qwen3.5 base checkpoints. In each case the release is a LoRA adapter plus a pointer head, published as an open-weights alternative to the hosted Jev API. Moreover the API contract matches TypeSafe’s System One endpoint, so an existing client can point at a local server unchanged.
The important part is what the model does not do. Kev never generates text. Instead it takes one document, called the state, plus a set of typed questions, and returns a calibrated probability distribution per question in a single forward pass. Specifically, question types are yes/no, multiple choice and rating.
That design changes the hardware maths completely. A generative model streams tokens, so throughput dominates. A decision model answers once, so latency per request is the only number that matters. As a result, a card that looks mediocre on token generation can still be perfectly usable here, because each call finishes in well under a second.
Workstation owners face a specific question. The W7900 offers 48 GB of VRAM at a fraction of a data-centre card’s price, but ROCm support on Windows remains thin. So the real question is not whether the model runs, but whether it runs correctly and how much speed the platform costs.
Three real questions from the benchmark suites
Abstract descriptions of a decision model rarely land, so here are three records taken straight from the same frozen suites the benchmarks scored, answered by Kev-4B on the W7900. They live in the project’s evals/v4/transfer-v4 and evals/v7/decision-v7 directories, and the project also publishes its frozen suites as the kev-suites dataset. Record IDs appear with each example, so you can look up the originals.
A yes/no question against a stated rule
This pair differs by one word: the name of the person who signed. Consequently it shows whether the model reads the rule or guesses from surface wording. Both records sit in transfer-v4 under the id contrastive/authorization/v3-legacy-transfer-authorization-0000.
{
"state": {
"policy": "A refund is authorized only when its sole authorization was signed by
someone who may authorize refunds for that account.",
"case": "Only Priya may authorize refunds for account 18. The refund amount is $900.
The sole authorization for this refund on account 18 was signed by Priya."
},
"questions": { "decision": { "type": "noul", "instructions": "Is the refund authorized?" } }
}
Answer: noul = 0.9935 gold = true correctSwap the signature to Elin, who may not authorize refunds on that account, and nothing else:
Answer: noul = 0.0057 gold = false correctThe probability moves from 0.99 to 0.006 on a single name change. In other words the rule is doing the work, not a keyword match.
A multiple-choice question with six options
This record (emotion/test/230) comes from the emotion source, which the model never saw during training. Note that the answer is a full distribution, not a label.
state: "i am feeling rather damaged"
question: { "type": "choice", "instructions": "Which emotion does the writer express?",
"criteria": { "sadness": null, "joy": null, "love": null,
"anger": null, "fear": null, "surprise": null } }
Answer: choice = "sadness" confidence = 0.794 gold = sadness correct
probabilities: sadness 0.829 | fear 0.100 | anger 0.049 | surprise 0.018 | joy 0.002 | love 0.002A rating question the model gets wrong
Date arithmetic is Kev’s documented weak spot, and this case (contrastive/deadline/v3-legacy-transfer-deadline-0001/a) shows it failing honestly. The report arrived nine days before its deadline, so the correct answer is “On time”.
state: { "policy": "Reports received by the deadline are on time. Reports received within
7 days after the deadline are late but accepted. Later reports are refused.",
"case": "The report was received on August 1, 2026. The filing deadline for
Elin's report was August 10, 2026." }
question: { "type": "score", "instructions": "How late is this report?",
"criteria": ["On time", "Late but accepted", "Refused"] }
Answer: score = 0.691 confidence = 0.763 gold = 0 (On time) WRONG
probabilities: On time 0.392 | Late but accepted 0.525 | Refused 0.083However, the calibration earns its keep here. The model reports 0.763 confidence rather than pretending to be sure, and that is below the threshold measured later in this article for safe automation. As a result a selective-automation pipeline routes this case to a human instead of acting on it.
Setting up Kev on AMD ROCm under Windows
Beyond copying a working environment, three things break, and none of them appear in the project documentation: the torch version pin, the missing Triton, and a POSIX-only import. Each one stops the run completely, so it pays to handle all three before starting.
Clone a working ROCm environment instead of building one
ROCm wheels for Windows are not on any public pip index. In practice the only working build on this machine arrived with ComfyUI: torch 2.9.1+rocm7.2.1 on Python 3.12, reporting HIP 7.2. Therefore the fastest safe route copies that environment rather than rebuilding it.
# Copy a known-good ROCm venv instead of touching the original
robocopy "C:ComfyUIvenv" "C:KevTesting.venv" /E /MT:32 /NFL /NDL
# Pin torch so pip cannot silently swap the ROCm build for a CPU one
echo torch==2.9.1+rocm7.2.1 > constraints.txt
.venvScriptspython.exe -m pip install -c constraints.txt ^
"transformers>=5.17,<6" "peft>=0.21" "datasets>=3.0" ^
"scikit-learn>=1.9.1" "fastapi>=0.115" "uvicorn>=0.30" "typesafe-sdk>=0.6.0"The constraints file matters more than it looks. Several of those packages depend on torch, so without the pin pip happily replaces the ROCm build with the CPU wheel. Then the model still runs, just fifty times slower, and nothing warns you.
Free ebook
Free AI Video, Generated Locally
Working scripts and measured benchmarks. Free.
No spam. Unsubscribe at any time.
Work around the torch version pin
Kev pins torch>=2.6,<2.9 in its project metadata. Meanwhile the only usable ROCm build is 2.9.1, so a normal install refuses. Installing the package without its dependencies resolves the conflict, and in practice the pin turned out to be conservative — every benchmark passed on 2.9.1.
.venvScriptspython.exe -m pip install --no-deps -e kevTwo imports that break on Windows
The first failure appears the moment the server starts:
ModuleNotFoundError: No module named 'fla'fla is flash-linear-attention, which Kev uses for fused Qwen3.5 kernels. It builds on Triton, and this ROCm wheel for Windows ships no Triton at all. Consequently the fused path is unavailable, and KEV_FUSED=0 declines it cleanly.
The second failure hits the evaluation harness:
ModuleNotFoundError: No module named 'fcntl'fcntl is POSIX-only, yet kev.suite imports it at module level for advisory file locking. So kev.benchmark refuses to load on Windows before it reaches any model code. A small shim on PYTHONPATH fixes it, because those locks only ever guarded multi-process orchestration:
# winshim/fcntl.py - msvcrt gives real non-blocking locks; anything it rejects degrades
# to a no-op, which is safe because a single benchmark run never contends.
import msvcrt
LOCK_SH, LOCK_EX, LOCK_NB, LOCK_UN = 1, 2, 4, 8
def flock(file, operation):
fd = file.fileno() if hasattr(file, "fileno") else file
try:
if operation & LOCK_UN:
msvcrt.locking(fd, msvcrt.LK_UNLCK, 1); return
mode = msvcrt.LK_NBLCK if operation & LOCK_NB else msvcrt.LK_LOCK
msvcrt.locking(fd, mode, 1)
except OSError as error:
if operation & LOCK_NB and getattr(error, "errno", None) in (11, 36):
raise BlockingIOError(str(error)) from None
returnHow I benchmarked Kev on AMD ROCm
In total, two harnesses ship with the project, and this run used both. First, the serving benchmark measures model time across four request shapes, with a fresh state and with a repeated one. It also compares bf16 output against an exact fp32 pass, which is the check that matters most on an untested platform.
set KEV_FUSED=0
set HF_HUB_OFFLINE=1
set PYTHONPATH=C:KevTestingwinshim
.venvScriptspython.exe scriptsserving_bench.py ^
--run jaredpalmer/kev-4b --reference fp32 --out resultskev-4bserving_bench
.venvScriptspython.exe -m kev.benchmark --run jaredpalmer/kev-4b ^
--suite evalsv4transfer-v4 --out resultskev-4btransfer-v4 --device cudaStill, one edit was needed. The serving script hard-codes fused=True, which cannot work without Triton, so a copy of it passes fused=False instead. That choice also fixes the comparison baseline: the reference runs in runs/serving-4b-* use the same unfused path, while runs/fused-4b-* does not.
In addition, quality ran on the two frozen suites behind the published headline numbers. transfer-v4 holds 656 questions from sources the model never saw during training. decision-v7 holds 1,468 questions from trained sources. Neither run rejected or truncated a single record.
Kev on AMD ROCm: speed results on the W7900
Overall, the W7900 lands in NVIDIA L4 territory. For the standard six-question request it needs 171.1 ms against 29.5 ms on an H100, so the gap is 5.8×. Meanwhile Kev-9B widens that gap to 7.2×.
| GPU | Kev-4B new / cached | Kev-9B new / cached | Load time 4B | VRAM 4B |
|---|---|---|---|---|
| H100 80GB | 29.5 / 20.2 ms | 36.6 / 24.5 ms | 5.7 s | 10.1 GB |
| L40S | 50.0 / 33.9 ms | 80.1 / 53.2 ms | 3.7 s | 10.1 GB |
| A100 80GB | 54.6 / 41.5 ms | 73.3 / 55.7 ms | 16.5 s | 10.1 GB |
| L4 | 156.6 / 115.9 ms | not measured | 6.3 s | 10.1 GB |
| Radeon PRO W7900 | 171.1 / 135.1 ms | 264.8 / 207.4 ms | 23.7 s | 14.5 GB |
Two AMD-specific observations stand out. First, memory sits 4.4 GB above every NVIDIA run of the same model, at 14.5 GB against 10.1 GB, which comes from HIP graph buffers. Second, loading weights takes 23.7 s against 5.7 s on an H100, and 64 s for the 9B checkpoint. So a cold start is noticeably slower, although it only happens once.
HIP graphs themselves worked without trouble. The 4B run captured 207 graphs and the 9B run captured 214, with zero failures in both cases.
Throughput saturates early. Kev-4B reached 1.70 requests per second with one client, 2.72 with two, and only 4.36 with eight. Meanwhile median latency climbed from 136 ms to 721 ms. In practice two concurrent clients is the sensible operating point.
Kev on AMD ROCm quality: the numbers do not move
This is the part that could have gone wrong, and it did not. Both checkpoints reproduced their published figures to three decimal places, including the calibration metrics that depend on exact probability values rather than just the winning answer.
| Metric | Kev-4B measured | Kev-4B published | Kev-9B measured | Kev-9B published | Jev |
|---|---|---|---|---|---|
| Accuracy, new sources | 0.817 | 0.817 | 0.822 | 0.822 | 0.857 |
| Accuracy, trained sources | 0.872 | 0.873 | 0.872 | 0.872 | 0.845 |
| Brier, out of domain | 0.243 | 0.243 | 0.264 | 0.264 | 0.211 |
| ECE, out of domain | 0.042 | 0.042 | 0.045 | 0.042 | 0.049 |
| Wrong at confidence 0.9+ | 0.9% | 0.9% | 4.1% | 4.0% | 3.7% |
| Automatable at 5% error | 0.620 | 0.620 | 0.447 | 0.45 | 0.70 |
The parity check carries even more weight. Serving in bf16 shifted probabilities by at most 0.0138 for Kev-4B and 0.0340 for Kev-9B, against the exact fp32 path. Crucially, zero answers flipped across 280 questions. By contrast, the reference H100 run recorded one flip on the same test.
So the practical conclusion is simple. Numerical differences between ROCm and CUDA exist, yet they stay far below the margin that would change a decision.
Kev-4B or Kev-9B on a workstation card
Choose Kev-4B. The bigger model gains half a point of out-of-domain accuracy and ties in domain, but it loses badly on the metric that decides real deployments.
At a 5% error budget, Kev-4B automates 62% of decisions. Kev-9B reaches only 45%. In addition Kev-9B produces confident mistakes four times as often, at 4.1% against 0.9%. It is also 1.5× slower and needs 7.5 GB more memory.
This is not a quirk of the AMD card, because the published model cards report the same 0.620 and 0.45. The larger checkpoint is simply calibrated worse, so its confidence scores carry less information.
Finally, measured thresholds make the point concrete. With Kev-4B, cutting off below 0.886 confidence automates half of all decisions at 97.6% accuracy. Lowering the cut to 0.627 automates 80% at 91.4%. Kev-9B delivers 94.5% and 89.0% at the same coverage levels.
What the state prefix cache saves
The server keeps the computed state prefix, so a repeated question about the same document pays only for the question rows. The saving scales with document length, and it becomes dramatic on long inputs.
Therefore the design advice is counter-intuitive. If you have many questions about one long document, send them as separate requests against the same state rather than bundling everything into one call. Kev-9B showed an identical pattern, saving 78% on the same shape.
Pitfalls and limits of Kev on AMD ROCm
torch.cuda.mem_get_info called from a separate process reports only that process’s own context on Windows. Use the GPU Process Memory(*)Dedicated Usage performance counter to measure another process.
However, the missing Triton is the real limitation. Without it the fused kernel path stays closed, and that path is worth a lot: on an H100 it takes Kev-4B from 29.5 ms down to 18.0 ms. Whether triton-windows now enables it on gfx1100 remains untested here, so treat that as the obvious next experiment rather than a solved problem.
Downloading the weights took longer than everything else combined. The HuggingFace client also stalled at zero bytes on large shards, while small files downloaded normally, so curl -L -C - straight into the cache directory replaced it. Splitting a file across parallel range requests did not help at all: four streams totalled 6.42 MB/s against 6.1 MB/s on one.
One more measurement trap deserves a mention. A pegged CPU core alongside a busy GPU looks alarming, yet it is normal, because the HIP runtime busy-waits on synchronisation instead of sleeping.
Where this setup makes sense
- Batch document triage — classifying a backlog of tickets, invoices or complaints overnight, where 171 ms per document is irrelevant.
- Privacy-bound decisions — routing or scoring records that cannot leave the building, replacing a hosted API call.
- Selective automation — auto-handling the confident 62% and queueing the rest for a human reviewer.
- Prototyping before renting a GPU — validating prompts, question schemas and thresholds locally, then deploying to an H100 unchanged.
- Not suited to — interactive online traffic above a couple of concurrent users.
Frequently asked questions
Does Kev run on an AMD GPU?
Yes. Kev-4B and Kev-9B both ran on a Radeon PRO W7900 through PyTorch 2.9.1+rocm7.2.1 on Windows 11, reporting the device as cuda. Quality matched the published numbers exactly.
How much VRAM does Kev need?
Kev-4B used 14.5 GB and Kev-9B used 22.0 GB on the W7900. Both figures sit about 4.5 GB above the equivalent NVIDIA runs, because HIP graph buffers cost extra.
Why does Kev fail with “No module named fla” on Windows?
Because the ROCm build of PyTorch for Windows ships no Triton, and flash-linear-attention needs it. Set KEV_FUSED=0 to decline the fused kernels; the model then runs on the standard path.
Is Kev on AMD ROCm accurate enough for a calibrated model?
Yes. Across 280 parity questions, bf16 on ROCm changed no answers against the exact fp32 path, and maximum probability drift stayed at 0.0138 for Kev-4B. The reference H100 run flipped one answer.
Should I use Kev-4B or Kev-9B?
Use Kev-4B. It matches Kev-9B on in-domain accuracy, runs 1.5× faster, and automates 62% of decisions at a 5% error budget against 45% for the larger model.
How fast is Kev on AMD ROCm compared to an H100?
About 5.8× slower for Kev-4B and 7.2× slower for Kev-9B, on a six-question request. That places the W7900 close to an NVIDIA L4.
Can I reuse my ComfyUI environment?
Copy it, never install into it. Kev needs transformers 5.17, which is newer than the version ComfyUI ships, so installing in place would break the image pipeline.
Summary
Running Kev on AMD ROCm works, and the quality is not a compromise. Kev-4B reproduced 0.817 out-of-domain accuracy, 0.243 Brier and 0.042 ECE, matching the published model card to three decimals, with zero answer flips against fp32 across 280 questions.
Speed is the price. A Radeon PRO W7900 needs 171 ms per six-question request against 29.5 ms on an H100, which puts it in NVIDIA L4 territory. Cold loading takes 23.7 s, and throughput flattens at roughly 4.4 requests per second.
Three setup obstacles stand between you and those numbers: a torch version pin, a POSIX-only fcntl import, and the absent Triton that closes the fused kernel path. All three have short workarounds, and none of them affect the results. Finally, remember the model choice: on this hardware the 4B checkpoint is the better tool, not the compromise.
Published numbers quoted here come from the Kev-4B model card. For the platform side, AMD documents the Windows PyTorch path in its own GPUOpen deployment guide.
Companion code — the fragments above are the working parts: the fcntl shim that lets kev.benchmark import on Windows, the constraints pin that stops pip swapping the ROCm build for a CPU wheel, and the one-line fused=False edit that makes the serving harness run without Triton. The commands in this article reproduce every number from scratch. The full scripts, the raw JSON results and a checksum manifest are in the download bundle (45 KB): kev-on-amd-rocm.zip.
Free ebook
Free AI Video, Generated Locally
Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.
No spam. Unsubscribe at any time.
Leave a Reply