Dashboards, reports and bug triage: how well a local vision model reads what the team sees

Part of AI Tools & Automation — automating real work with local AI models.

The most useful screens a team looks at every day are not photographs. They are dashboards, reports and error consoles — KPI cards, bar charts, expense tables, and red stack traces. This article measures how well a local vision-language model reads exactly that kind of content, and how you can wire it into a real bug-triage workflow.

Instead of a vague “it works,” the test uses fixtures with a known answer and scores the model over ten runs per question. The result is a scorecard you can trust and reproduce.

Why reading a dashboard is hard for machines

A dashboard encodes meaning in layout, not just text. A number is a KPI because of the card around it. A bar is “the peak” because it stands taller than its neighbours. Traditional tools miss this context. OCR returns a bag of words with no structure, and DOM scraping fails the moment a chart is drawn on a canvas.

A VLM reads the rendered image as a whole. It sees the KPI card, the tallest bar, and the red variance cell together. Therefore it can answer interpretive questions that neither OCR nor scraping can handle, such as “which category is over budget?”

What OCR sees versus what a VLM sees

Consider the expense report in this test. Classic OCR would return a flat stream: “Cloud 5000 6200 +1200 Salaries 40000 39500 -500.” The numbers are there, but the relationships are gone. Which figure is a budget, and which is an overrun? OCR cannot say.

The VLM keeps the grid intact. It knows 6200 sits in the Actual column for Cloud, and it reads the red +1200 as a variance. Ask it “which category is over budget,” and it answers “Cloud” directly. That leap from text to meaning is the whole point of using a vision model here.

How the test is designed

The benchmark renders three local fixtures with Playwright. Each stands in for a common team screen, and each has an exact ground truth so scoring stays objective.

• Dashboard: three KPI cards (revenue, active users, churn) plus a signups-per-day bar chart.
• Report: an expense table with budget, actual, variance, and a totals row.
• Error console: a browser console showing an HTTP 500 and a JavaScript exception.

From these, the script builds four task categories. Each category has two questions, and each question runs 10 times. An answer counts as correct only when it contains the exact expected fact.

The scorecard — results per category

The model handled every category cleanly. It read $1.24M from a KPI card, identified Wednesday as the signup peak, summed the report to 53,700, and pulled both 500 and TypeError from a dark console. For everyday BI screens, that is exactly the reliability a triage or monitoring pipeline needs.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

Processing time and OCR behaviour

The bug-triage mean sits higher for one reason: the first call on a new screenshot pays a warm-up cost of a few seconds, and the “explain the error” question produces a longer answer. Every warm, short query then returns in about half a second. So in a batch, per-screen cost collapses once the model is loaded.

Practical bug-triage workflow

The scorecard becomes useful when you attach it to a real workflow. The pattern is simple: capture the error screenshot, ask for a structured triage, and write the result straight into a ticket. Asking for JSON keeps the output machine-readable.

import base64, json
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8082/v1", api_key="not-needed")

def triage(screenshot_png: bytes) -> dict:
    b64 = base64.b64encode(screenshot_png).decode()
    prompt = ("You are a bug triage assistant. Return a JSON object with keys: "
              "error_type, http_status, likely_file, severity (low/medium/high), summary.")
    resp = client.chat.completions.create(
        model="local-model",
        messages=[{"role": "user", "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
            {"type": "text",      "text": prompt},
        ]}],
        max_tokens=300, temperature=0.0,
        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    )
    return json.loads(resp.choices[0].message.content)

Fed the console fixture, the model returns a clean object. A typical response looks like this:

{
  "error_type": "TypeError",
  "http_status": 500,
  "likely_file": "checkout.js:88",
  "severity": "high",
  "summary": "Checkout POST to /v1/orders failed with 500; a TypeError reads 'id' of undefined."
}

That JSON drops straight into Jira or GitHub Issues, which removes the manual step of reading and re-typing an error. The model even infers severity from the context, because a failed checkout request is clearly high impact.

Prompting the model for reliable reads

Three habits keep the answers stable. First, ask for one fact at a time when accuracy is critical, rather than a paragraph that mixes several values. Second, name the exact format you want — “answer with only the number” removes hedging. Third, set temperature to zero so the same screenshot always yields the same answer.

These small choices matter more than model size. A precise prompt turns a chatty model into a dependable extractor, and it is the difference between a demo and a pipeline you can schedule.

Scaling to continuous monitoring

Once triage works for one screenshot, scheduling it is trivial. A cron job or task scheduler captures your dashboards every few minutes, runs each through the model, and stores the JSON. A simple diff then flags when a KPI crosses a threshold or an error appears.

Because the model runs locally, this loop costs nothing per run beyond electricity. You can watch dozens of screens around the clock without a single cloud API call, and without exposing internal metrics to a third party.

Cost and privacy for teams

For a team, the economics are stark. A cloud VLM bills per image and per token, so a monitoring loop that runs every few minutes adds up fast. A local model turns that recurring bill into a one-time hardware cost. After the GPU is paid for, ten thousand screenshots cost the same as ten.

Privacy tips the balance further. Dashboards expose revenue, churn, user counts, and internal errors — exactly the data a company least wants to send outside. Keeping the model on your own hardware means those pixels never leave the building. For regulated industries, that is often not a preference but a requirement.

When a VLM replaces manual triage

This approach fits repetitive, visual triage well. Screenshots arrive from QA, from monitoring, or from users, and the model reads them faster than a person can open the image. It never gets bored, and it applies the same rubric every time.

It does not replace judgement on ambiguous incidents. A VLM reads what is on screen, yet it cannot know your system’s history or business impact. Use it to draft the ticket and classify the obvious cases, then let an engineer confirm severity on the hard ones.

Where accuracy drops

These fixtures are clean and high-contrast, so 100% reflects a favourable case. Real dashboards are noisier. Tiny sparklines, dense grids, low-contrast dark themes, and overlapping tooltips all reduce accuracy. When precision matters, crop to the widget in question and raise the screenshot resolution, so the important pixels dominate the frame.

Remember too that this is one model, not a comparison. A smaller vision model will trade accuracy for speed and lower VRAM. These numbers set a strong baseline for a capable local VLM, not a universal ranking.

Frequently asked questions

Can a VLM read numbers inside a chart image?

Yes, when the chart shows value labels or clearly comparable bars. It read the Wednesday peak and its value of 150 every time. For unlabelled charts, it estimates from bar heights, which is less exact.

Is it reliable enough for automated alerts?

For clear, well-formatted screens, the 100% score suggests yes. Still, add a confidence check: ask the model to flag when it is unsure, and route those cases to a human rather than firing an alert blindly.

How do I get consistent, parseable output?

Ask for JSON with named fields and set temperature to zero. That combination produces stable, structured answers that a script can consume without fragile text parsing.

Does it work on dark-mode consoles?

Yes. The error fixture uses a dark console, and the model read the red HTTP 500 and the TypeError without trouble. Contrast matters more than colour scheme.

Which VRAM do I need for a model like this?

A capable multimodal model in a 4-bit quantization fits on a 24 GB card, and a 48 GB card leaves ample room for a long context. Smaller vision models drop to 6–8 GB, which suits laptops and entry GPUs at some cost to accuracy.

Can it compare two screenshots, like before and after?

Send both images in the same message and ask what changed. The model then describes the differences it sees. For pixel-exact diffs, pair it with a traditional image-diff tool and let the VLM explain the flagged regions.

Summary

A local vision model reads team dashboards, reports, and error consoles with high reliability. On controlled BI fixtures, Qwen3.6-35B-A3B scored 100% across KPIs, charts, tables, and bug triage, with sub-second latency after warm-up. Wired into a triage workflow, it converts a raw error screenshot into a structured ticket in seconds. Reproduce the scorecard with the script: 06_vision_scorecard_bench.py.

Companion code — every script and benchmark from this series lives in one repository: github.com/bestin-it/qwen-local-ai-scripts.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.