Vision in practice: agentic screenshot analysis with Playwright and VLM models

Part of AI Tools & Automation — automating real work with local AI models.

Pairing a vision-language model (VLM) with Playwright for screenshot analysis sounds like science fiction, yet it runs entirely on local hardware. Playwright captures a screenshot of any page. A VLM then reads that image and answers questions about it. No screen-scraping selectors, no cloud API, and no data leaving your machine.

This article builds that pipeline step by step. It then measures how well a single local model — Qwen3.6-35B-A3B with its vision projector — actually reads real interfaces: tables, forms, and error screens. Every number below comes from a script you can rerun.

Why analyse screenshots with a VLM at all?

Traditional automation reads the DOM. That breaks the moment content lives inside a canvas, a chart image, or a PDF render. A VLM sidesteps the problem, because it reads the rendered pixels the same way a person does.

Privacy is the second reason. Screenshots of internal dashboards often contain sensitive data. A local model keeps every pixel on your own GPU, so nothing reaches a third party. Finally, the cost is fixed. You pay once for the hardware, and every screenshot after that is free to analyse.

VLM vs OCR vs DOM scraping

Three tools can extract information from a page, and each fails differently. Classic OCR reads text but ignores layout and meaning. DOM scraping reads structure but breaks on canvas, charts, and dynamic rendering. A VLM understands both the text and its visual context, so it answers questions rather than just dumping content.

In other words, a VLM shines when the question is interpretive: “is this dashboard healthy?” or “what error is shown?” For that class of task, neither OCR nor scraping comes close.

How the model turns pixels into tokens

A VLM does not see pixels directly. The vision projector splits the image into patches and encodes each patch into tokens the language model can read. As a result, a larger image produces more tokens, which raises both latency and memory use. That single fact explains why resolution tuning matters so much, and why cropping beats shrinking when text is small.

Pipeline architecture

The pipeline has four stages, and each one is small:

1. Playwright opens the page in headless Chromium and captures a PNG screenshot.
2. Preprocessing resizes the image to a VLM-friendly width, so tokens are not wasted on huge resolutions.
3. Local VLM receives the image plus a question through the OpenAI-compatible /v1/chat/completions endpoint.
4. Post-processing extracts the answer and stores it as JSON for later use.

Because the model server speaks the OpenAI API, the same code works whether you serve the model through llama.cpp or another local runtime. You only change the base URL.

Free ebook

Free AI Video, Generated Locally

Working scripts and measured benchmarks. Free.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.

Installation and setup

pip install playwright openai pillow
playwright install chromium

# Serve the multimodal model locally (llama.cpp example, vision projector attached)
llama-server --model Qwen3.6-35B-A3B-UD-Q4_K_S.gguf 
             --mmproj mmproj-F32.gguf 
             -ngl 99 --port 8082 --host 127.0.0.1

The --mmproj flag is the key detail. It loads the vision projector that turns image patches into tokens the language model understands. Without it, the same weights run as a text-only model.

Implementation — the core function

The whole pipeline fits in one function. Playwright grabs the screenshot, and the OpenAI client sends it to the local server. One detail matters for reasoning models like Qwen3.6: disable thinking mode for short extraction tasks, otherwise the model spends its token budget on internal reasoning instead of the answer.

import base64
from playwright.sync_api import sync_playwright
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8082/v1", api_key="not-needed")

def screenshot_and_ask(url: str, question: str) -> str:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(viewport={"width": 900, "height": 620})
        page.goto(url, wait_until="networkidle")
        png = page.screenshot()
        browser.close()

    b64 = base64.b64encode(png).decode()
    resp = client.chat.completions.create(
        model="local-model",
        messages=[{"role": "user", "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
            {"type": "text",      "text": question},
        ]}],
        max_tokens=200,
        temperature=0.0,
        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    )
    return resp.choices[0].message.content

Tuning the screenshot resolution

Resolution is the single biggest lever on speed and accuracy. Too small, and text becomes unreadable. Too large, and the image eats thousands of tokens for no benefit. A width around 900–1120 pixels hits the sweet spot for most interfaces.

For dense dashboards, a different tactic works better. Instead of shrinking the whole page, crop to the region you care about and keep it at full resolution. The model then sees sharp text where it matters, and the token count stays low. Playwright supports element-level screenshots, so you can target a single card or table directly.

Running it as a batch pipeline

One screenshot is a demo. The real value shows up when you process a list of pages unattended. The loop below reads a set of URL-and-question pairs, calls the model, and collects the answers. Because the model stays warm between calls, throughput climbs quickly after the first request.

import json

TASKS = [
    {"url": "http://localhost:3000/dashboard", "q": "What is the current error rate shown?"},
    {"url": "http://localhost:3000/billing",   "q": "List the plan names and their prices."},
]

results = []
for task in TASKS:
    answer = screenshot_and_ask(task["url"], task["q"])
    results.append({"url": task["url"], "question": task["q"], "answer": answer})

json.dump(results, open("audit.json", "w"), ensure_ascii=False, indent=2)

From here, you can schedule the script to run every night. The JSON output feeds a diff tool, a dashboard, or a ticketing system. As a result, a small local model quietly watches your interfaces while you sleep.

Benchmark methodology

Vague claims help nobody, so the benchmark uses fixtures with a known answer. Three small HTML pages stand in for common real-world screens: a sales table with numbers, a login form, and a Python stack trace. Each page has an exact ground truth, which means correctness is objective rather than a matter of opinion.

The script asks two questions per page, across three categories:

• Tables & numbers: read a specific total, and name the top region.
• UI elements: read the button label, and list the input fields.
• Error / OCR: name the exception type, and read the line number.

Each question runs 10 times. The script records processing time and marks an answer correct only when it contains the exact expected fact. That gives an accuracy percentage and a mean latency per category.

Accuracy results

The result is clear. On clean, well-contrasted interfaces, the model reads structured content flawlessly. It pulled the number 410 from a table cell, named West as the top region, and read JSONDecodeError straight from a dark-themed stack trace. For UI automation and bug triage, that reliability is exactly what you need.

Processing time

The first request on each screenshot pays a warm-up cost of roughly 3–4 seconds, because the server encodes the image for the first time. Every repeat query on the same image then returns in about 0.2–0.3 seconds. Therefore, in a batch pipeline, the per-image cost drops fast once the model is warm. Add one to two seconds per URL for the Playwright screenshot itself.

Practical use cases

The pattern is always the same: capture a screenshot, ask a targeted question, and act on the answer. That simple loop covers a surprising range of everyday engineering tasks.

• UI monitoring: capture a dashboard each morning, then ask the model whether a key metric changed or an element is missing.
• Bug triage: feed an error screenshot to the model, and let it read the exception and line number straight into the ticket.
• Accessibility checks: ask the model whether text is readable and whether buttons carry clear labels.
• Living documentation: screenshot each step of a workflow, then generate captions automatically.

Where the approach struggles

These fixtures are clean and high-contrast, so 100% accuracy reflects an easy, controlled case rather than the messy web. Real pages are harder. Tiny fonts, low-contrast themes, dense dashboards, and overlapping elements all reduce accuracy. In those cases, capture a higher-resolution screenshot and crop to the region of interest.

One model is also not a leaderboard. This benchmark measures a single capable VLM, not a comparison across models. Results will differ on smaller vision models, which often trade accuracy for speed and lower VRAM. Treat these numbers as a solid baseline, not a universal verdict.

Frequently asked questions

Do I need a special vision model for this?

You need a multimodal model plus its vision projector. With llama.cpp, that means passing --mmproj alongside the model weights. Without the projector, the server runs text only and rejects images.

Why disable thinking mode?

Reasoning models spend tokens on internal thought before answering. For a short extraction like “read this number,” that overhead wastes time and can exhaust the token budget. Disabling it returns a direct answer in a fraction of the time.

Can it read small or low-contrast text?

Up to a point. Legible, well-contrasted text reads reliably. For small fonts, increase the screenshot resolution and crop to the relevant area, so the important pixels fill more of the frame.

How many screenshots can it process per hour?

After warm-up, short queries run in well under a second each. With screenshot capture included, a single-threaded pipeline comfortably handles hundreds of images per hour on one W7900.

Does it work on pages behind a login?

Yes. Playwright drives a real browser, so you can log in, set cookies, or reuse a saved session before the screenshot. The model only ever sees the rendered image, never your credentials.

Can I get structured JSON instead of prose?

Ask for it directly. Phrase the question as “return a JSON object with fields x and y,” and lower the temperature to zero. The model then returns parseable output that slots straight into a pipeline, which removes brittle text parsing.

Summary

A local VLM plus Playwright is a practical, private way to turn screenshots into answers. On controlled UI fixtures, Qwen3.6-35B-A3B read tables, forms, and stack traces with 100% accuracy and sub-second latency after warm-up. Real-world pages will be harder, yet the pipeline holds up well for monitoring, bug triage, and accessibility work. Reproduce every number with the script: 05_vlm_ui_benchmark.py.

Companion code — every script and benchmark from this series lives in one repository: github.com/bestin-it/qwen-local-ai-scripts.

Free ebook

Free AI Video, Generated Locally

Run Wan 2.1 in ComfyUI on your own GPU — the scripts I use, measured times, sample clips. No cloud, no API keys.

Your e-mail address

Leave this field empty

Get the ebook

No spam. Unsubscribe at any time.