How we accidentally built a browser agent
We spent the last few years building AI agents for customer support.
The first version was basically agentic search over support context: docs, tickets, prior resolutions, internal notes.
That gets you surprisingly far, but support work quickly runs into product state. The answer often depends on the customer's account: configuration, permissions, billing state, or some admin setting buried three screens deep.
So the agent has to inspect the product, and sometimes change it.
We didn't start with "let's build a browser agent."
We started with "the agent needs to be connected to the API."
Why Claude uses the browser like a drunk intern
You've probably used at least one browser agent by now, whether it's Comet, Claude, ChatGPT Atlas, Browser Use, or anything else.
The loop is always the same:
Explore the page. Maybe even take a screenshot. Find the likely target. Move the mouse. Click. Wait. Screenshot again. Scroll. Try a dropdown. Recover from a modal. Guess whether the state changed.
I get it - this is the information we have, and the functionality that's exposed. But it's horribly slow, and highly inefficient.
It's like asking a voice agent to book a barber appointment by calling another voice agent. Technically possible. Maybe even magical the first time. But absurd once you realize both systems could have exchanged structured data directly.
The UI is optimized for humans: layout, labels, progressive disclosure, hover states, responsive breakpoints, visual hierarchy. None of that is the product's actual execution model.
Underneath, the frontend is already speaking to a structured system. A click becomes a request, mutation, RPC call, websocket message, server action, signed request, persisted GraphQL operation, or some product-specific command. There are inputs, outputs, errors, permissions, side effects, and state transitions.
That's the interface an agent should want.
The obvious solution
Agents should use the structured interface underneath the UI, not pixels.
One operation can replace a dozen UI steps. A small script can replace a fragile click path.
Instead of asking a model to infer intent from screenshots and layout, give it objects, fields, methods, responses, errors, and invariants.
That's a much better fit for current models. They're already extremely good at code, structured data, and tool calls. They're much less reliable when the task is "interpret this screenshot, click the right thing, and hope the UI didn't shift."
Pixels are a lossy transport for behavior that already exists in code.
So the question seems easy: why not use that interface directly?
The not-so-obvious solution
The hard part isn't that the interface doesn't exist.
The UI uses it constantly.
The hard part is that it is not structurally exposed or documented.
It's usually not public. It might not even be stable. It's often split across REST endpoints, GraphQL operations, websocket messages, signed requests, WASM modules, web workers, client caches, server actions, and frontend-only conventions. Some of the important behavior may live in request sequencing, id resolution, validation responses, cache invalidation, or permission-dependent branches.
"Just use the API" skips the part where the agent has to learn the actual contract the UI is relying on.
The agent needs to know:
- what objects exist
- which operations create or mutate them
- how authentication works
- how server-side language maps to the user-facing UI language
- heck, sometimes even that status ids depend on the user's language (*cough* Zendesk)
That information is often present only implicitly: in frontend code, network traffic, runtime state, cache updates, and error handling.
So the browser still matters.
Not as the runtime interface.
As the discovery interface.
When the frontend performs an action, it reveals the product contract. It fetches objects. It sends operations. It validates inputs. It receives errors. It updates state.
An agent can check the source code, observe what happens, and extract the shape of the workflow:
- routes, operations, or message patterns
- request and response shapes
- required fields and object relationships
- validation, authorization, and success states
So how can we make an agent learn from that?
By building skills.
A simple loop already works quite well: give an agent its own sessions, then tell it to update its skills based on what worked, where it struggled, and what it newly discovered.
Since this learning happens across users, we need to be extremely careful about sensitive data. Luckily, we have spent the last three years figuring out how to make agents learn securely from past support tickets without compromising user data. The same principles apply here. Using that approach, we have already accumulated product-specific skill collections for more than 4,300 web apps now.
Those skills include API paths, code snippets, workflow logic, domain knowledge, useful edge cases, and notes about how the product actually behaves. There's a lot to improve about the extraction process, but even the simplest approach, telling the agent to generate helpful skills from its sessions, gets you surprisingly far.
Once you have the skills, it's almost like having a perfect MCP server.
The agent can talk to the API directly from code. It can write scripts that execute multi-step workflows. And it understands product-specific things that you'd never know from abstract tool descriptions.
Once those skills exist across products, workflows also stop being tied to one tab: export from one tool, transform the data, update another tool, and verify the result.
Imagine Claude using VS Code via computer use, instead of Claude Code working on the files directly.
That's the speed difference between Pluno and other browser agents.
A different benchmark for browser agents
We've created a benchmark, and it's bullshit.
But I don't know how else to show the effect of learning the product.
We compared Pluno against other browser agents across 312 real-world tasks in 24 tools, including HubSpot, Notion, Stripe, Linear, and more.
The comparison is unfair.
Pluno had learned those products before. It knew their APIs, workflows, object models, permissions, and weird edge cases. The other agents mostly approached them like fresh websites: inspect the page, infer the next click, try something, recover if it fails.
That isn't an academically clean comparison.
But it is the product experience we're trying to build. Users don't want an agent that rediscovers Stripe, Notion, Linear, Salesforce, or HubSpot every morning. They want an agent that already knows the software underneath and is 10x faster because of it.
We expected the main gap to be speed. Direct execution avoids screenshots, cursor movement, retries, and waiting for pages to settle.
But there was also a big difference in reliability. Many browser agents did not just run slowly. They got stuck in the UI, clicked into dead ends, lost state, or gave up. The interface was not only the bottleneck for speed. It was the bottleneck for success.
So read the numbers with that caveat. They don't prove that Pluno is better at using a previously unseen product. They show what happens when the agent has learned the product, then uses that knowledge on the next task.
We'll soon publish a full writeup with the task set.
Where pixels still matter
There are still cases where UI use is the right tool.
First-time exploration often needs the browser. Visual products need visual context. Canvas tools, design tools, dashboards, maps, and ambiguous workflows may require looking at the screen. Some products may not expose a usable backend contract. Some tasks need visual confirmation.
The point isn't that agents should never use UIs.
The point is that the UI shouldn't be the default execution layer for repeated SaaS operations.
Pixels should be the bootstrap layer, not the runtime.
The interface is the bottleneck
Browser agents aren't mainly slow because models are bad at reasoning.
They're slow because they're using the wrong interface.
We take models that are increasingly competent with code and structured data, then force them through screenshots, buttons, dropdowns, layout shifts, and modal state.
That's backwards.
Software already has structured behavior underneath the UI. The hard part is exposing that behavior to the agent safely: learning the contract, redacting traces, validating schemas, bounding permissions, requiring confirmation for side effects, and falling back to visual inspection when needed.
The future isn't agents that get better at clicking buttons.
It's agents that learn the contracts underneath the UI, then use them directly.
If you'd like to try it, here's the browser extension.
