The best AI agent in the world was handed real expert work. Work that takes a professional hours, not minutes. It scored 12%.
In finance and legal, almost nothing.
And in security tests, when the agents were given rules to follow, they broke them 98 times out of 100.
That is not a typo. Told what not to do, they did it anyway, 98 times out of 100.
So what do you do with a number like that?
First, get precise about what that number means
This was not a test of everyday jobs. It was the hardest expert work out there: long tasks inside simulated workplaces, the kind of thing that takes a professional most of a day. Short jobs, clear questions, one step and one answer, agents handle those fine. It is the long, messy work that breaks them.
That distinction matters, because it tells you exactly where the danger is. Not in the small stuff. In the important stuff. The work you would never hand to a new hire on their first day, unsupervised.
And one more honest note. That 12% is a score, not a count of finished jobs. On a strict pass or fail basis, finance and legal came out at basically zero. The agents got partway through most tasks and then got stuck, or got something wrong, or broke a rule. That is the whole point. Getting 90% of the way there and still having nothing usable is exactly how real work fails.
Why does this happen?
Agents are very good at the kind of job you see in a demo. Short. Clear. One question, one answer. Real work is nothing like that. Real work is long and messy and full of judgment calls. It needs context nobody gave the agent. It needs to know when to stop and ask, and when a small mistake actually matters.
So the agent does its best, fills in the gaps, and reports back that everything went fine. It is not lying. It just never knew what good looked like.
The rule breaking works the same way. A rule sitting in a document somewhere is not a rule the agent feels. If nobody checks its work while it happens, the rule is just words on a page.
So what actually works?
Test the agent on your real work first. Not a demo, not a sample. Your actual work, with your actual mess. If it can only finish a small part, you have learned something cheap. Find out a month later and you have learned something expensive.
Give it clear permissions. A simple list of what it may touch and what it may not. An agent that only needs to read information should never be able to change it. An agent that touches money should never act without approval. Think of it like giving someone a key to one room, not the whole building.
Check its work while it happens. Rules written down somewhere are not enough. If an agent tries something outside its permissions, it should be stopped right there, not found in a report a week later.
Watch the whole chain, not just each agent on its own. The danger lives in the handovers, in what changes as work passes from one agent to the next. Keep a record of every step, so when something goes wrong you can trace it back to the exact moment.
And keep a human on the big decisions. Money. Promises to customers. Anything regulated. Those pause for a person. Everything else needs a stop button, a way to halt the whole chain the second it starts going wrong.
None of this is complicated. It is just the stuff every good team already does. It was never done for agents, because they never complained.
What this means for you
Before you add another agent, ask one question. Do you know what your agents can do when nobody is watching?
Most businesses cannot answer that. Not because they are careless. Because nobody ever tested the agents on real work, set the rules, and put the watching in place. Agents arrived one at a time, each one impressive in a demo, and the gaps between them never got closed.
That is the work we do. We start by testing what your agents can actually do today, on your real work. Then we put the guardrails around them. Permissions, checks, watching, records, human checkpoints. You keep the speed. You lose the hidden risk.
Agents do the repetitive work. Humans make the important decisions. And you always know what is happening in between.
Book a 30 minute call. We will test what your agents can really do and show you where the gaps are.
