Watch, Try, Compare, Repeat: How Our Agents Learn Your Workflows

Opnova has been granted US Patent 12,717,599 for teaching computer-use AI agents from video demonstrations. Record a task once. The agent writes its own instructions, runs them, checks its work against yours, and corrects itself until the two match.

“The person who knows exactly how a task gets done is almost never the person who can write the automation for it. That gap has held back enterprise automation for twenty years. This patent closes it: you show the agent once, and it has to match you before it is allowed to run.”

 

 

Every IT security team has that one person. She knows that new contractors go in a different organizational unit, that the password reset dialog needs the "must change at next logon" box ticked, that the search in the admin console fails silently if you don't close the previous window first. None of it is in a runbook. All of it is in her hands.

That is the knowledge enterprise automation has never captured, and it is why so much of it still fails. RPA needed a developer to turn her clicks into scripts, and the scripts broke the first time a button moved. Computer-use agents replaced scripts with natural language, which was progress. But someone still has to write a precise, unambiguous prompt for a sixteen-step workflow across two applications and a dialog that only appears half the time. Translation: prompt engineering became the new scripting. The bottleneck moved. It did not disappear.

Last month the USPTO granted us a patent for a different approach. Here is what it does, and why it is not the thing your RPA vendor already sold you.

The recording is a lesson, not a script

The starting point is a task recording: the screen, every keyboard and mouse action, and optionally what she said out loud while doing it. Most of a recording is noise. What matters are the moments that changed something: the click on "Find", the Enter key, the name typed into the search box. For each of those interactions, the system keeps the screen just before and just after. A five-minute video collapses into a short sequence of before, action, after, and from that sequence the system writes a step-by-step task prompt in natural language. The values she typed become input variables. A first name becomes a first name field. A password is marked sensitive the moment it is captured.

Narration is where the implicit knowledge comes in. If she says "contractors always go under this OU, employees under that one" while she does it, that rule becomes a condition in the prompt. That is the part of the work that never made it into documentation, and it is the part that decides whether an automation holds up in month six.

A prompt is a hypothesis until the agent proves it

A generated prompt is a guess about what the person meant. The patent is about what happens next.

The agent executes the prompt in a controlled environment, and the system compares what it did to the recording, action by action, in order. Where the two diverge, the feedback is specific. Step four was a right-click on an empty area of the details pane, when the demonstration right-clicked the organizational unit in the navigation tree, because the prompt said "navigate to the OU" without saying where the context menu has to come from. The prompt is rewritten to remove the ambiguity. The agent runs again. The loop continues until there is no discrepancy left between what the agent does and what the human did.

We stopped treating a prompt as something a person writes and an agent follows. In our system, a prompt is something the agent earns. It has to reproduce the demonstration, step for step, before it is allowed to become a workflow. Natural language is flexible, and flexibility is non-determinism. Ground truth squeezes that out before anything touches production.

"Our RPA tool has a recorder too"

I hear this on almost every call with a large US bank, along with "we ran a process mining project and it went nowhere." Both are fair. Both are also fundamentally different from what I have just described, and the difference is the whole point.

An RPA recorder captures where you clicked and gives you back a script that replays it. The recording is the automation. It has no idea what the task was, only where the mouse went, so it breaks when a window opens in a different position or a vendor moves a button, and fixing it means a developer opens the script. Process mining goes the other direction: it reads logs to discover what your processes actually are. The output is a map, and a useful one, but a map of the process is not an automation of it. The build is still waiting for the same developers.

What our patent covers is neither. The recording is a teaching example. The agent watches a person do the task, works out what each step was for, writes that down in natural language, and proves it understood by reproducing the task. At runtime it is reading the screen and reasoning about what to do next, the way the person did. "Right-click the target organizational unit in the left navigation tree" still works after the UI update. A pair of coordinates does not.

Same starting point as the recorder your RPA vendor sold you. One replays. One learns.

Nothing about the recording leaves the building

Most of our 20,000+ production executions are in banking, and that shaped the design. Recordings are made on the same machine the agent will later run on, inside the customer's environment. The method is covered for private cloud and on-premises operation. 

Human intervention is part of the system, not an exception to it. When an agent hits something it has never seen, long after the workflow was certified, it can ask for help. A person takes over, resolves the step, and that intervention is itself a recording. The system uses it to propose an update to the prompt, so the edge case that appeared once in production is handled the next time without anyone rewriting the workflow. It also leaves an audit trail of who did what, when, and why the workflow changed, which is what a change advisory board wants to see.

Nothing about the recording needs to leave the network boundary to become a workflow, and the person doing the teaching does not need to be an engineer. The access administrator who has run offboarding for eight years is the right person to teach an agent offboarding.

The operator demonstrates. The agent learns. Nobody opens a script.

Patents do not ship product. This is a capability our Field AI Engineers use every week to onboard new workflows in hours rather than the months a scripting project used to take.

But day one was never the hard part of IAM automation. Month six is. The vendor moves a button, an MFA prompt appears, an acquisition adds an organizational unit with its own naming rules. What decides whether automation survives is who fixes it and how.

The current answer, increasingly, is to have an LLM write the connector. It works on Tuesday. Then the application changes and you own code nobody on your team has read, written against an interface that no longer exists. Opnova CTO José Luís Pereira wrote about this last month: generated code breaks exactly the way handwritten code breaks. Vibe-coded connectors are the connector treadmill with a faster on-ramp.

Computer-use agents taught by demonstration change who does the maintenance. The administrator who noticed the new dialog is the person who fixes it. She records the step once. The agent updates its prompt, proves it matches, and the change is logged with her name and the reason attached. No ticket, no sprint, no developer reading a script he did not write. That is what self-service means here. Not a low-code studio your administrators will never learn. Their own hands, doing the job the way they already do it, once.

Before your next automation project, ask one question: when the application changes, who fixes the automation? If the answer is anyone other than the person who runs the workflow today, you have bought a connector with a longer name.

This is our first granted patent. It will not be the last.