A computer action model that keeps up with the screen
Moenjo watches the screen as live video and works the mouse and keyboard from plain-English instructions. It answers a new frame in 14–26 milliseconds and runs on one graphics card, so the screen never leaves the PC.
Research in progress at LiveTawk. Not a product yet.
What it is
Today's computer-use agents mostly work from screenshots: take a picture, send it to a large model, wait, act, repeat. Each step takes seconds, and anything that moves on screen in the meantime is missed.
Moenjo is built the other way round. A small vision-action model looks at every frame and reacts to it directly, the way a person's eyes and hands do. A separate language model plans the steps of a task and hands them over one at a time. The first use we are aiming at is everyday office work on Windows: spreadsheets, documents, settings and the browser.
The project started in October 2026. One person builds it, on one PC.
How it works
Three parts run together on the same machine.
14–26 ms per frame
Reflex
Our own vision-action model, trained in PyTorch. It finds the element an instruction names, decides whether it is on screen at all, and clicks or types.
0.85 s per plan
Mind
A small open-weights language model that breaks a task into steps. It runs on the same GPU and is only called when a new step is needed.
Under 1 ms p95
Real-time core
Written in Rust. It captures the screen on the GPU, detects what changed, passes frames straight into GPU memory, sends input, and holds the safety checks.
Where the time goes
Measured on one NVIDIA GeForce RTX 5090. For comparison, a display at 60 Hz shows a new frame every 16.7 ms.
Results so far
These are the numbers as of November 2026, about six weeks in. Every test screen listed as held out was kept out of training.
| Test | Result |
|---|---|
| Whole tasks in a Windows 11 virtual machineA fixed suite of 102 tasks, run the same way after each change | 66 of 102 |
| Whole tasks on a real desktop28 tasks; up from 46% two days earlier | 75% |
| Finding the named element, held-out Office screens | about 92% |
| Finding the named element, held-out virtual machine screens | about 90% |
| Finding the named element, held-out real Windows screens | about 89% |
| Finding the named element, held-out web pages | about 84% |
| Microsoft 365 screens of a kind the model has not trained on | 62% |
Demo
Three minutes and 24 seconds of Moenjo working in Excel, Word, Settings and the browser. Each task is a single real-time take.
If the player does not load, open the video on Google Drive.
Training data
Everything the model trains on is cleared for commercial use. Nothing is scraped from video sites.
- Our own data farm. Six Windows 11 virtual machines open apps, read what is on screen and label it automatically, about 3,000 screens an hour. More than 225,000 labelled screens so far.
- Paid recordings. Contributors are paid to record themselves doing set tasks.
- Open datasets under the MIT licence. VideoCUA, AgentNet, GUI-360 and GroundCUA.
A new model replaces the current best only if it wins on fixed test sets that are never used for training. More than 40 rounds have gone through that gate.
What does not work yet
- 36 of the 102 suite tasks still fail.
- On Microsoft 365 screens of a kind it has not trained on, the model finds the right element 62% of the time.
- Small text and dense spreadsheets are often misread at the resolution the model sees today.
- Dragging, scrolling and anything that depends on timing are weak, because most training so far used single screens and not video.
- Progress has slowed. None of the last 40 training rounds beat the current best.
- It works on Windows only.
What comes next
All training so far ran on one graphics card. A round takes about five hours, so only one idea can be tested at a time. With more compute, the plan is:
- Several experiments at once, so a week of ideas can be tested in a day.
- A larger Reflex model that sees the screen at higher resolution, for small text and dense spreadsheets.
- Training on screen video, for dragging, scrolling and timing.
- Running the 102-task suite after every round, without pausing training to do it.
There is no product, price or customer yet. If the research holds up, the first version will be for office work on Windows.
Company and contact
Moenjo is the main work of LiveTawk, a one-person company in Islamabad. The name comes from Mohenjo-daro, the Indus Valley city laid out on a grid some 4,500 years ago.
- Legal name
- LIVETAWK (SMC-PRIVATE) LIMITED
- Registered in
- Pakistan, company no. 0356221
- Office
- Islamabad, Pakistan
- Founder
- Abou Hurerah
- admin@livetawk.com
- Company site
- livetawk.com
Latency reports and per-round test tables are available on request.