Hands lets your LLM agent control your actual Windows desktop

Most people trying to build AI agents for web browsing get stuck in the "automation trap." You either use Playwright or Puppeteer, which triggers all those "automated software" flags, or you try to use a headless browser that can't handle the complex, messy reality of a real user session. I've been looking into a new tool called Hands, and it takes a completely different approach by using a Rust-based MCP (Model Context Protocol) and CLI to let agents interact with your OS exactly like a human would.

Instead of trying to hijack a browser via remote debugging ports, Hands uses the Windows SendInput API to move the mouse along Bézier paths and type into active windows. It essentially turns your actual, daily Chrome profile—the one with all your logins and cookies—into a controllable environment without needing any special launch flags.

How the architecture actually works

The setup relies on a few moving parts to make sure the LLM isn't just "guessing" where pixels are. It uses a combination of visual observation and structural data:

• The Observe Tool: This provides the agent with a screenshot path and a lightweight list of elements. It pulls from UIA (User Interface Automation) and, if you use the extension, the Chrome DOM.
• The Fusion Layer: This is the clever part. A tiny, unpacked Chrome extension maps out the page structure (like specific IDs or listing cards). This way, when the model wants to "click the third car in the list," it isn't just clicking coordinates; it's interacting with a known element.
• The Click Mechanism: Because it uses OS-level input, it avoids the LLMHF_INJECTED flags that many anti-bot systems look for.

Implementation and Deployment

If you want to try this as a hands-on guide for your own local AI workflow, you can't just npm install it. It requires a bit of manual setup because it's interacting with your hardware:

1. Build the executable from the Rust source.
2. Register the project as a native-messaging host on your Windows machine.
3. Sideload the provided Chrome extension manually.
4. Point your MCP client (like Claude Code or any other LLM agent harness) to the Hands MCP server.

If you run into issues, the logs are tucked away in %LOCALAPPDATA%\hands\logs\.

Real-world limitations to keep in mind

This isn't a "set it and forget it" sandbox. Since it's operating on your actual desktop, there are real risks. If you tell an agent to "find a cheap flight" and it accidentally clicks "Confirm Purchase" on a high-priced ticket, the tool won't stop it. The "confirm-before-money" check is just a best-effort classification within the binary; it’s not a hard safety guarantee.

Also, don't expect it to be a CAPTCHA-solving god. It can try a couple of visible interactions, but if it hits a puzzle, it will yield and wait for you to handle it manually.

For those interested in the technical specifics of the observe and fusion logic, the source is available here:

https://github.com/Ryan-AI-Studios/hands

It's a fascinating look at how we can bridge the gap between high-level LLM reasoning and low-level OS control without the overhead of traditional web automation frameworks.