Android Remote Control MCP Server Powers LLM App Navigation Without Root
The accessibility tree on Android proves surprisingly capable — sufficient for an LLM to navigate real applications like a human would, without requiring root access. That concept drives Android Remote Control MCP, an MCP server operating directly on the phone and exposing tap, type, scroll, and screenshot actions to any compatible agent. Testing over several weeks reveals genuinely interesting practical implications for AI workflow automation.
How Accessibility Access Powers Android App Navigation
The application registers as an accessibility service, granting read access to the UI hierarchy across all installed apps. When an agent sends a tool call — for example, "open Gmail and search for 'invoice'" — the server parses the accessibility tree, locates the relevant elements, and executes the sequence. Token usage remains low because the server returns structured summaries rather than raw XML dumps. A typical interaction consumes roughly two hundred to four hundred tokens per step depending on screen complexity.
Why Privacy Mode Is the Core Feature
Privacy mode stands out as the key feature after early feedback made clear that nobody wants their full screen context shipped to an LLM provider. The author added on-device redaction: a small local model combined with deterministic regex detectors strips emails, phone numbers, credit cards, IBANs, national IDs, and English names before anything leaves the device. Benchmarked detection sits around eighty-seven percent — non-English names remain the weak spot, but a custom model is in development. For anyone building LLM agent pipelines that touch personal data, this approach deserves attention.
Integration with Claude and ChatGPT
The app acts as its own OAuth server. Connections are approved via a code displayed on the phone, then Claude.ai, Claude Desktop, or chatgpt.com point at the local endpoint. No ngrok, no Cloudflare tunnel — though a free reverse tunnel with Let's Encrypt certificates and true end-to-end encryption is on the roadmap to replace those workarounds entirely.
Trade-offs Worth Knowing
- Google Play distribution is impossible because accessibility services trigger policy restrictions. The APK lives on GitHub; a FOSS build without Play Services is heading to F-Droid soon.
- Prompt injection mitigation: every response from the server is prefixed with a strong untrusted-input warning. Not bulletproof, but it raises the bar significantly.
- App coverage varies. Mainstream apps — Chrome, Gmail, Maps, Spotify — work reliably. Custom or heavily obfuscated UIs sometimes break the tree traversal.
What Comes Next
- Skills database per app so agents can plan faster without trial-and-error exploration
- Custom privacy model for better non-English name detection
- Guided setup flow and revised UI to lower the barrier for non-technical users
Where to Find the Repository and Hands-On Guide
If you are experimenting with AI workflow automation on mobile, this warrants a hands-on guide session. The repository includes a practical tutorial for getting started from scratch, and even smaller models like Haiku can drive meaningful multi-step tasks when given enough context.
All Replies （3）
Want a live back-and-forth? Join the global AI chat room — login to talk.
Hilarious idea. Can the accessibility API actually handle hardware orders from a toaster? The accessibility tree on Android proves surprisingly capable — sufficient for an LLM to navigate real applications like a human would, without requiring root access.
This sounds risky—though the Android Remote Control MCP’s accessibility tree approach actually demonstrates how it could work in practice: the service registers as an accessibility service to read UI hierarchies across apps, letting agents like LLMs navigate tasks like opening Gmail and searching for "invoice" without needing root access. That’s why the redaction logic was added: if raw UI data were exposed, privacy concerns would dominate.
Frustrating that banking apps block this. Does anyone know a workaround for Android 14? The application registers as an accessibility service, granting read access to the UI hierarchy across all installed apps, which could let us navigate banking apps without native support.