NON906/omniparser-autogui-mcp

CategoryGeneral
AuthorCommunity
Stars23
PricingFree

Overview

The omniparser-autogui-mcp server bridges the gap between LLMs and local desktop environments by enabling direct GUI interaction. Unlike standard API-driven tools, this MCP implementation leverages OmniParser to visually analyze screen coordinates and execute precise mouse and keyboard events. For developers, this means you can build agents capable of navigating legacy software, complex web apps, or any desktop application without requiring a native API or accessibility tree. It transforms the LLM from a text generator into an operator that can perceive UI elements and perform actions based on visual feedback, making it ideal for automated testing, RPA, or creating personalized desktop assistants.

Highlights

  • Visual screen parsing for precise element localization
  • Direct mouse and keyboard control via PyAutoGUI
  • Works with any application regardless of API availability
  • Enables autonomous desktop navigation for LLM agents
  • Easy integration into MCP-compliant AI clients

Full Documentation

omniparser-autogui-mcp

（日本語版はこちら）

This is an MCP server that analyzes the screen with OmniParser and automatically operates the GUI.
Confirmed on Windows.

License notes

This is MIT license, but Excluding submodules and sub packages.
OmniParser's repository is CC-BY-4.0.
Each OmniParser model has a different license (reference).

Installation

1. Please do the following:

code
git clone --recursive https://github.com/NON906/omniparser-autogui-mcp.git
cd omniparser-autogui-mcp
uv sync
set OCR_LANG=en
uv run download_models.py

(Other than Windows, use `export instead of set.)
(If you want langchain_example.py to work, uv sync --extra langchain instead.)

2. Add this to your claude_desktop_config.json:

`claude_desktop_config.json
{
"mcpServers": {
"omniparser_autogui_mcp": {
"command": "uv",
"args": [
"--directory",
"D:\\CLONED_PATH\\omniparser-autogui-mcp",
"run",
"omniparser-autogui-mcp"
],
"env": {
"PYTHONIOENCODING": "utf-8",
"OCR_LANG": "en"
}
}
}
}
`

(Replace D:\\CLONED_PATH\\omniparser-autogui-mcp with the directory you cloned.)

env allows for the following additional configurations:

  • OMNI_PARSER_BACKEND_LOAD
If it does not work with other clients (such as LibreChat), specify 1.
  • TARGET_WINDOW_NAME
If you want to specify the window to operate, please specify the window name. If not specified, operates on the entire screen.
  • OMNI_PARSER_SERVER
If you want OmniParser processing to be done on another device, specify the server's address and port, such as 127.0.0.1:8000. The server can be started with uv run omniparserserver.
  • SSE_HOST, SSE_PORT
If specified, communication will be done via SSE instead of stdio.
  • SOM_MODEL_PATH, CAPTION_MODEL_NAME, CAPTION_MODEL_PATH, OMNI_PARSER_DEVICE, BOX_TRESHOLD`
These are for OmniParser configuration. Usually, they are not necessary.

Usage Examples

  • Search for "MCP server" in the on-screen browser.

etc.

View official source
Join our Telegram