Coasty: Automating Legacy Software via API

Legacy desktop apps without APIs are the final boss of DevOps automation. Most teams just throw RPA at the problem, but as anyone who has maintained a Selenium script knows, the moment a UI element shifts by two pixels or a random pop-up appears, the whole pipeline crashes.

I've been looking into Coasty as a way to bridge this gap. Instead of rigid coordinate-based clicking, it uses a vision-based LLM agent to observe the screen and react in real-time. From a workflow perspective, this is a massive shift from "record and replay" to "observe and act."

Here is how the deployment looks from a developer's angle:

1. Task Initiation: You send a natural language instruction and a target environment (VM or browser).
2. Execution: The agent operates the mouse and keyboard based on screenshots, meaning it doesn't need DOM access or accessibility trees.
3. Verification: It checks the resulting state after every action to ensure it hasn't drifted into an error state.
4. Human-in-the-loop: You can set specific checkpoints where the agent must pause for manual approval before proceeding.

The request structure is straightforward:

run = coasty.runs.create(
    environment="vm_123",
    task="""
    Open the patient record in the billing portal.
    Enter the attached authorization data.
    Do not submit if the member ID or procedure code does not match.
    Return the confirmation number.
    """,
    files=["authorization.pdf"],
    approval_required=["final_submission"]
)

And the response provides a structured log for auditing:

{
    "status": "completed",
    "output": {
      "confirmation_number": "PA-184392"
    },
    "replay_url": "...",
    "events": [
      {
        "type": "verification",
        "field": "member_id",
        "result": "matched"
      }
    ]
}

The real value here isn't just the "computer use" capability—it's the reliability layer. In a production AI workflow, a demo that works once is useless. The ability to define invariants and have the agent re-plan when the application state diverges is what actually makes this viable for an SRE or Ops team to trust. It turns a fragile script into a resilient LLM agent.