Fly.io AI Agents: Moving from LLMs to Virtual Machines

The fundamental bottleneck for LLM agents isn't the reasoning capability of the model, but the environment in which they operate. Most current "agents" are just API calls wrapped in a loop, attempting to manipulate a world they can't actually see or touch. Fly.io is pivoting toward a "computer-per-agent" architecture, which effectively treats a lightweight virtual machine (VM) as the agent's physical body.

The "Sandboxed OS" Architecture

Standard prompt engineering for agents usually involves giving the LLM a set of tools (functions) it can call. The problem is that these tools are brittle. If a tool fails or returns an unexpected format, the agent loops or crashes. By deploying agents into their own dedicated VMs, the agent gains a full POSIX-compliant environment.

Instead of calling a read_file() function, the agent simply executes cat filename.txt in a bash shell. This shifts the burden of reliability from the tool-definition layer to the operating system layer.

To implement this locally for testing before deploying to a cloud provider, you can simulate this "agent-in-a-box" using Docker. Here is a basic configuration for a Python-based agent environment that restricts the agent's capabilities while providing a full shell:

# docker-compose.yml for Agent Sandbox
services:
  agent-env:
    image: python:3.11-slim
    volumes:
      - ./agent_workspace:/home/agent/workspace
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
    working_dir: /home/agent/workspace
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
    command: /bin/bash

Why VMs Beat Function Calling

When you move from tool-calling to a VM-based AI workflow, you change the error-handling paradigm. In a traditional LLM agent setup, a syntax error in a generated script usually requires a complex feedback loop to the model. In a VM environment, the agent receives the actual stderr from the shell.

• State Persistence: A VM maintains a filesystem. The agent can install a library via pip, run a script, check the output, and then modify the script based on the result without the developer having to explicitly manage a "state" object in the prompt.
• Dependency Management: If an agent needs a specific version of a tool (e.g., ffmpeg for video processing), it can attempt to install it via apt-get rather than the developer needing to pre-build every possible tool into the agent's API.
• Security Isolation: Running an agent in a VM provides a hard boundary. If the LLM generates a destructive command like rm -rf /, it only kills the ephemeral instance, not the host server.

Real-World Deployment Logic

For those building an LLM agent deployment, the key is managing the lifecycle of these VMs. You cannot keep a VM running 24/7 for every single user query due to cost. The optimal pattern is "Spin up -> Execute -> Snapshot/Destroy."

Here is a conceptual Python snippet showing how to manage an ephemeral agent session via a CLI interface:

import subprocess

class AgentSession:
    def __init__(self, session_id):
        self.session_id = session_id
        self.vm_id = None

    def start_vm(self):
        # Example command to spin up a lightweight Firecracker VM or Docker container
        cmd = f"docker run -d --name agent_{self.session_id} agent-base-image"
        self.vm_id = subprocess.check_output(cmd, shell=True).decode().strip()

    def execute_command(self, command):
        # Executing a command directly in the agent's "body"
        exec_cmd = f"docker exec agent_{self.session_id} /bin/bash -c '{command}'"
        result = subprocess.run(exec_cmd, shell=True, capture_output=True, text=True)
        return result.stdout if result.returncode == 0 else result.stderr

    def terminate(self):
        subprocess.run(f"docker rm -f agent_{self.session_id}", shell=True)

# Usage
session = AgentSession("user_123")
session.start_vm()
print(session.execute_command("ls -la"))
session.terminate()

The Performance Trade-off

The shift to VM-based agents introduces latency. Cold-starting a VM takes longer than a simple API request. However, for complex tasks—like autonomous coding or data analysis—the time spent booting a VM is negligible compared to the time spent in a "hallucination loop" where a tool-based agent fails repeatedly because it lacks the proper environment context.

This approach turns the agent from a "chatbot with plugins" into a "remote operator." By providing a real filesystem and a real shell, we stop trying to simulate a computer inside a prompt and simply give the AI an actual computer.