Training RL agents on standard games is a nightmare of

Most researchers trying to train game-playing agents run straight into a wall: the games themselves aren't built for reinforcement learning. You find yourself fighting with non-deterministic physics, messy APIs, or environments that are way too heavy to run in parallel. I've been looking into a project called DelveRL, which aims to solve this by providing a purpose-built, open-source roguelike designed specifically for agent training.

The core idea here is to bridge the gap between the complex strategic depth of a DeepMind-style environment and the practical need for a streamlined agent harness. Instead of trying to wrap a commercial game in a brittle interface, DelveRL is built from scratch as a human-playable game that natively supports a structured API.

Technical breakdown of the environment

The architecture isn't just about making a game; it's about creating a controlled sandbox for high-level decision-making. Here are the specific features that make it viable for a deep dive into RL:

• Deterministic Simulation: This is huge for debugging and reproducibility. You don't want your agent failing because of a random floating-point error in the physics engine.
• Procedural Generation: The levels change every time, forcing agents to learn generalizable strategies rather than just memorizing a specific map layout.
• Partial Observability: Agents can't see the whole map at once, which introduces the need for memory and spatial reasoning.
• Resource Management: It’s an endless turn-based loop where players (or agents) must balance exploration, combat, and risk mitigation to escape each floor.
• Batch-Friendly Execution: The environment includes a renderer-free mode. This allows for massive parallelization, which is essential when you're trying to scale up training.

Performance and baseline metrics

The project doesn't just provide the game; it includes a complete AI workflow. They’ve shipped a recurrent PPO (Proximal Policy Optimization) trainer alongside the environment. Looking at the benchmarks, the baseline agent is actually quite capable:

• Median Floor Reach: 18
• Extended Run Peak: Floor 33

Seeing a baseline hit floor 33 in a roguelike setting is impressive because the difficulty scaling in these types of games is usually exponential. It provides a solid floor for anyone wanting to test more advanced architectures, like transformer-based agents or hierarchical RL.

If you want to experiment with this, the entire stack—the game engine, the training scripts, the checkpoints, and the documentation—is available on GitHub. It’s a great starting point for anyone looking to move beyond simple grid-worlds and into something with actual strategic headroom.

https://github.com/SnyderConsulting/DelveRL