DQN vs Q-Learning: Scaling RL with Neural Networks

A Q-table is useless the moment your state space exceeds a few hundred entries. If you're dealing with something like Atari frames instead of a tiny grid, you can't possibly map every single pixel configuration to a row in a table. This is exactly why Deep Q-Networks (DQN) are necessary—they replace the lookup table with a function approximator.

The core technical shift is simple:
Instead of q_value = q_table[state, action], we use q_values = neural_network(state).

The network takes the state as input and outputs a Q-value for every available action. The Bellman equation and epsilon-greedy strategy remain identical; we've just changed how the values are stored and retrieved. The real power here is generalization. While a table only knows what it has explicitly seen, a neural network recognizes patterns. If three different states share similar features, the network can infer the value of an unvisited state based on its proximity to known ones.

However, combining neural networks with RL is notoriously unstable. To make this work in a real-world AI workflow, two specific architectural fixes are required to prevent the model from collapsing:

• Experience Replay: In standard RL, consecutive states are highly correlated (state 4 → 5 → 6). Training on this sequence is like feeding a model the same image 32 times; it overfits and diverges. By storing transitions in a buffer and sampling random batches, we break this correlation.

replay_buffer.add(state, action, reward, next_state, done)
batch = replay_buffer.sample(batch_size=32) # Random sampling for stability

• Target Networks: The Bellman target depends on the network's own prediction of the next state. If you update the network, the target moves. It's like chasing a moving goalpost. The fix is to maintain two identical networks: an "online" network that learns and a "target" network that remains frozen for a set number of steps to provide a stable baseline.

# Stable target calculation using the frozen target network
target = reward + gamma * max(target_network(next_state))

# Gradient update applied only to the online network
loss = (target - online_network(state)[action]) ** 2

I've been implementing this using JAX to handle the 4x4 gridworld. Even in a simple environment, the difference in how the agent "perceives" the state space via a three-layer MLP versus a table is a great deep dive into why DQN is the foundation for most modern LLM agent logic.

for each episode:
 while not done:
 action = epsilon_greedy(online_network, state)
 next_state, reward, done = env_step(state, action)
 replay_buffer.add(state, action, reward, next_state, done)

 if len(buffer) >= 200:
 batch = replay_buffer.sample(batch_size=32)
 # perform gradient descent on online_network