Spot vs On-Demand GPU: A Practical Cost Breakdown
Here is the framework I now use to decide between on-demand and spot for GPU workloads, based on real experience rather than vendor marketing.
The core tradeoff is simple but easy to misread. On-demand gives you guaranteed capacity at a fixed hourly rate. Spot gives you a steep discount — AWS claims up to 90% off, Google Cloud up to 91% for many machine types — but the provider can reclaim that capacity at any time. The discount exists because the capacity is not yours.
What makes spot work in practice
Spot is worth it when your workload meets all of these conditions:
- Checkpointing is automatic and frequent. If your training script saves state every N minutes and can resume from the last checkpoint, a sudden reclaim is an inconvenience, not a catastrophe.
- The job is retryable. Hyperparameter sweeps, grid searches, and stateless batch inference fit here. If one run dies at hour 6, you lose 6 hours of compute but nothing more.
- There is no hard deadline. If the result is needed next week, a 12-hour delay from queueing and restarting is tolerable.
- You have persistent storage outside the instance. Models, datasets, and intermediate artifacts should live on a separate volume or bucket, not on the ephemeral instance disk.
When on-demand is the smarter choice
I default to on-demand for anything that touches these categories:
- Production inference serving. An interruption means your API goes down and users notice. The cost of downtime dwarfs the hourly savings.
- Live demos and interactive notebooks. If you are presenting to stakeholders or iterating on a prompt in real time, a spot reclaim mid-sentence is a bad look and a lost workflow.
- Deadline-bound fine-tuning. If you have a fixed delivery date, you cannot afford to guess how many interruption cycles you can absorb.
- Long single-run jobs with fragile checkpoints. If your checkpoint logic is buggy or the model state is too large to save frequently, spot becomes a gamble.
The hybrid approach that actually works
My current setup runs a small on-demand baseline for anything latency-sensitive or deadline-critical, and a larger spot fleet for elastic batch work. This is the pattern I recommend for most AI teams because it balances cost control with reliability without requiring every job to be interruption-ready.
One thing I wish I had known earlier: before reaching for spot, compare the lower-cost on-demand GPU tiers first. A smaller instance at full on-demand price can beat a spot instance once you factor in the engineering time spent building resilience around interruptions.
The real question is not which instance type has the lower sticker price. It is whether your workflow can absorb the loss of a machine mid-compute without losing more than the discount saves.