Scraping at Scale: What Actually Breaks

Processing millions of pages daily reveals that the actual "scraping" code—the fetch and parse logic—is the easiest part. The real engineering challenge is the infrastructure surrounding it. After maintaining a platform with a 95% success rate, I've found that the failures follow a predictable pattern of systemic collapse rather than simple code bugs.

The Memory-Eating Queue

Crawling is inherently bursty. A single sitemap refresh can dump half a million URLs into your system instantly. If your queue is unbounded, it doesn't just grow; it consumes all available Redis memory until the entire broker crashes.

The fix is simple: bound your queues. It is far better for a crawler to pause discovery for an hour than to trigger an OOM (Out of Memory) event that kills your entire production environment.

The Retry Stampede

When a target site throws 500 errors, a fixed backoff strategy is a recipe for a self-inflicted DDoS. If 40,000 failed pages are rescheduled for the exact same ten-second window, you'll crash the site again and likely get flagged by their WAF.

To solve this, you need a per-domain circuit breaker and full jitter on your backoff logic.

import random

def backoff(attempt, base=2.0, cap=300.0):
    # Full jitter spreads retries across the window to prevent spikes
    return random.uniform(0, min(cap, base * 2 ** attempt))

Efficiency and "Parser Drift"

Fetching the same product via six different URLs (tracking params, variants, etc.) is a waste of resources and increases your block risk. You must canonicalize and deduplicate at the enqueue stage, not after the fetch.

More dangerous is "parser drift." Sites rarely break your code with a crash; instead, a price moves from a DOM node into a script blob, and your field quietly goes null. To combat this, implement field-level fill rate monitoring. Track the percentage of pages yielding values and alert on any significant delta.

I also recommend a multi-strategy extraction workflow:

• Embedded JSON
• Microdata
• DOM selectors

Run these in parallel and merge by confidence. If a redesign kills your CSS selectors, the JSON fallback keeps your data flowing while you update the code.

The Async CPU Trap

Most developers treat scraping as an I/O problem and build everything using asyncio. However, parsing massive HTML documents is CPU-bound. If you aren't careful, your "async" system will spend all its time blocking the event loop on HTML parsing, killing your throughput. For a real-world AI workflow involving large-scale data ingestion, offloading the parsing to a process pool is often the only way to maintain speed.