Debugging the timeout cascade that killed our UPI integration

We pushed a routine config change to the payment gateway last Thursday — bumped the Kafka consumer max.poll.records from 500 to 2000 thinking higher throughput would help with the evening peak. Instead, the whole ingestion pipeline melted down in 47 seconds.

The error that started it all:

org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this by either increasing max.poll.interval.ms or by reducing max.poll.records.

Classic rebalance storm. But the weird part? Our processing logic hadn't changed — each transaction still takes ~12ms end-to-end. The math should've worked: 2000 records × 12ms = 24 seconds, well under the default 5-minute max.poll.interval.ms.

Turns out the GC pauses told a different story. During peak load (we hit ~7,200 TPS, not quite UPI's 8k but close), the larger batch size triggered more frequent young GC cycles. Each pause added 200-400ms. Multiply that across 2000 records and suddenly your poll loop exceeds the interval. The coordinator kicks you out, partitions reassign, the new consumer starts fetching from the last committed offset — which is now stale because the previous commit failed. Duplicate processing spikes, downstream idempotency keys collide, and the database connection pool exhausts.

The cascade:

1. Consumer A polls 2000 records
2. GC pause hits at record 1,400 (320ms)
3. max.poll.interval.ms exceeded → LeaveGroup sent
4. Partition revocation → Consumer B takes over
5. Consumer B polls from last committed offset (record 0)
6. Duplicate processing on records 0-1,399
7. Idempotency key conflicts on Redis → retries → more latency
8. HikariCP pool exhausted → HTTP 500s to merchant APIs
9. Circuit breakers trip → full outage

Fix wasn't reverting the config. We kept max.poll.records=2000 but:

• Dropped max.poll.interval.ms to 180000 (3 min) — tighter guardrail
• Added max.poll.interval.ms monitoring with alert at 60% utilization
• Switched to incremental processing: commit every 500 records inside the poll loop via commitSync() instead of waiting for batch completion
• Tuned G1GC: -XX:MaxGCPauseMillis=100 -XX:InitiatingHeapOccupancyPercent=35
• Added a dead-letter topic for idempotency failures so they don't block the main flow

Recovery took 12 minutes once we killed the stuck consumers and let the group stabilize. Zero data loss — exactly 347 duplicates caught by the idempotency layer.

The UPI article mentions 8,000 TPS sustained. We're at ~1/10th that scale and still hitting these coordination edges. Makes you appreciate what NPCI's engineering looks like under the hood — especially their partition strategy across 300+ banks. Anyone here running similar throughput on Kafka? Curious what max.poll.records sweet spot you landed on.