Resolved

Distributed API Race Condition Leading to Duplicate Email Delivery

Distributed Systems
Redis
Kubernetes
Backend
Debugging
Race Conditions

Duplicate email notifications due to distributed API pods processing the same SNS webhook. Solution: Atomic Redis-based deduplication at the webhook entry point.

Issue

A distributed API system processed the same SNS webhook multiple times across multiple pods, resulting in duplicate email notifications for users. Each duplicate notification included a success confirmation followed by a retry suggestion, despite only one booking being created in the database. Log analysis confirmed that both API pods processed the identical webhook independently due to Kubernetes load balancing behavior.

Culprit

The root cause was a distributed systems race condition where:

  • The SNS webhook was distributed to multiple API pods simultaneously via a Kubernetes Service load balancer.
  • A network hiccup or health check timeout triggered a retry to a secondary pod.
  • Both pods processed the webhook independently without shared state coordination, leading to duplicate processing.
  • The second pod legitimately failed to process the webhook (due to the booking already being taken by the first pod) and sent a retry suggestion.

The lack of a deduplication mechanism at the webhook entry point and insufficient tracking of the SNS messageId across pods exacerbated the issue.

Solution

Atomic Deduplication with Redis

An atomic Redis SET NX (Set if Not eXists) operation was implemented at the webhook entry point to ensure only one pod processes each webhook. Key components include:

Redis Deduplication Check

  • A 5-minute TTL was set to cover retry windows while preventing memory leaks.
  • An atomic Redis operation (SET NX) ensures only the first pod to process the webhook succeeds in marking it as processed.
  • If Redis is unavailable, a fail-open strategy allows processing to continue, prioritizing email delivery over strict deduplication.

Webhook Handler Integration

  1. Extract the SNS messageId from both direct and SNS-wrapped messages.
  2. Perform the atomic deduplication check before any processing begins.
  3. Immediately return for duplicate messages with an already_processed status.
  4. Early termination of losing pods to avoid unnecessary processing costs.

Design Decisions

  • Atomicity: Ensures no race conditions by using a single Redis operation.
  • Early return: Losing pods exit immediately after detecting a duplicate.
  • Message ID extraction: Handles multiple message formats (SES and SNS-wrapped).
  • Fail-open: Processes duplicates if Redis is unavailable to ensure no emails are lost.

Result: Guaranteed single email delivery by ensuring only one pod processes each webhook. Losing pods detect duplicates and exit immediately, improving efficiency and reliability.