Resolved

YouTube API quota errors incorrectly archive videos instead of retrying

Python
API
YouTube Data API
Asyncio
Error Handling
Debugging
Backend
Retry Logic

Google YouTube Data API v3 rate limit errors caused videos to be archived prematurely | Python asyncio system failed to classify HTTP 429/403 quota errors correctly, bypassing exponential backoff

A Python-based asyncio system using the Google YouTube Data API v3 was designed to handle video uploads with priority-based scheduling and exponential backoff for rate limits. However, the system incorrectly archived videos when encountering YouTube API quota errors (HTTP 403/429), bypassing the intended retry logic.

Issue

Videos were prematurely moved to the uploads/youtube/archive/ directory when encountering YouTube API rate limits, despite the system’s design to retry such errors with exponential backoff. This resulted in videos disappearing from the upload queue during quota-exceeded periods and being incorrectly archived as if successfully processed.

Culprit

Three interconnected issues in the error handling pipeline caused this behavior:

  1. Implicit None Return in Scheduler: The chnamel verification function returned None implicitly instead of explicitly False for missing chnamels, causing the system to bypass proper error routing.

  2. Incomplete HTTP Error Classification: The error classification logic only caught the 403 quotaExceeded pattern, missing other quota-related errors such as:

    • HTTP 429 (Too Many Requests)
    • Alternative quota messages: userLimitExceeded, dailyLimitExceeded, rateLimitExceeded
    • Auth errors (401) misclassified as PERMANENT (archived after 0 retries).

  3. Overly Broad Exception Default: Non-HTTP error exceptions (including transport errors during quota exhaustion) defaulted to SERVER_ERROR (3 retries → archive), when they should have been classified as QUOTA_EXCEEDED (indefinite 12hr retries).

Solution

Fix 1: Explicit Boolean Return

Modified the chnamel verification logic to return an explicit False when no chnamel was found, ensuring proper error handling flow.

Fix 2: Comprehensive HTTP Error Classification

Expanded the error classification logic to handle all quota and rate limit patterns, including:

  • HTTP 429 (Too Many Requests)
  • Alternative quota messages: userLimitExceeded, dailyLimitExceeded, rateLimitExceeded
  • All quota-related errors now trigger exponential backoff.

Fix 3: Exception Handler Quota Detection

Added string-based quota detection in the exception handler to classify non-HTTP errors (such as transport errors) as QUOTA_EXCEEDED, preventing premature archiving and ensuring indefinite retries.

Retry Behavior Changes:

  • QUOTA_EXCEEDED: Triggers 6 retries followed by indefinite 12-hour retries (never archives).
  • SERVER_ERROR: Reduced from default path (was archiving after 3 retries).
  • AUTH_ERROR: Infinite retries with manual intervention required.

Result: Quota/rate limit errors now consistently trigger exponential backoff (5min → 15min → 1hr → 3hr → 6hr → 12hr) with indefinite 12-hour retries, eliminating premature archiving.